Python Big Data Architecture: Full-Stack Development and Applications

Name: Python Big Data Architecture: Full-Stack Development and Applications
ISBN: 9787121453038

宋天龙 / 张伟松 / 触脉咨询技术团队Publishing House of Electronics Industry · ChinaMay 2023

3.6

Word count: 247K words
Chapters: 85
ISBN: 9787121453038

Report an error Find this book

Summary

This book explains how to use Python for enterprise-grade full-stack development, design, and programming in big data. Topics covered include overall data architecture design, data sources and data collection, data synchronization, message queues, relational databases, NoSQL databases, batch processing, stream processing, graph computing, artificial intelligence, and data product development.

The book provides an accessible introduction to the fundamental principles of various technology components, offers detailed comparisons to help readers choose best-practice technical solutions for different scenarios, and uses hands-on code exercises to help readers quickly master the application of common technologies. Finally, a project case study shows how to apply what has been learned to real business scenarios.

Editor's review

The greatest value of this book lies in the word “full-stack”: it does not settle for explaining how to use a single framework, but instead links data architecture design, data source collection, data synchronization, message queues, relational and NoSQL databases, batch processing, stream processing, graph computing, AI integration, and data product development into a complete engineering chain. The author team combines frontline consulting and implementation experience, so while introducing the principles of each technology component, they also compare the applicable scenarios of different solutions side by side, helping readers build the judgment to “select” rather than “apply mechanically.” The code exercises and the final project case study bring scattered knowledge points back to real business problems, making the book a bridge from single-point skills to system design capability. When reading, it is best to use it as both a reference book and a roadmap: first read through the architectural overview, then read the chapters corresponding to the stages encountered in your project in depth, and build a minimal runnable example while reading—this is far more effective than reading linearly from beginning to end.

Who it's for

A good fit for

Developers who already know Python basics and want to enter the field of big data engineering
Mid- to senior-level engineers who need to undertake data platform architecture design and technology selection
Practitioners moving from traditional backend development or data analysis to full-stack data development
Data product managers who want to understand the entire data pipeline to collaborate better

Not a good fit for

Readers with no programming foundation; the code exercises in the book require some Python proficiency
Readers who only want to learn how to use a single framework and are not interested in architecture or technology selection
Research-oriented readers seeking systematic study of distributed theory derivation

Reading tips

Read the architecture design and technology selection chapters first to build a global map before going deeper into specific components
Run the minimal examples in each chapter hands-on, then practice adapting them to business scenarios
Use the project case study as review material; after finishing the book, try to independently design a simplified data pipeline
Keep the book at hand as a reference manual and read the relevant chapters in a targeted way when encountering specific stages

About the publisher

Publishing House of Electronics Industry China · Founded in 1982

The Publishing House of Electronics Industry is a comprehensive publisher distinguished by its focus on electronic information and related fields. Judging from the more than 3,000 titles in its catalogue, its publishing covers programming and design, artificial intelligence, computer science, finance and economics, photography, and social sciences, attending both to cutting-edge technology and to general readers. Representative titles include Factor Investing: Methods and Practice, x86 Assembly Language: From Real Mode to Protected Mode, The RISC-V Reader, Artificial Intelligence: Speech Recognition, Understanding and Practice, Phased Array Radar Technology, Radar Target Characteristics, and 100 Go Mistakes and How to Avoid Them, most of which centre on technical methods and engineering practice. Its authors include Shi Chuan, Li Zhong, Wang Xiaobo, David Patterson, Andrew Waterman, and Yu Zihao, among others from China and abroad. Owing to limited public information, no further publishing features are elaborated here.

版权信息
内容简介
前言
第1章数据架构整体设计
1.1 数据架构概述
1.2 数据架构设计的8个考虑因素
1.3 数据架构设计的4个核心内容
1.4 常见的6种数据架构
1.5 案例：某B2B企业的数据架构选型
1.6 常见问题
第2章数据源和数据采集
2.1 数据源概述
2.2 企业内部流量数据采集技术选型
2.3 企业外部互联网数据采集技术选型
2.4 使用Requests+BeautifulSoup抓取数据并写入Sqlite
2.5 使用Scrapy+XPath抓取数据并写入MongoDB
2.6 案例：某B2C电商企业的数据源结构
2.7 常见问题
第3章数据同步
3.1 数据同步概述
3.2 数据同步的技术选型
3.3 Python操作DataX实现数据同步
3.4 Python操作第三方库实现Google Analytics数据同步
3.5 案例：某O2O企业离线数据同步案例
3.6 常见问题
第4章消息队列
4.1 消息队列概述
4.2 消息队列的技术选型
4.3 Python操作RabbitMQ处理消息队列服务
4.4 Python操作Kafka处理消息队列服务
4.5 Python操作ZeroMQ处理消息队列服务
4.6 案例：利用消息队列采集电商用户行为数据
4.7 常见问题
第5章关系数据库
5.1 关系数据库概述
5.2 关系数据库的技术选型
5.3 使用基于DB-API 2.0规范的PyMySQL操作MySQL数据库
5.4 使用基于ORM技术的SQLAlchemy操作PostgreSQL数据库
5.5 案例：某传统零售企业基于关系数据库的数据集市
5.6 常见问题
第6章 NoSQL数据库
6.1 NoSQL数据库概述
6.2 不同类型NoSQL数据库的技术选型
6.3 使用Python操作HBase
6.4 使用Python操作Redis
6.5 使用Python操作ES
6.6 使用Python操作Neo4j
6.7 使用Python操作MongoDB
6.8 案例：某菜谱网站基于ES+Redis构建智能搜索推荐引擎
6.9 常见问题
第7章批处理
7.1 批处理概述
7.2 批处理的技术选型
7.3 Python使用PyHive操作HQL进行批处理
7.4 PySpark操作DataFrame进行批处理
7.5 案例：某B2C企业基于PySpark实现用户画像标签的构建
7.6 常见问题
第8章流处理
8.1 流处理概述
8.2 流处理的依赖条件
8.3 流处理的技术选型
8.4 Python操作Structured Streaming实现流处理
8.5 案例：某B2C企业基于Structured Streaming实现实时话题热榜统计
8.6 常见问题
第9章图计算
9.1 图计算概述
9.2 图计算引擎的技术选型
9.3 Python操作GraphFrames实现图计算
9.4 案例：基于用户社交行为的分析
9.5 常见问题
第10章人工智能
10.1 人工智能概述
10.2 人工智能的技术选型
10.3 PySpark ML的应用实践
10.4 案例：某B2C企业推荐系统的搭建与演进
10.5 常见问题
第11章数据产品开发
11.1 数据产品开发概述
11.2 数据产品的路线选型
11.3 Python数据产品自研的技术选型
11.4 基于Django的产品开发
11.5 案例：某企业基于Django构建内部用户画像标签产品
11.6 常见问题
附录A Docker安装使用
附录B 使用EMR搭建Hadoop大数据集群

FAQ

Can I understand this book if I have no big data background and only know Python?

You can get started, but you need to be prepared to fill in some gaps. The book introduces component principles in an accessible way, and the code exercises are mainly in Python, so Python basics are necessary. If you have never encountered concepts such as databases or message queues, it is advisable to first learn the basics of relational databases and general distributed computing knowledge, then read the chapters in order; when you encounter unfamiliar technology, run the examples first and then go back to understand the principles.

Are the technology selection comparisons in the book helpful for actual work?

Yes, and this is exactly what distinguishes the book from pure tutorials. When introducing each stage, the author compares the applicable scenarios of different technical solutions side by side, helping readers understand “why choose this rather than that.” In actual work, technology selection is often more critical than writing code. When reading, it is advisable to compare the content with your company’s data scale and team situation, and organize the comparison conclusions into your own selection checklist.

Is this book suitable as material for internal team sharing?

It is fairly suitable. The book is organized by stages of the data pipeline, and the chapters are relatively independent, so it can be split into a series of sharing sessions on topics such as collection, synchronization, storage, computing, and application. It is advisable to have colleagues responsible for the corresponding stages present, and to adapt the content to the team’s existing technology stack, so that the sharing both covers the knowledge and directly addresses the team’s actual architecture issues.

Comments

Find this book

This site is an index: you start “Search all sources” in your browser, and the entries it lists all point to third-party sites; please judge their availability yourself.

About the author

宋天龙 Author

Public information about Song Tianlong is limited, and his nationality, dates of birth and death, and educational background cannot be confirmed. Judging from the three works held in the collection, his writing centers on computer technology fields such as big data, artificial intelligence, and programming design, with particular attention to the practical application of AIGC technologies. His representative works include AIGC-Assisted Data Analysis and Mining: Methods and Practice Based on ChatGPT, AIGC-Assisted Data Analysis and Data-Driven Operations: Scenario-Based Solutions and Case Studies, and Full-Stack Development and Application of Python Big Data Architecture. These works show that he excels at combining generative tools such as ChatGPT with data analysis, data-driven operations, and full-stack development, pairing methodological explanations with scenario-based cases for technical staff and business readers seeking to use AI to improve data work efficiency. Overall, his writing leans toward practical application and scenario-based solutions and offers some reference value in the categories of general computing and artificial intelligence.

张伟松 Author · China

Zhang Weisong is a Chinese author of books on programming and design. Publicly available information about him is limited, and his detailed biography and educational background cannot be confirmed. One of his works is held in the collection: Full-Stack Development and Application of Python Big Data Architecture. The book falls under the category of programming and design and centers on the Python technology stack, covering full-stack development and practical application in big data architecture. Judging from its subject matter, the author focuses on the complete chain of using Python to build data systems and put engineering into practice, with an emphasis on hands-on operation and an overall grasp of architecture. It is suitable for readers who wish to move from development toward architecture. Given the limited available material, his writing style and industry influence are difficult to assess comprehensively, and readers may refer to the book's specific chapters to understand his technical focus and approach.

触脉咨询技术团队 Author · China

The Touch Consultancy Technical Team appears under a collective byline, and publicly available information is limited, so its membership and specific background cannot yet be confirmed. According to the records in the database, the team's work focuses on programming design and big data technologies, with Python Big Data Architecture Full-Stack Development and Application as its representative title. The book's title indicates an interest in the Python language, big data architecture, and full-stack development and application, making it a comprehensive topic oriented toward technical practice. Since only this one work by the team is included in the database, its writing style and overall influence will require more information before a judgment can be made.