Spark Full-Stack Data Analytics

Originally published in English as Agile Data Science 2.0: Building Full-Stack Data Analytics Applications with Spark, this book draws on Russell Jurney's experience building analytical products at companies such as LinkedIn and Ning to set out a practical methodology for doing data science in an agile way. With Spark as the core processing engine, it connects the entire chain—from event data collection and serialized storage, through batch and stream processing, to publishing data with MongoDB, providing search with Elasticsearch, and presenting charts and reports on the front end—grounding "data science" in an analytical application that can actually be delivered to users. The book is divided into two parts. Part I, "Preparation," explains the methodology of agile data science and the division of roles within a team, introduces the selection of a toolset and the setup of local and cloud environments (Vagrant, EC2, Jupyter notebooks), and presents the example dataset used throughout; it covers the basics of tools including Python 3, JSON lines and Parquet serialization, Spark data processing, MongoDB, Elasticsearch, Apache Kafka, and PySpark Streaming. Part II, "Climbing the Pyramid," is a step-by-step tutorial for building an analytical application: collecting and displaying flight records, adding charts for visualization, exploring entity relationships in the data through interactive reports, predicting whether flights will be on time, deploying the prediction system, and then iterating to improve prediction performance. An appendix provides a complete installation guide. The book does not aim for mathematical depth in algorithms but emphasizes the engineering loop that "makes data valuable," making it suitable as a first full-stack practical book for data science beginners and engineers changing roles.