Inside Spark SQL

Inside Spark SQL focuses on Spark SQL, the subsystem of Apache Spark responsible for structured data processing, dismantling its overall architecture and the implementation mechanisms of its internal modules from a source-code-level perspective. It was among the earlier Chinese technical books to systematically explain the internals of Spark SQL. The book has 11 chapters, broadly divided into four parts. Part One, background and foundations (Chapters 1–2), reviews the evolution of big data and Spark, of the relational model and SQL, and briefly introduces the RDD programming model and the basic concepts of DataFrame and Dataset. Part Two covers the stages of SQL implementation (Chapters 3–6): using a simple query as an example, it traces the complete execution path from SQL to RDD, introduces the three internal systems—InternalRow, TreeNode, and Expression—and the internal data type system, then examines in turn the ANTLR-based SQL parser, the generation of the logical plan (LogicalPlan) and the Analyzer, the Optimizer's rules, and the generation of the physical plan (PhysicalPlan) and common Strategies. Part Three consists of special topics (Chapters 7–10), focusing on the implementation of the two core operators, Aggregation and Join, analyzing in depth the memory management and code generation optimizations of the Tungsten project, and explaining how Spark SQL connects to Hive. Part Four is practice (Chapter 11), sharing experience in system modification and performance optimization in production environments. The book assumes readers already have some experience with Spark and a foundation in SQL, and is suited to engineers who want to move from "knowing the API" to "understanding the internals."