Mastering Python Web Scraping with the Scrapy Framework

This book is the Chinese translation of the English original Learning Scrapy. Based on Scrapy 1.0, it provides a systematic guide to building scalable, maintainable automated data collection systems with Python and the Scrapy framework. The book comprises 11 chapters plus appendices on software installation and troubleshooting. It opens by introducing Scrapy's role and core components, then explains HTML structure and XPath expressions to help readers reliably extract structured data from pages. Next, starting from crawling a real website, it walks step by step through creating a project, writing spiders, designing items and pipelines, and storing scraped results in databases, delivering them to mobile applications, or exposing them through APIs. The middle section moves on to advanced topics: deploying to the Scrapinghub cloud platform, project configuration and run management, extending Scrapy through extensions and middleware, and techniques for cleaning and storing data in pipelines. The latter part focuses on performance and scale: using intuitive models to understand the utilization and bottlenecks of Scrapy engine components, diagnosing performance issues with telnet and benchmarking systems, and finally building a distributed crawling architecture with Scrapyd, service monitoring, and real-time analytics. Emphasizing learning by doing, the book features example code throughout and is intended for developers who already know basic Python syntax and want to turn crawlers from scripts into engineered systems.