Python Data Scraping: Techniques and Practice

Using Python as its implementation language, this book follows the complete chain of "getting data—analyzing data—presenting data" and is organized into three parts—data scraping, data analysis, and data visualization—each relatively independent so that readers can read selectively. Chapter 1 opens with a complete small example that walks readers through installing the Python environment, using pip, and looking up help, while quickly reviewing the language elements that later chapters will inevitably use: file operations, loops, exceptions, tuples, lists, dictionaries, sets, random numbers, and enumerate. Chapter 2 focuses on string parsing, introducing common string functions, regular expressions, BeautifulSoup, and JSON structures as a foundation for extracting useful information from web page text. Chapter 3 moves into single-machine data scraping, covering the basic workflow of sequential scraping, use of the requests library, and concurrent and parallel scraping. Chapter 4 expands the perspective to distributed systems, introducing RPC calls and the Celery task system. Chapter 5 is devoted to Selenium, covering single-machine scraping, distributed scraping, and running under Linux without a graphical interface, for pages that depend on script rendering. Chapter 6 discusses the problem of IP blocking during scraping, explaining Tor installation and use, multithreaded approaches, and integration with Selenium. Chapter 7 collects common difficult scenarios such as Flash, desktop applications, USB drives, secondary and tertiary pages, image processing, and App data scraping. Chapter 8 presents the design and examples of a monitoring framework, and Chapter 9 briefly introduces the Hadoop ecosystem and setting up a Cloudera environment as groundwork for subsequent processing of massive data. Throughout, the book emphasizes applying mature principles and popular frameworks to concrete problems, with runnable example code that readers can run and modify, turning the examples into scraping programs suited to their own needs.