Python Crawler Development: From Beginner to Practice (Micro-Course Edition)

谢乾坤Posts & Telecom Press · ChinaJul 2019

Rating 4.0
Word count
208K words
Chapters
85
ISBN
9787115490995

Tags

General Computing Advanced TechWeb ScrapingProject-Based LearningStep-by-Step GuideField CasesBeginner Friendly

Summary

This book offers a fairly comprehensive introduction to the development process of targeted crawlers, methods for cracking various anti-crawler mechanisms, and related crawler development techniques. The book consists of 13 chapters, including an introduction, Python fundamentals, regular expressions and file operations, simple web crawler development, high-performance HTML content parsing, Python and databases, asynchronous loading and request headers, simulated login and CAPTCHAs, packet capture and man-in-the-middle crawlers, Android native app crawlers, Scrapy, Scrapy applications, and legal and ethical issues in crawler development. Except for Chapters 1, 12, and 13, every chapter ends with hands-on practice to help readers consolidate what they have learned in that chapter and previous ones. Video explanations are also provided for difficult content in the book to better demonstrate the related operations.

Editor's review

Python Web Scraping: From Beginner to Practice (Micro-course Edition), written by Xie Qiankun and published by Posts & Telecom Press, is an introductory scraping tutorial for beginners. Built on Python 3, it systematically explains core web scraping concepts, commonly used libraries (such as Requests, Beautiful Soup, and Scrapy), data storage, anti-scraping strategies, and other practical techniques. Its highlight lies in combining micro-course videos with case-driven instruction to help readers progressively master the entire scraping development workflow.

In terms of structure, the book starts with environment setup and basic syntax and gradually moves into advanced topics such as dynamic page scraping and distributed scraping, balancing systematic coverage with practicality. The cases are rich, including real-world scenarios such as e-commerce sites and social media, which help readers understand how scraping applies in actual work. The author, Xie Qiankun, has years of scraping development experience, and his teaching style emphasizes code standards and troubleshooting, making the book suitable for readers who want to learn scraping systematically.

Note that the book focuses on beginner-to-intermediate content and may seem basic to developers with a strong programming background. Nevertheless, as an introductory guide to scraping, it offers a clear learning path and actionable exercises, helping readers quickly build scraping projects and understand solutions to common problems.

Who it's for

A good fit for

Not a good fit for

Reading tips

  1. Follow the chapters in order and focus on hands-on practice with each chapter's code examples
  2. Use the accompanying micro-course videos to deepen understanding of complex concepts
  3. Try modifying the case code and applying it to data sources you are interested in

About the publisher

Posts & Telecom Press China · Founded in 1953

Posts & Telecom Press is known for its strengths in science and technology, computing, and information and communications titles, and has extended its publishing into photography, art, software learning, and inspirational growth, giving it a fairly diverse range. Its catalog includes computing and communications works such as Refactoring and Patterns (Revised Edition) and Huawei Switch Learning Guide (Second Edition), alongside bilingual collector's editions such as The Little Prince (Chinese-English-French illustrated collector's edition), The Great Gatsby, and Three Days to See. Beijing Ten Years: 1980-1990 and Ghost Blade: WLOP Personal Illustration Collection reflect its presence in photography and art. Its authors include Antoine de Saint-Exupéry, Helen Keller, Joshua Kerievsky, Wang Da, Zhang Zhaozeng, and WLOP.

Contents

  1. 版权信息
  2. 内容提要
  3. 前言
  4. 第1章 绪论
  5. 1.1 爬虫
  6. 1.2 爬虫可以做什么
  7. 1.3 爬虫开发技术
  8. 第2章 Python基础
  9. 2.1 Python的安装和运行
  10. 2.2 Python开发环境
  11. 2.3 Python的数据结构和控制结构
  12. 2.4 函数与类
  13. 2.5 阶段案例——猜数游戏
  14. 2.6 本章小结
  15. 2.7 动手实践
  16. 第3章 正则表达式与文件操作
  17. 3.1 正则表达式
  18. 3.2 Python文件操作
  19. 3.3 阶段案例——半自动爬虫开发
  20. 3.4 本章小结
  21. 3.5 动手实践
  22. 第4章 简单的网页爬虫开发
  23. 4.1 使用Python获取网页源代码
  24. 4.2 多线程爬虫
  25. 4.3 爬虫的常见搜索算法
  26. 4.4 阶段案例——小说网站爬虫开发
  27. 4.5 本章小结
  28. 4.6 动手实践
  29. 第5章 高性能HTML内容解析
  30. 5.1 HTML基础
  31. 5.2 XPath
  32. 5.3 Beautiful Soup4
  33. 5.4 阶段案例——大麦网演出爬虫
  34. 5.5 本章小结
  35. 5.6 动手实践
  36. 第6章 Python与数据库
  37. 6.1 MongoDB
  38. 6.2 Redis
  39. 6.3 MongoDB的优化建议
  40. 6.4 阶段案例
  41. 6.5 本章小结
  42. 6.6 动手实践
  43. 第7章 异步加载与请求头
  44. 7.1 异步加载
  45. 7.2 请求头（Headers）
  46. 7.3 模拟浏览器
  47. 7.4 阶段案例
  48. 7.5 本章小结
  49. 7.6 动手实践
  50. 第8章 模拟登录与验证码
  51. 8.1 模拟登录
  52. 8.2 验证码
  53. 8.3 阶段案例——自动登录果壳网
  54. 8.4 本章小结
  55. 8.5 动手实践
  56. 第9章 抓包与中间人爬虫
  57. 9.1 数据抓包
  58. 9.2 中间人爬虫
  59. 9.3 阶段案例——Keep热门
  60. 9.4 本章小结
  61. 9.5 动手实践
  62. 第10章 Android原生App爬虫
  63. 10.1 实现原理
  64. 10.2 综合应用
  65. 10.3 阶段案例——BOSS直聘爬虫
  66. 10.4 本章小结
  67. 10.5 动手实践
  68. 第11章 Scrapy
  69. 11.1 Scrapy的安装
  70. 11.2 Scrapy的使用
  71. 11.3 Scrapy与MongoDB
  72. 11.4 Scrapy与Redis
  73. 11.5 阶段案例——博客爬虫
  74. 11.6 本章小结
  75. 11.7 动手实践
  76. 第12章 Scrapy高级应用
  77. 12.1 中间件（Middleware）
  78. 12.2 爬虫的部署
  79. 12.3 分布式架构
  80. 12.4 阶段案例
  81. 12.5 本章小结
  82. 第13章 爬虫开发中的法律和道德问题
  83. 13.1 法律问题
  84. 13.2 道德协议
  85. 13.3 本章小结

FAQ

How much programming background does this book require? You need basic Python syntax, such as variables, loops, and functions. If you have no background, it is advisable to first build up introductory Python knowledge.
Do the cases in the book apply to the latest website structures? The cases mainly teach methodology and tool usage, but some website structures may change over time, so readers need to adapt the code flexibly.
How can I access the micro-course videos? Companion video resources can be viewed via the QR codes provided in the book or the publisher's official website.

Related on WeRead

Python爬虫开发：从入门到实战（微课版） 谢乾坤

This site is an index: you start “Search all sources” in your browser, and the entries it lists all point to third-party sites; please judge their availability yourself.

About the author

谢乾坤 Author

Xie Qiankun is an author about whom little is recorded in publicly available sources. What is known is a single computer-related work under his name: Python Web Crawler Development: From Beginner to Practice (Micro-course Edition). Since information such as his nationality, dates of birth and death, and professional background cannot be verified, public material is limited, and only the work itself can be briefly described here. As the title suggests, the book is aimed at beginners in Python web crawling, covering a complete path from basic fundamentals to practical project development, and it includes micro-course teaching materials as a supplement. It is a typical hands-on technical tutorial. Books of this kind usually emphasize practical operation and case demonstrations, helping readers grasp the basic workflow and common methods of crawler development in a relatively short time. Based on the available clues, Xie Qiankun's writing focuses on technology popularization and introductory teaching, and his work is classified under general computer science, making it suitable as a starting reference for learning Python web crawling. More material is needed to fill in the author's other writings and background.