High Performance Python (2nd Edition)

This book is aimed at developers who already know the basics of Python and want to make their programs run faster. Its guiding principle is "measure first, optimize second": first use test and benchmark scripts, cProfile, line_profiler, memory_profiler, and other tools to locate bottlenecks in the program and in memory, then work from pure-Python choices of data structures and algorithms, caching and memoization, and comprehensions and generators, to NumPy and Pandas array-style coding with numexpr, to using Cython for near-C execution speed, and to just-in-time compilation approaches such as Numba and PyPy. The second half discusses ways to implement concurrency and parallelism, including asynchronous programming, multiprocessing, and OpenMP, then distributed computing tools such as Dask, PySpark, and mpi4py, and finally engineering practices for high-performance programs: design thinking, source organization, virtual environments and containerization, and continuous integration. A particle simulator runs through several chapters, comparing the same code at different levels of optimization so readers can understand the applicable boundaries and costs of each technique.