Apache Kafka in Practice

Name: Apache Kafka in Practice
ISBN: 9787121337765

胡夕Publishing House of Electronics Industry · ChinaJul 2018

4.2

Word count: 1.6M words
Chapters: 271
ISBN: 9787121337765

Report an error Find this book

Summary

Combining typical use cases, the author provides a fairly comprehensive explanation of the entire Kafka technical system so that readers can draw inferences and apply it directly in practice. The book also explores in some depth Kafka's design principles and its stream-processing components, and provides detailed case studies. The book is divided into 10 chapters: Chapter 1 comprehensively introduces messaging engine systems and Kafka's basic concepts and features, quickly leading readers into the world of Kafka; Chapter 2 briefly reviews the development history of Apache Kafka; Chapter 3 describes in detail how to set up a Kafka cluster environment; Chapters 4 and 5 explore in depth how to use Kafka clients; Chapter 6 gives readers an overview of Kafka's internal design principles; Chapters 7–9 explain Kafka cluster management, monitoring, and tuning through examples; and Chapter 10 introduces Kafka's newly introduced stream-processing component.

Editor's review

Written by Hu Xi and published by Publishing House of Electronics Industry, Apache Kafka in Action is a hands-on guide to Apache Kafka, a mainstream distributed messaging system. The book has earned strong ratings on Douban, where readers generally praise its detailed content and close alignment with production environments. As a seasoned expert in the Kafka field with extensive practical experience, Hu Xi ensures the book's practicality and authority.

From a professional standpoint, the book's greatest value lies in its clear practical orientation. It not only systematically explains Kafka's core concepts, architectural design, and working principles, but more importantly offers in-depth analysis of concrete approaches to cluster deployment, performance tuning, monitoring and operations, and integration with upstream and downstream ecosystems such as Spark and Flink. The book includes numerous configuration examples, troubleshooting approaches, and best practices drawn from real business scenarios, offering direct reference value for developers and architects who are using or planning to use Kafka in mission-critical business systems. It helps readers bridge the gap between understanding concepts and solving real-world problems.

Reading this book, you gain not only a Kafka user manual but also a methodology for building highly reliable, high-performance data pipelines. The book's coverage of Kafka's new features and its outlook on future trends also reflect the timeliness and forward-looking nature of its content. For technical professionals who wish to delve into messaging queues and stream processing, this is a desk reference worth reading closely.

Who it's for

A good fit for

Engineers and architects who are using or planning to use Kafka for system development in production environments.
Mid- to senior-level backend developers who want a deeper understanding of the principles of distributed messaging systems and stream processing platforms.
Operations engineers responsible for Kafka cluster deployment, performance tuning, and day-to-day maintenance.
Data platform developers interested in integrating big data ecosystems such as Spark and Flink with messaging queues.

Not a good fit for

Complete programming beginners with no foundational concepts of distributed systems or messaging queues.
Shallow users who only need to know Kafka's most basic API calls and do not require in-depth principles or operations knowledge.

Reading tips

It is recommended to read alongside a real development or test environment and to practice the configuration and code examples in the book hands-on.
The book can serve as a reference manual; when specific deployment or performance issues arise, consult the relevant chapters as needed.
Pay attention to the author's discussion of design trade-offs and troubleshooting approaches, which is more valuable than memorizing specific commands.

About the publisher

Publishing House of Electronics Industry China · Founded in 1982

The Publishing House of Electronics Industry is a comprehensive publisher distinguished by its focus on electronic information and related fields. Judging from the more than 3,000 titles in its catalogue, its publishing covers programming and design, artificial intelligence, computer science, finance and economics, photography, and social sciences, attending both to cutting-edge technology and to general readers. Representative titles include Factor Investing: Methods and Practice, x86 Assembly Language: From Real Mode to Protected Mode, The RISC-V Reader, Artificial Intelligence: Speech Recognition, Understanding and Practice, Phased Array Radar Technology, Radar Target Characteristics, and 100 Go Mistakes and How to Avoid Them, most of which centre on technical methods and engineering practice. Its authors include Shi Chuan, Li Zhong, Wang Xiaobo, David Patterson, Andrew Waterman, and Yu Zihao, among others from China and abroad. Owing to limited public information, no further publishing features are elaborated here.

版权信息
作者介绍
前言
第1章认识ApacheKafka
1.1Kafka快速入门
1.1.1 下载并解压缩Kafka二进制代码压缩包文件
1.1.2 启动服务器
1.1.3 创建topic
1.1.4 发送消息
1.1.5 消费消息
1.2 消息引擎系统
1.2.1 消息设计
1.2.2 传输协议设计
1.2.3 消息引擎范型
1.2.4 Java消息服务
1.3Kafka概要设计
1.3.1 吞吐量/延时
1.3.2 消息持久化
1.3.3 负载均衡和故障转移
1.3.4 伸缩性
1.4Kafka基本概念与术语
1.4.1 消息
1.4.2 topic和partition
1.4.3 offset
1.4.4 replica
1.4.5 leader和follower
1.4.6 ISR
1.5Kafka使用场景
1.5.1 消息传输
1.5.2 网站行为日志追踪
1.5.3 审计数据收集
1.5.4 日志收集
1.5.5 Event Sourcing
1.5.6 流式处理
1.6 本章小结
第2 章 Kafka发展历史
2.1Kafka的历史
2.1.1 背景
2.1.2Kafka横空出世
2.1.3Kafka开源
2.2Kafka版本变迁
2.2.1Kafka的版本演进
2.2.2Kafka的版本格式
2.2.3 新版本功能简介
2.2.4 旧版本功能简介
2.3 如何选择Kafka版本
2.3.1 根据功能场景
2.3.2 根据客户端使用场景
2.4Kafka与Confluent
2.5 本章小结
第3章 Kafka线上环境部署
3.1 集群环境规划
3.1.1 操作系统的选型
3.1.2 磁盘规划
3.1.3 磁盘容量规划
3.1.4 内存规划
3.1.5 CPU规划
3.1.6 带宽规划
3.1.7 典型线上环境配置
3.2 伪分布式环境安装
3.2.1 安装Java
3.2.2 安装ZooKeeper
3.2.3 安装单节点Kafka集群
3.3 多节点环境安装
3.3.1 安装多节点ZooKeeper集群
3.3.2 安装多节点Kafka
3.4 验证部署
3.4.1 测试topic创建与删除
3.4.2 测试消息发送与消费
3.4.3 生产者吞吐量测试
3.4.4 消费者吞吐量测试
3.5 参数设置
3.5.1 broker端参数
3.5.2 topic级别参数
3.5.3 GC参数
3.5.4 JVM参数
3.5.5 OS参数
3.6 本章小结
第4章 producer开发
4.1 producer概览
4.2 构造producer
4.2.1 producer程序实例
4.2.2 producer主要参数
4.3 消息分区机制
4.3.1 分区策略
4.3.2 自定义分区机制
4.4 消息序列化
4.4.1 默认序列化
4.4.2 自定义序列化
4.5 producer拦截器
4.6 无消息丢失配置
4.6.1 producer端配置
4.6.2 broker端配置
4.7 消息压缩
4.7.1Kafka支持的压缩算法
4.7.2 算法性能比较与调优
4.8 多线程处理
4.9 旧版本producer
4.10 本章小结
第5章 consumer开发
5.1 consumer概览
5.1.1 消费者（consumer）
5.1.2 消费者组（consumer group）
5.1.3 位移（offset）
5.1.4 位移提交
5.1.5__consumer_offsets
5.1.6 消费者组重平衡（consumer group rebalance）
5.2 构建consumer
5.2.1 consumer程序实例
5.2.2 consumer脚本命令
5.2.3 consumer主要参数
5.3 订阅topic
5.3.1 订阅topic列表
5.3.2 基于正则表达式订阅topic
5.4 消息轮询
5.4.1 poll内部原理
5.4.2 poll使用方法
5.5 位移管理
5.5.1 consumer位移
5.5.2 新版本consumer位移管理
5.5.3 自动提交与手动提交
5.5.4 旧版本consumer位移管理
5.6 重平衡（rebalance）
5.6.1 rebalance概览
5.6.2 rebalance触发条件
5.6.3 rebalance分区分配
5.6.4 rebalance generation
5.6.5 rebalance协议
5.6.6 rebalance流程
5.6.7 rebalance监听器
5.7 解序列化
5.7.1 默认解序列化器
5.7.2 自定义解序列化器
5.8 多线程消费实例
5.8.1 每个线程维护一个KafkaConsumer
5.8.2 单KafkaConsumer实例+多worker线程
5.8.3 两种方法对比
5.9 独立consumer
5.10 旧版本consumer
5.10.1 概览
5.10.2 high-level consumer
5.10.3 low-level consumer
5.11 本章小结
第6章 Kafka设计原理
6.1 broker端设计架构
6.1.1 消息设计
6.1.2 集群管理
6.1.3 副本与ISR设计
6.1.4 水印（watermark）和leader epoch
6.1.5 日志存储设计
6.1.6 通信协议（wire protocol）
6.1.7 controller设计
6.1.8 broker请求处理
6.2 producer端设计
6.2.1 producer端基本数据结构
6.2.2 工作流程
6.3 consumer端设计
6.3.1 consumer group状态机
6.3.2 group管理协议
6.3.3 rebalance场景剖析
6.4 实现精确一次处理语义
6.4.1 消息交付语义
6.4.2 幂等性producer（idempotent producer）
6.4.3 事务（transaction）
6.5 本章小结
第7章管理Kafka集群
7.1 集群管理
7.1.1 启动broker
7.1.2 关闭broker
7.1.3 设置JMX端口
7.1.4 增加broker
7.1.5 升级broker版本
7.2 topic管理
7.2.1 创建topic
7.2.2 删除topic
7.2.3 查询topic列表
7.2.4 查询topic详情
7.2.5 修改topic
7.3 topic动态配置管理
7.3.1 增加topic配置
7.3.2 查看topic配置
7.3.3 删除topic配置
7.4 consumer相关管理
7.4.1 查询消费者组
7.4.2 重设消费者组位移
7.4.3 删除消费者组
7.4.4 kafka-consumer-offset-checker
7.5 topic分区管理
7.5.1 preferred leader选举
7.5.2 分区重分配
7.5.3 增加副本因子
7.6Kafka常见脚本工具
7.6.1 kafka-console-producer脚本
7.6.2 kafka-console-consumer脚本
7.6.3 kafka-run-class脚本
7.6.4 查看消息元数据
7.6.5 获取topic当前消息数
7.6.6 查询__consumer_offsets
7.7 API方式管理集群
7.7.1 服务器端API管理topic
7.7.2 服务器端API管理位移
7.7.3 客户端API管理topic
7.7.4 客户端API查看位移
7.7.5 0.11.0.0版本客户端API
7.8 MirrorMaker
7.8.1 概要介绍
7.8.2 主要参数
7.8.3 使用实例
7.9Kafka安全
7.9.1 SASL+ACL
7.9.2 SSL加密
7.10 常见问题
7.11 本章小结
第8章监控Kafka集群
8.1 集群健康度检查
8.2 MBean监控
8.2.1 监控指标
8.2.2 指标分类
8.2.3 定义和查询JMX端口
8.3 broker端JMX监控
8.3.1 消息入站/出站速率
8.3.2 controller存活JMX指标
8.3.3 备份不足的分区数
8.3.4 leader分区数
8.3.5 ISR变化速率
8.3.6 broker I/O工作处理线程空闲率
8.3.7 broker网络处理线程空闲率
8.3.8 单个topic总字节数
8.4 clients端JMX监控
8.4.1 producer端JMX监控
8.4.2 consumer端JMX监控
8.5 JVM监控
8.5.1 进程状态
8.5.2 GC性能
8.6 OS监控
8.7 主流监控框架
8.7.1 JmxTool
8.7.2 kafka-manager
8.7.3KafkaMonitor
8.7.4KafkaOffset Monitor
8.7.5 CruiseControl
8.8 本章小结
第9章调优Kafka集群
9.1 引言
9.2 确定调优目标
9.3 集群基础调优
9.3.1 禁止atime更新
9.3.2 文件系统选择
9.3.3 设置swapiness
9.3.4 JVM设置
9.3.5 其他调优
9.4 调优吞吐量
9.5 调优延时
9.6 调优持久性
9.7 调优可用性
9.8 本章小结
第10章 KafkaConnect与KafkaStreams
10.1 引言
10.2KafkaConnect
10.2.1 概要介绍
10.2.2 standalone Connect
10.2.3 distributed Connect
10.2.4 开发connector
10.3KafkaStreams
10.3.1 流处理
10.3.2KafkaStreams核心概念
10.3.3KafkaStreams与其他框架的异同
10.3.4 Word Count实例
10.3.5KafkaStreams应用开发
10.3.6KafkaStreams状态查询
10.4 本章小结

FAQ

Which version of Kafka are the code examples in this book based on?

The book is mainly based on a relatively recent Kafka version at the time of writing; please refer to the preface or relevant chapter notes for the specific version. Since Kafka iterates quickly, be mindful of potential changes to APIs or configurations, though the core architecture and design philosophy retain long-term reference value.

Does the book cover Kafka security authentication and permission management?

Yes. As a practical guide, the book typically covers security topics essential for production environments, such as the configuration and use of SASL and ACL mechanisms; please refer to the table of contents for specific chapters.

What are the advantages of this book compared with the official documentation?

The official documentation focuses on feature descriptions and API references, whereas this book's strengths lie in its integrated, systematic knowledge framework, best practices from real-world experience, solutions to common problems, and approaches to integrating with other systems, offering context and decision guidance closer to engineering practice.

Comments

Find this book

This site is an index: you start “Search all sources” in your browser, and the entries it lists all point to third-party sites; please judge their availability yourself.

About the author

胡夕 Author

Hu Xi is a Chinese author of technical books, about whom limited public information is available; the only confirmed representative work is Apache Kafka in Action. The book focuses on Apache Kafka, a distributed streaming platform, and is a practice-oriented technical title classified under artificial intelligence. Judging from the subject matter, Hu Xi's interests lie in message middleware and real-time data processing, and his writing leans toward hands-on application, aimed at developers and engineers who wish to deploy and use Kafka in real projects. Since only one of his works is held in the collection, no reliable information is available about his nationality, dates of birth and death, education, or professional career, and this introduction makes no inferences on those points. Based on the available material, Hu Xi's writing is oriented toward practical implementation, combining platform mechanisms with engineering practice, and his work offers some reference value among Chinese-language learning materials on Kafka.