Skip to content

Apache Kafka® Reference Documentation

Apache Kafka® is a distributed event streaming platform designed for high-throughput, fault-tolerant data pipelines. This documentation provides comprehensive coverage of Kafka architecture, operations, and best practices for production deployments.

Documentation Scope

This reference documentation covers Apache Kafka versions 2.8 through 4.1.x, with particular emphasis on KRaft-mode deployments. Kafka 4.0 (March 2025) removed ZooKeeper entirely—KRaft is now the only supported metadata management mode.

Version Range ZooKeeper Mode KRaft Mode Documentation Status
2.8.x ✅ ⚠️ Early Access Legacy reference
3.0.x - 3.2.x ✅ ⚠️ Preview Legacy reference
3.3.x - 3.5.x ✅ ✅ Supported
3.6.x - 3.9.x ⚠️ Deprecated ✅ Fully Documented
4.0.x ❌ Removed ✅ Fully Documented
4.1.x ❌ Removed ✅ Current (4.1.1)

Legend: ✅ Production Ready | ⚠️ Limited/Deprecated | ❌ Not Supported

What's New

Kafka 4.1.1 (November 2025) - Current Release

  • Bug fixes and stability improvements

Kafka 4.1.0 (September 2025)

  • Queues for Kafka preview (KIP-932) - Promoted from early access to preview (4.0 early-access clusters cannot upgrade directly)
  • Streams Rebalance Protocol (KIP-1071) - Early access broker-coordinated rebalancing for Kafka Streams
  • OAuth jwt-bearer grant (KIP-1139) - New grant type avoiding secrets in configuration
  • Plugin metrics (KIP-877) - Plugins and connectors can register metrics via Monitorable interface
  • Connector versioning - Install and run multiple versions of same connector plugins
  • Transaction filtering (KIP-1152) - Transactional ID pattern filter in ListTransactions API
  • Consumer close options - New Consumer.close(CloseOptions) method controlling group leave behavior
  • Rack-aware assignment - Memory efficiency improvements for large consumer groups
  • KRaft voter upgrade (KIP-853) - Upgrade voters from static to dynamic configuration
  • High-watermark fix (KIP-1166) - Fixes pending fetch requests blocking high-watermark progression

Kafka 4.0.1 (October 2025)

  • Fixes 49 issues since 4.0.0 release

Kafka 4.0.0 (March 2025) - Major Release

  • ZooKeeper removed - KRaft is now the only metadata management mode
  • New Consumer Group Protocol GA (KIP-848) - Dramatically faster rebalances
  • Queues for Kafka early access (KIP-932) - Traditional queue semantics (promoted to preview in 4.1)
  • Java 17 required for brokers, Connect, and tools (Java 11 for clients)
  • Log4j2 replaces Log4j 1.x
  • MirrorMaker 1 removed - Use MirrorMaker 2

Kafka 3.9.1 (May 2025)

  • Fixes 66 issues since 3.9.0 release

Kafka 3.9.0 (November 2024)

  • Dynamic KRaft quorum (KIP-853) - Add/remove controllers without downtime
  • Final 3.x release before ZooKeeper removal

Kafka 3.8.1 (October 2024)

  • Fixes 17 issues since 3.8.0 release

Kafka 3.8.0 (July 2024)

  • Compression level configuration (KIP-390)
  • Tiered storage JBOD compatibility (early access)

Kafka 3.7.2 (December 2024)

  • Fixes 22 issues since 3.7.1 release

Kafka 3.7.1 (June 2024)

  • Bug fixes and stability improvements

Kafka 3.7.0 (February 2024)

  • JBOD in KRaft (KIP-858) - Early access
  • Client metrics (KIP-714)
  • Next-gen consumer rebalance (KIP-848) - Early access

Getting Started

New to Apache Kafka? Begin with installation and initial configuration.

  • Installation

    Deploy Kafka on Linux, containerized environments, or managed cloud services.

    Installation Guide

  • Client Drivers

    Configure client libraries for Java, Python, Go, and other languages.

    Driver Documentation

Core Concepts

Understand the fundamental concepts underpinning Kafka's distributed architecture.

  • Architecture Patterns

    Event sourcing, CQRS, and streaming architectures built on Kafka.

    Architecture Patterns

  • Delivery Semantics

    At-least-once, at-most-once, and exactly-once delivery guarantees.

    Delivery Semantics

  • Data Integration

    Integrating Kafka with external systems using Connect and Streams.

    Data Integration

  • Multi-Datacenter

    Cross-datacenter replication strategies and disaster recovery.

    Multi-Datacenter

Architecture

Deep dive into Kafka's internal architecture and distributed systems design.

  • Topics & Partitions

    Topics, partitions, partition leaders, ISR, and leader election.

    Topics & Partitions

  • Brokers

    Broker architecture, request handling, and controller responsibilities.

    Broker Architecture

  • Topology

    Cluster topology, rack awareness, and network configuration.

    Cluster Topology

  • Storage Engine

    Log segments, compaction, and retention policies.

    Storage Engine

  • Replication

    Partition replication, ISR management, and leader election.

    Replication

  • Fault Tolerance

    Failure detection, recovery mechanisms, and high availability.

    Fault Tolerance

  • Transaction Coordinator

    Exactly-once semantics, producer IDs, and two-phase commit.

    Transactions

Producers and Consumers

Configuration and best practices for Kafka clients.

  • Producers

    Producer configuration, batching, compression, and delivery guarantees.

    Producer Guide

  • Consumers

    Consumer groups, offset management, and rebalancing strategies.

    Consumer Guide

Stream Processing

Build real-time stream processing applications.

  • Kafka Connect

    Source and sink connectors for data integration pipelines.

    Kafka Connect

  • Kafka Streams

    Stream processing DSL for stateful transformations.

    Kafka Streams

  • Schema Registry

    Schema management for Avro, Protobuf, and JSON Schema.

    Schema Registry

Operations

Production deployment, monitoring, and maintenance procedures.

  • CLI Tools

    Command-line tools for cluster administration and troubleshooting.

    CLI Reference

  • Configuration

    Broker, producer, and consumer configuration reference.

    Configuration Guide

  • Monitoring

    JMX metrics, health checks, and alerting strategies.

    Monitoring Guide

  • Backup & Restore

    Data backup strategies and disaster recovery procedures.

    Backup & Restore

  • Maintenance

    Rolling upgrades, partition reassignment, and cluster expansion.

    Maintenance

  • Performance Tuning

    Throughput optimization, latency tuning, and capacity planning.

    Performance

Security

Authentication, authorization, and encryption for Kafka deployments.

  • Authentication

    SASL/SCRAM, SASL/GSSAPI (Kerberos), and mTLS authentication.

    Authentication

  • Authorization

    ACL-based authorization and role-based access control.

    Authorization

  • Encryption

    TLS/SSL encryption for data in transit.

    Encryption

Cloud Deployments

Deploy Kafka on cloud platforms and container orchestration systems.

Troubleshooting

Diagnostic procedures and solutions for common issues.

  • Common Errors

    Error codes, root causes, and resolution procedures.

    Error Reference

  • Diagnosis

    Diagnostic procedures for cluster issues.

    Diagnosis Guide

  • Log Analysis

    Log patterns, analysis techniques, and monitoring.

    Log Analysis

Quick Reference

  • Reference

    Configuration reference, CLI commands, and metrics catalog.

    Quick Reference

  • Migration

    ZooKeeper to KRaft migration and version upgrade guides.

    Migration Guide

Version Compatibility

This documentation follows Apache Kafka's semantic versioning. Behavioral differences between versions are explicitly noted throughout.

KRaft Migration Timeline

Milestone Version Date Status
KRaft Early Access 2.8.0 April 2021 Development only
KRaft Preview 3.0.0 - 3.2.x Sept 2021 - May 2022 Testing environments
KRaft Production Ready 3.3.0 October 2022 General availability
ZooKeeper Deprecated 3.6.0 October 2023 Migration recommended
Dynamic KRaft Quorum 3.9.0 November 2024 Add/remove controllers
ZooKeeper Removed 4.0.0 March 2025 KRaft required
Current Release 4.1.1 November 2025 Latest stable

Kafka 4.0+ Migration Path

Kafka 4.0 does not support ZooKeeper mode. To upgrade:

  1. If on Kafka < 3.3: Upgrade to 3.9.x first
  2. Migrate from ZooKeeper to KRaft mode
  3. Then upgrade to Kafka 4.0+

Direct upgrade from ZooKeeper mode to 4.0 is not possible.

Documentation Conventions

This documentation uses RFC 2119 terminology (must, should, may) to indicate requirement levels. Version-specific behaviors are explicitly noted with the applicable Kafka version range.