Skip to content

Kafka Wire Protocol Specification

This documentation provides a comprehensive specification of the Apache Kafka binary wire protocol. The protocol defines how clients communicate with brokers over TCP connections using a request-response model with explicit versioning.

Specification Scope

This specification covers Kafka protocol versions through Kafka 3.7. For the authoritative grammar reference and exhaustive API schemas, the Apache Kafka Protocol Documentation provides additional detail. This documentation provides:

  • Behavioral contracts and guarantees
  • Failure semantics and error handling
  • Implementation guidance and constraints
  • Version-specific behavioral differences
  • Complete error code reference with recovery actions

Protocol Documentation

Core Specifications

Document Description
Protocol Primitives Data types: integers, varints, strings, bytes, arrays, tagged fields
Protocol Messages Message framing, request/response headers, correlation IDs
Protocol Records Record batch format, compression, timestamps, transactions
Protocol Errors Complete error code reference (133+ codes)

API References

Document APIs Covered
Core APIs Produce, Fetch, Metadata, ListOffsets, ApiVersions
Consumer APIs FindCoordinator, JoinGroup, Heartbeat, SyncGroup, OffsetCommit
Admin APIs CreateTopics, DeleteTopics, ACLs, Configs, ElectLeaders
Transaction APIs InitProducerId, AddPartitionsToTxn, EndTxn, TxnOffsetCommit

Protocol Overview

Design Principles

The Kafka protocol is a binary, request-response protocol with the following characteristics:

Principle Description
Binary encoding All messages use binary encoding for efficiency
Length-prefixed Messages are prefixed with a 4-byte size field
Explicit versioning Each API request specifies its version
Correlation-based Requests and responses matched by correlation ID
Multiplexed Multiple requests may be in-flight per connection

Message Flow

uml diagram

Behavioral Contract

Request Processing Guarantee

Brokers must process requests from a single connection in the order received. Responses must be sent in request order.

Protocol Layers

Layer Architecture

uml diagram

Layer Responsibility
Application Business logic, data serialization
Client API Producer/Consumer abstractions
NetworkClient Connection management, request dispatch
Selector Non-blocking I/O, channel management
Transport TCP, TLS encryption, SASL authentication

API Key Summary

All Protocol APIs

Key Name Category Description
0 Produce Core Send records to partitions
1 Fetch Core Retrieve records from partitions
2 ListOffsets Core Query offset by timestamp
3 Metadata Core Discover cluster topology
4-7 Controller APIs Internal Broker coordination
8-9 OffsetCommit/Fetch Consumer Offset management
10-16 Group APIs Consumer Consumer group protocol
17-18 ApiVersions/SASL Core Version negotiation, auth
19-21 Topic Management Admin Create, delete, modify topics
22-28 Transaction APIs Transaction Exactly-once semantics
29-31 ACL APIs Admin Authorization management
32-51 Config/Admin Admin Cluster administration
52-75 KRaft/Advanced Internal KRaft consensus, features

See individual API documentation for complete details.

Version Compatibility

Client-Broker Compatibility

Client Version Broker 2.x Broker 3.x Notes
0.10.x ⚠️ ⚠️ Limited API support
2.0.x ✅ ✅ Full support
2.4.x ✅ ✅ Flexible versions
3.0.x ✅ ✅ KRaft APIs
3.5.x+ ⚠️ ✅ May use new APIs

Legend: ✅ Full Support | ⚠️ Partial Support | ❌ Not Supported

Flexible Versions

Kafka 2.4 introduced "flexible versions" (KIP-482):

Feature Non-Flexible Flexible
Strings STRING (INT16 length) COMPACT_STRING (VARINT)
Arrays ARRAY (INT32 count) COMPACT_ARRAY (VARINT)
Tagged fields ❌ ✅
Forward compatibility Limited Improved

Implementation Requirements

Client Requirements

Requirement Level Reference
Negotiate API versions before requests must Protocol Messages
Track correlation IDs for response matching must Protocol Messages
Handle all error codes appropriately must Protocol Errors
Refresh metadata on leader change errors should Core APIs
Use exponential backoff on retries should Protocol Errors
Ignore unknown tagged fields must Protocol Primitives

Broker Requirements

Requirement Level Reference
Process requests in order per connection must Protocol Messages
Echo correlation ID exactly in response must Protocol Messages
Support advertised API version ranges must Core APIs
Respond to ApiVersions pre-authentication must Core APIs

Quick Reference

Common Error Codes

Code Name Retriable Action
0 NONE N/A Success
6 NOT_LEADER_OR_FOLLOWER ✅ Refresh metadata
7 REQUEST_TIMED_OUT ✅ Retry with backoff
25 UNKNOWN_MEMBER_ID ❌ Rejoin group
27 REBALANCE_IN_PROGRESS ✅ Rejoin group
47 INVALID_PRODUCER_EPOCH ❌ Close producer

See Protocol Errors for complete reference.

Key Configurations

Setting Default Purpose
request.timeout.ms 30000 Request timeout
retry.backoff.ms 100 Retry backoff
metadata.max.age.ms 300000 Metadata refresh interval
max.in.flight.requests.per.connection 5 Request pipelining

Protocol Specifications

API References

Client Architecture