Kafka Client Drivers¶
Kafka clients are available for all major programming languages. This section provides language-specific guides covering installation, configuration, producer/consumer patterns, and best practices.
Official and Confluent-Maintained Clients¶
Apache Kafka Java Client¶
| Library | org.apache.kafka:kafka-clients |
| Repository | github.com/apache/kafka |
| Documentation | kafka.apache.org/documentation |
| Package | Maven Central |
| Current Version | 4.1.x (as of 2025) |
| Maintainer | Apache Software Foundation |
| License | Apache License 2.0 |
The Java client is the reference implementation, developed as part of the core Apache Kafka project. First released with Kafka 0.8.0 in 2013, it replaced the older Scala-based client. The Java client is the most feature-complete, supporting Kafka Streams, exactly-once semantics, and all protocol features immediately upon Kafka release.
librdkafka (C/C++)¶
| Library | librdkafka |
| Repository | github.com/confluentinc/librdkafka |
| Documentation | docs.confluent.io/platform/current/clients/librdkafka |
| Current Version | 2.12.x (as of 2025) |
| Maintainer | Confluent, originally Magnus Edenhill |
| License | BSD 2-Clause |
librdkafka is a high-performance C library created by Magnus Edenhill in 2012, predating the official Java client redesign. Confluent acquired the project in 2016. It serves as the foundation for most non-Java Kafka clients, providing consistent behavior and configuration across Python, Go, .NET, Node.js, Ruby, PHP, and Rust.
confluent-kafka-python¶
| Library | confluent-kafka |
| Repository | github.com/confluentinc/confluent-kafka-python |
| Documentation | docs.confluent.io/platform/current/clients/confluent-kafka-python |
| Package | PyPI |
| Current Version | 2.12.x (as of 2025) |
| Maintainer | Confluent |
| License | Apache License 2.0 |
| Base | librdkafka |
The official Python client wrapping librdkafka via C extensions. First released in 2016, it provides significantly higher performance than pure-Python alternatives. Supports Schema Registry integration with Avro, Protobuf, and JSON Schema serializers.
kafka-python (Pure Python)¶
| Library | kafka-python |
| Repository | github.com/dpkp/kafka-python |
| Documentation | kafka-python.readthedocs.io |
| Package | PyPI |
| Current Version | 2.2.x (as of 2025) |
| Maintainer | Dana Powers (dpkp) |
| License | Apache License 2.0 |
A pure Python implementation with no native dependencies. Mirrors the Java client API with Pythonic features like consumer iterators. Supports transactions, all compression types, and consumer groups. With 5.9k GitHub stars and 35,000+ dependent projects, it's ideal for serverless and environments where native compilation is problematic.
confluent-kafka-go¶
| Library | github.com/confluentinc/confluent-kafka-go |
| Repository | github.com/confluentinc/confluent-kafka-go |
| Documentation | docs.confluent.io/platform/current/clients/confluent-kafka-go |
| Package | pkg.go.dev |
| Current Version | v2.12.x (as of 2025) |
| Maintainer | Confluent |
| License | Apache License 2.0 |
| Base | librdkafka |
The official Go client using cgo bindings to librdkafka. First released in 2017 as v0.9.x, the v2.x series (2023) introduced Go modules support and improved APIs. Requires librdkafka to be installed or uses a pre-built static library.
confluent-kafka-dotnet¶
| Library | Confluent.Kafka |
| Repository | github.com/confluentinc/confluent-kafka-dotnet |
| Documentation | docs.confluent.io/platform/current/clients/confluent-kafka-dotnet |
| Package | NuGet |
| Current Version | 2.12.x (as of 2025) |
| Maintainer | Confluent |
| License | Apache License 2.0 |
| Base | librdkafka |
The official .NET client providing P/Invoke bindings to librdkafka. First released in 2016 as RdKafka, rebranded to Confluent.Kafka in 2017. Supports .NET Standard 2.0+, .NET Framework 4.6.2+, and .NET Core 2.0+.
Community-Maintained Clients¶
KafkaJS (Node.js)¶
| Library | kafkajs |
| Repository | github.com/tulios/kafkajs |
| Documentation | kafka.js.org |
| Package | npm |
| Current Version | 2.2.x (as of 2025) |
| Maintainer | Tulio Ornelas and community |
| License | MIT |
| Base | Pure JavaScript (no native dependencies) |
A pure JavaScript implementation requiring no native compilation. Created by Tulio Ornelas in 2018, it gained popularity for its easy installation and modern async/await API. No librdkafka dependency makes it ideal for serverless and containerized environments where native compilation is problematic.
node-rdkafka (Node.js)¶
| Library | node-rdkafka |
| Repository | github.com/Blizzard/node-rdkafka |
| Documentation | blizzard.github.io/node-rdkafka |
| Package | npm |
| Current Version | 3.6.x (as of 2025) |
| Maintainer | Blizzard Entertainment |
| License | MIT |
| Base | librdkafka |
Native Node.js bindings to librdkafka, developed by Blizzard Entertainment. Offers higher performance than KafkaJS at the cost of native compilation requirements. First released in 2016.
rust-rdkafka (Rust)¶
| Library | rdkafka |
| Repository | github.com/fede1024/rust-rdkafka |
| Documentation | docs.rs/rdkafka |
| Package | crates.io |
| Current Version | 0.38.x (as of 2025) |
| Maintainer | Federico Giraud and community |
| License | MIT |
| Base | librdkafka |
Rust bindings to librdkafka with idiomatic Rust API. Created by Federico Giraud in 2016. Supports async/await via Tokio runtime. The library remains at 0.x version indicating ongoing API evolution.
Pure Go Clients¶
For Go developers who need to avoid cgo dependencies (cross-compilation, Alpine Linux, serverless), two popular pure Go implementations exist:
| Library | Repository | Version | Features |
|---|---|---|---|
| franz-go | github.com/twmb/franz-go | v1.20.x | Full EOS, transactions, all SASL |
| kafka-go | github.com/segmentio/kafka-go | v0.4.x | Simple API, Go stdlib style |
franz-go is the most feature-complete pure Go client, supporting transactions and exactly-once semantics. Created by Travis Bischel, it's used in production by Redpanda, Mux, and others.
kafka-go by Segment provides a simpler API modeled after Go's standard library. It lacks transaction support but is well-suited for straightforward producer/consumer use cases.
Other Language Clients¶
| Language | Library | Repository | Maintainer | Base |
|---|---|---|---|---|
| Ruby | rdkafka-ruby | github.com/appsignal/rdkafka-ruby | AppSignal | librdkafka |
| PHP | php-rdkafka | github.com/arnaud-lb/php-rdkafka | Arnaud Le Blanc | librdkafka |
| Erlang | brod | github.com/kafka4beam/brod | Klarna | Native |
| Elixir | broadway_kafka | github.com/dashbitco/broadway_kafka | Dashbit | librdkafka |
| Scala | fs2-kafka | github.com/fd4s/fs2-kafka | Community | Java client |
librdkafka Ecosystem¶
Most non-Java clients are built on librdkafka, a high-performance C library:
librdkafka Benefits¶
| Benefit | Description |
|---|---|
| Performance | Highly optimized C implementation |
| Feature parity | All clients support same Kafka features |
| Consistent behavior | Same configuration options across languages |
| Battle-tested | Powers production systems at massive scale |
librdkafka Configuration¶
librdkafka-based clients share a common configuration namespace documented at github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md:
# Common librdkafka configuration
bootstrap.servers=kafka:9092
client.id=my-application
acks=all
enable.idempotence=true
compression.type=lz4
Version Compatibility¶
Kafka Protocol Compatibility¶
| Client | Minimum Kafka | Recommended Kafka |
|---|---|---|
| Java 4.1.x | 0.10.0+ | 3.0+ |
| librdkafka 2.12.x | 0.8.0+ | 2.0+ |
| KafkaJS 2.2.x | 0.10.0+ | 2.0+ |
Feature Availability by Version¶
| Feature | Java | librdkafka | KafkaJS |
|---|---|---|---|
| Idempotent producer | 0.11.0+ | 0.11.4+ | 2.0.0+ |
| Transactions | 0.11.0+ | 0.11.4+ | 2.0.0+ |
| Headers | 0.11.0+ | 0.11.4+ | 1.4.0+ |
| Cooperative rebalancing | 2.4.0+ | 1.4.0+ | Not supported |
| Static membership | 2.3.0+ | 1.4.0+ | 2.0.0+ |
Choosing a Client¶
Decision Factors¶
| Factor | Consideration |
|---|---|
| Language ecosystem | Match application's primary language |
| Performance requirements | librdkafka-based clients offer highest throughput |
| Deployment constraints | Pure-language clients avoid native dependencies |
| Feature requirements | Check client support for transactions, exactly-once |
| Community support | Consider maintenance activity and documentation |
Feature Comparison¶
| Feature | Java | Python | Go | Node.js (KafkaJS) | .NET | Rust |
|---|---|---|---|---|---|---|
| Producer | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Consumer | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Transactions | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Idempotence | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Admin API | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Schema Registry | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ |
| SASL/SSL | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Kafka Streams | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
✅ Full support | ⚠️ Partial/community support | ❌ Not available
Performance Comparison¶
General throughput characteristics (actual results vary by workload):
| Client | Relative Throughput | Notes |
|---|---|---|
| Java | 100% | Reference implementation |
| librdkafka (C) | 95-105% | Sometimes faster than Java |
| Python (confluent-kafka) | 90-95% | librdkafka overhead minimal |
| Go (confluent-kafka-go) | 90-95% | cgo overhead minimal |
| .NET (Confluent.Kafka) | 85-95% | P/Invoke overhead |
| KafkaJS | 50-70% | Pure JS, no native code |
Common Patterns¶
Producer Pattern¶
All clients follow a similar producer pattern:
- Create configuration
- Instantiate producer
- Send messages (async with callback or sync)
- Handle delivery reports
- Flush and close
Consumer Pattern¶
All clients follow a similar consumer pattern:
- Create configuration with group.id
- Instantiate consumer
- Subscribe to topics
- Poll for messages in loop
- Process messages
- Commit offsets
- Close on shutdown
Security Configuration¶
SASL/PLAIN¶
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.username=user
sasl.password=password
SASL/SCRAM¶
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
sasl.username=user
sasl.password=password
mTLS¶
security.protocol=SSL
ssl.ca.location=/path/to/ca.crt
ssl.certificate.location=/path/to/client.crt
ssl.key.location=/path/to/client.key
Language-Specific Guides¶
- Java - Native Apache Kafka client
- Python - confluent-kafka-python with librdkafka
- Go - confluent-kafka-go with librdkafka
- Node.js - KafkaJS (pure JS) and node-rdkafka
- .NET - confluent-kafka-dotnet with librdkafka
- Rust - rdkafka with librdkafka
External Resources¶
Official Documentation¶
Community Resources¶
Related Documentation¶
- Producer Development - Producer patterns and configuration
- Consumer Development - Consumer patterns and configuration
- Client Connections - Protocol and connection architecture