Skip to content

Getting Started with Kafka

This guide covers Kafka installation, client setup, and basic operations to begin working with event streaming.

Prerequisites

Requirement Minimum Recommended
Java JDK 11 JDK 17 or 21
Memory 4 GB 8+ GB
Disk 10 GB SSD recommended
OS Linux, macOS, Windows Linux for production

Installation Options

Option Use Case Complexity
Local Installation Development, learning Low
Docker Development, CI/CD Low
Kubernetes Production, cloud-native Medium
Managed Service Production without operational overhead Low

Quick Start

1. Download and Extract

# Download Kafka
curl -O https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz

# Extract
tar -xzf kafka_2.13-3.6.0.tgz
cd kafka_2.13-3.6.0

2. Start Kafka (KRaft Mode)

Kafka 3.3+ supports KRaft mode, eliminating the ZooKeeper dependency.

# Generate cluster ID
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

# Format storage
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

# Start Kafka
bin/kafka-server-start.sh config/kraft/server.properties

3. Create a Topic

bin/kafka-topics.sh --create \
  --topic quickstart-events \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

4. Produce Messages

bin/kafka-console-producer.sh \
  --topic quickstart-events \
  --bootstrap-server localhost:9092

Type messages and press Enter to send each one. Press Ctrl+C to exit.

5. Consume Messages

bin/kafka-console-consumer.sh \
  --topic quickstart-events \
  --from-beginning \
  --bootstrap-server localhost:9092

Client Drivers

Kafka provides official and community clients for multiple languages.

Language Client Documentation
Java Apache Kafka Client Driver Guide
Python confluent-kafka-python Driver Guide
Go confluent-kafka-go Driver Guide
Node.js kafkajs Driver Guide
.NET confluent-kafka-dotnet Driver Guide

→ Client Drivers

Verification

After installation, verify the cluster is operational:

# Check broker status
bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# List topics
bin/kafka-topics.sh --list --bootstrap-server localhost:9092

# Describe cluster
bin/kafka-metadata.sh --snapshot /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log --command "describe"

Configuration

Essential Broker Settings

Property Description Default
broker.id Unique broker identifier Required
listeners Network listeners PLAINTEXT://:9092
log.dirs Data directory /tmp/kafka-logs
num.partitions Default partitions for new topics 1
default.replication.factor Default replication factor 1

Production Recommendations

Setting Development Production
log.dirs Single directory Multiple directories on separate disks
num.partitions 1-3 Based on throughput requirements
default.replication.factor 1 3
min.insync.replicas 1 2

→ Configuration Reference

Next Steps

Goal Documentation
Understand Kafka concepts Event Streaming Concepts
Build a producer application Producer Guide
Build a consumer application Consumer Guide
Set up Kafka Connect Kafka Connect Guide
Configure for production Operations Guide