Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, and replayable message streams. Unlike traditional message brokers, Kafka persists messages to disk in ordered, append-only topic logs and retains them for a configurable period (days or weeks). Consumers read messages at their own pace using consumer offsets, and any consumer can re-read the same messages. This makes Kafka ideal for event sourcing, real-time analytics pipelines, log aggregation, and change data capture (CDC). Kafka uses ZooKeeper for cluster coordination in older versions, but from Kafka 3.3+ the KRaft (Kafka Raft) mode eliminates the ZooKeeper dependency entirely. This guide covers installing Kafka 3.7 in KRaft mode on RHEL 9, creating topics, and producing/consuming messages with the command-line tools and Python.

Prerequisites

  • RHEL 9 with Java 17+ installed
  • At least 4 GB RAM

Step 1 — Install Java

dnf install -y java-17-openjdk-headless
java -version

Step 2 — Download and Install Kafka

KAFKA_VER="3.7.1"
cd /opt
curl -LO "https://downloads.apache.org/kafka/${KAFKA_VER}/kafka_2.13-${KAFKA_VER}.tgz"
tar xzf kafka_2.13-${KAFKA_VER}.tgz
ln -s kafka_2.13-${KAFKA_VER} kafka
useradd -rs /bin/false kafka
chown -R kafka:kafka /opt/kafka*

Step 3 — Configure Kafka in KRaft Mode (No ZooKeeper)

# Generate a unique cluster UUID
KAFKA_UUID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
echo "Cluster UUID: $KAFKA_UUID"

# Format the storage directory
/opt/kafka/bin/kafka-storage.sh format 
    -t "$KAFKA_UUID" 
    -c /opt/kafka/config/kraft/server.properties

Step 4 — Create systemd Service

# /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka Server (KRaft)
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/java-17-openjdk"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now kafka
/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Step 5 — Create Topics and Test with CLI Tools

# Create a topic with 3 partitions and 1 replica (single-node)
/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 
    --create --topic events --partitions 3 --replication-factor 1

# Produce messages
echo "Hello Kafka" | /opt/kafka/bin/kafka-console-producer.sh 
    --bootstrap-server localhost:9092 --topic events

# Consume from the beginning
/opt/kafka/bin/kafka-console-consumer.sh 
    --bootstrap-server localhost:9092 --topic events --from-beginning

Step 6 — Produce and Consume with Python (kafka-python)

pip install kafka-python
# producer.py
from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)
producer.send('events', {'user_id': 123, 'action': 'page_view', 'page': '/home'})
producer.flush()
print("Event sent")
# consumer.py
from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'events',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',
    value_deserializer=lambda v: json.loads(v.decode('utf-8')),
    group_id='analytics-group'
)
for message in consumer:
    print(f"Partition {message.partition} | {message.value}")

Conclusion

Apache Kafka in KRaft mode on RHEL 9 eliminates the ZooKeeper dependency and simplifies single-node and cluster deployments. The key operational parameter is log.retention.hours in server.properties — default 168 hours (7 days). Increase this for event sourcing use cases where consumers may need to replay historical events. For production clusters, deploy at least 3 brokers with replication factor 3 and min.insync.replicas=2 to ensure no message loss during a broker failure.

Next steps: How to Install RabbitMQ on RHEL 9, How to Configure Kafka Clusters on RHEL 9, and How to Monitor Kafka with Prometheus JMX Exporter on RHEL 9.