Posted inSecurity

Production Apache Kafka Security: Hardening with TLS Encryption, SASL/SCRAM, and ACLs

Security tutorial - IT technology blog
Security tutorial - IT technology blog

Why Open Kafka Clusters Fail in Production

Our team learned this lesson the hard way. Early on a Tuesday morning, network egress unexpectedly jumped from our baseline 15 MB/s to nearly 400 MB/s. A staging pod inside our VPC had booted up pointing to the broker's default plaintext port 9092. Because the broker had no encryption, no credentials, and no access controls, that single test container started pulling unmasked credit card transactions straight off the wire.

Out of the box, Apache Kafka defaults to developer convenience. Every message travels across the network in plaintext. Any host that can route traffic to port 9092 can read any topic, push corrupt records, or silently delete partitions. Securing a production deployment requires three distinct safeguards:

  • Transport Layer Security (TLS): Encrypts broker-to-broker and client-to-broker traffic. On modern CPUs with AES-NI instructions, this typically adds less than 3% CPU overhead.
  • SASL/SCRAM-512: Authenticates clients using salted challenge-response hashes. It avoids sending raw passwords over the network and skips the operational headache of a Kerberos/KDC infrastructure.
  • Access Control Lists (ACLs): Restricts authenticated principals to explicit operations on designated topics and consumer groups.

Step 1: Certificate Authority and Keystore Generation

Before modifying Kafka configuration files, you need a Public Key Infrastructure (PKI). We will spin up a dedicated Certificate Authority (CA) and issue signed keystores containing Subject Alternative Names (SANs) so modern JVM hostname checks pass.

1. Generate an Internal Certificate Authority

# Generate an internal CA key and root certificate valid for 3 years
openssl req -new -x509 -keyout ca-key -out ca-cert -days 1095 \
  -subj "/CN=Internal-Kafka-CA/OU=Infra/O=ItFromZero/C=US" \
  -nodes

2. Generate Broker Keystores with SANs

Run these steps on each broker node (for example, kafka1.internal at IP 10.0.1.10). Modern Java versions enforce TLS hostname verification by default, so omitting the SAN extension will break client handshakes.

# 1. Create the broker keystore with SAN extension
keytool -keystore server.keystore.jks -alias kafka1.internal \
  -validity 365 -genkey -keyalg RSA -keysize 2048 \
  -storepass "KeystoreSecret123!" -keypass "KeystoreSecret123!" \
  -dname "CN=kafka1.internal,OU=Infra,O=ItFromZero,C=US" \
  -ext SAN=DNS:kafka1.internal,IP:10.0.1.10

# 2. Export the certificate signing request (CSR)
keytool -keystore server.keystore.jks -alias kafka1.internal \
  -certreq -file cert-file -storepass "KeystoreSecret123!" \
  -ext SAN=DNS:kafka1.internal,IP:10.0.1.10

# 3. Sign the CSR with your internal CA
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file \
  -out cert-signed -days 365 -CAcreateserial \
  -extfile <(printf "subjectAltName=DNS:kafka1.internal,IP:10.0.1.10")

# 4. Import the CA certificate into the keystore
keytool -keystore server.keystore.jks -alias CARoot \
  -import -noprompt -file ca-cert -storepass "KeystoreSecret123!"

# 5. Import the signed certificate
keytool -keystore server.keystore.jks -alias kafka1.internal \
  -import -noprompt -file cert-signed -storepass "KeystoreSecret123!"

3. Build Truststores for Brokers and Clients

Brokers and producers must trust the root CA that signed your cluster certificates.

# Create a truststore with the CA certificate
keytool -keystore server.truststore.jks -alias CARoot \
  -import -noprompt -file ca-cert -storepass "TruststoreSecret123!"

# Distribute a copy for client runtimes
cp server.truststore.jks client.truststore.jks

Step 2: Hardening Broker Configurations

1. Broker Listener Setup

Edit your broker config (typically /etc/kafka/server.properties). We will configure port 9093 for mutual TLS encryption paired with SASL/SCRAM authentication, leaving port 9092 closed entirely.

# Listeners and protocols
listeners=SASL_SSL://kafka1.internal:9093
advertised.listeners=SASL_SSL://kafka1.internal:9093
listener.security.protocol.map=SASL_SSL:SASL_SSL
inter.broker.listener.name=SASL_SSL
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512

# TLS Keystore and Truststore
ssl.keystore.location=/var/private/ssl/server.keystore.jks
ssl.keystore.password=KeystoreSecret123!
ssl.key.password=KeystoreSecret123!
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=TruststoreSecret123!
ssl.endpoint.identification.algorithm=HTTPS

# Authentication
sasl.enabled.mechanisms=SCRAM-SHA-512

# Authorization (KRaft StandardAuthorizer / Kafka 3.x+)
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
allow.everyone.if.no.acl.found=false
super.users=User:admin

2. Configure the Broker JAAS File

Brokers authenticate incoming client handshakes through Java Authentication and Authorization Service (JAAS). Save this file as /etc/kafka/kafka_server_jaas.conf. When managing production clusters, generating resilient passwords is non-negotiable. I use the password generator at toolcraft.app/en/tools/security/password-generator for server passwords — it runs entirely in the browser, so no data is ever sent over the network, making it reliable for offline infrastructure setup.

KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"
    password="e9V#kP9$mR2@zQ7!xT5";
};

Client {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"
    password="e9V#kP9$mR2@zQ7!xT5";
};

Point the Kafka process to your JAAS file before starting the daemon:

export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"
systemctl restart kafka

3. Provision User Credentials via SCRAM-SHA-512

With the broker listening on TLS, register user credentials directly in cluster metadata. Bootstrap using an admin credentials file (/tmp/admin-bootstrap.properties):

# Register the cluster admin user
kafka-configs.sh --bootstrap-server kafka1.internal:9093 \
  --entity-type users --entity-name admin \
  --alter --add-config 'SCRAM-SHA-512=[password=e9V#kP9$mR2@zQ7!xT5]' \
  --command-config /tmp/admin-bootstrap.properties

# Register the application service account
kafka-configs.sh --bootstrap-server kafka1.internal:9093 \
  --entity-type users --entity-name payments-worker \
  --alter --add-config 'SCRAM-SHA-512=[password=uP4!vL9$xN2@bM7^kQ1]' \
  --command-config /tmp/admin-bootstrap.properties

4. Grant Scoped Access with ACLs

Because allow.everyone.if.no.acl.found=false is active, all topics are denied by default. Grant payments-worker precise rights to write outbound events and read inbound tasks:

# Allow payments-worker to write to topic 'payment-events'
kafka-acls.sh --bootstrap-server kafka1.internal:9093 \
  --command-config /tmp/admin-bootstrap.properties \
  --add --allow-principal User:payments-worker \
  --operation Write --operation Describe \
  --topic payment-events \
  --resource-pattern-type literal

# Allow payments-worker to consume 'settlement-tasks' inside 'payments-group'
kafka-acls.sh --bootstrap-server kafka1.internal:9093 \
  --command-config /tmp/admin-bootstrap.properties \
  --add --allow-principal User:payments-worker \
  --operation Read --operation Describe \
  --topic settlement-tasks \
  --group payments-group

Step 3: Verification and Observability

Verify your perimeter with real producer commands before rolling changes into production.

1. Verify Authorized Traffic

Define client settings in client-payments.properties:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/var/private/ssl/client.truststore.jks
ssl.truststore.password=TruststoreSecret123!
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="payments-worker" \
    password="uP4!vL9$xN2@bM7^kQ1";

Publish a test message to the approved topic:

echo '{"tx_id": "98412", "amount": 120.50}' | kafka-console-producer.sh \
  --bootstrap-server kafka1.internal:9093 \
  --topic payment-events \
  --producer.config client-payments.properties

2. Confirm Unauthorized Requests Get Blocked

Attempting to produce to an unassigned topic like audit-logs must fail cleanly:

echo 'test event' | kafka-console-producer.sh \
  --bootstrap-server kafka1.internal:9093 \
  --topic audit-logs \
  --producer.config client-payments.properties

The broker will reject the batch immediately:

[2026-11-07 02:45:12,104] ERROR Error when sending message to topic audit-logs with key: null
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [audit-logs]

3. Production Metrics to Track in Prometheus

Expose Kafka JMX metrics to alert on probing attempts and expiring keys:

  • kafka.server:type=RequestMetrics,name=ErrorsPerSec,error=TOPIC_AUTHORIZATION_FAILED: Any sustained rate above 0 req/sec points to misconfigured credentials or unauthorized network discovery scans.
  • kafka.server:type=RequestMetrics,name=ErrorsPerSec,error=SASL_AUTHENTICATION_FAILED: High spikes signal brute-force authentication attacks or outdated service passwords.
  • ssl-certificate-expiration-date: Trigger alerts 30 days before broker certificates expire to prevent accidental TLS handshake lockouts.

Enforcing TLS, SCRAM-512, and least-privilege ACLs turns an open event bus into a hardened messaging layer ready for compliance audits.

Share:
Tags: