Apache Kafka Monitoring - Stream Processing & Message Broker Monitoring

Monitor Apache Kafka message broker with comprehensive Kafka monitoring for collecting Kafka cluster metrics, topic metrics, consumer lag, and producer performance. This guide explains how to monitor Apache Kafka for stream processing monitoring, message broker observability, and real-time data pipeline monitoring using OpenTelemetry Collector.

Overview

Apache Kafka monitoring is essential for ensuring message broker performance, stream processing reliability, and data integrity. With OpenTelemetry and OpenObserve, you gain real-time visibility into Kafka clusters, Kafka topics, consumer groups, and producer performance for comprehensive Kafka monitoring and observability.

Steps to Integrate

Prerequisites

OpenObserve account (Cloud or Self-Hosted)
Kafka running

Step 1: Create Test Kafka Topics (Optional)

Create test topics to simulate workloads:

bin/kafka-topics.sh --create --topic quickstart-events-1 --partitions 3 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic quickstart-events-2 --partitions 10 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic openobserve --partitions 2 --bootstrap-server localhost:9092

List topics to confirm:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 2: Install OpenTelemetry Collector Contrib

Download and install the latest otelcol-contrib.

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.116.1/otelcol-contrib_0.116.1_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.116.1_linux_amd64.deb

Note: Replace version numbers in download links if a newer release is available. Always check the GitHub Releases page.

Step 3: Get OpenObserve URL and Access Key

In OpenObserve: go to Data Sources → Custom → Metrics → Otel Collector
Copy the ingestion URL and Access Key

Step 4: Configure OpenTelemetry Collector

Edit the configuration file:

sudo vi /etc/otelcol-contrib/config.yaml

Update config for Kafka metrics:

receivers:
kafkametrics:
    brokers: localhost:9092
    protocol_version: 2.0.0
    scrapers:
    - brokers
    - topics
    - consumers

exporters:
otlphttp/openobserve:
    endpoint: OPENOBSERVE_ENDPOINT
    headers:
    Authorization: OPENOBSERVE_TOKEN
    stream-name: OPENOBSERVE_STREAM

service:
pipelines:
    metrics:
    receivers: [kafkametrics]
    exporters: [otlphttp/openobserve]

Replace the following with your OpenObserve details:

OPENOBSERVE_ENDPOINT → API endpoint (e.g., https://api.openobserve.ai)
OPENOBSERVE_TOKEN → API token
OPENOBSERVE_STREAM → Stream name

Step 5: Start OpenTelemetry Collector

Start and check status:

sudo systemctl start otelcol-contrib
sudo systemctl status otelcol-contrib

Check logs:

journalctl -u otelcol-contrib --no-pager -n 50

Step 6: Visualize in OpenObserve

Go to Streams → select metrics. Expand on any metric stream to see the events.

Tip

You can use prebuilt dashboards to quickly get started with Kafka monitoring. Browse and import dashboards from: OpenObserve Community Dashboards

Troubleshooting

Collector Fails to Start: failed to build pipelines: unknown receiver kafkametrics
- Cause: Using an older otelcol-contrib version that doesn’t support the kafkametrics receiver.
- Fix: Upgrade to at least v0.116.0 or newer from GitHub Releases.
No Metrics Visible in OpenObserve
- Double-check values in your config.yaml against your OpenObserve → Data Sources → Custom → Metrics → Otel Collector page.
Kafka Connection Errors: dial tcp 127.0.0.1:9092: connect: connection refused
- Cause: Kafka broker not running or wrong address.
- Fix: Ensure Kafka is started and reachable. Run:
```
bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092
```