Skip to content

Integration with Apache Kafka

This guide explains how to monitor Apache Kafka using the OpenTelemetry Collector Contrib (otelcol-contrib) and export metrics to OpenObserve for visualization.

Overview

Monitoring Kafka is essential for ensuring performance, reliability, and data integrity.With OpenTelemetry and OpenObserve, you gain real-time visibility into Kafka clusters, topics, consumers, and producers.

Steps to Integrate

Prerequisites
Step 1: Create Test Kafka Topics (Optional)

  1. Create test topics to simulate workloads: 

    bin/kafka-topics.sh --create --topic quickstart-events-1 --partitions 3 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic quickstart-events-2 --partitions 10 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic openobserve --partitions 2 --bootstrap-server localhost:9092

  2. List topics to confirm: 

    bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 2: Install OpenTelemetry Collector Contrib

Download and install the latest otelcol-contrib.

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.116.1/otelcol-contrib_0.116.1_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.116.1_linux_amd64.deb

Note: Replace version numbers in download links if a newer release is available. Always check the GitHub Releases page.

Step 3: Get OpenObserve URL and Access Key
  1. In OpenObserve: go to Data Sources → Custom → Metrics → Otel Collector
  2. Copy the ingestion URL and Access Key Get OpenObserve Ingestion URL and Access Key
Step 4: Configure OpenTelemetry Collector
  1. Edit the configuration file: 
    sudo vi /etc/otelcol-contrib/config.yaml

  2. Update config for Kafka metrics:

    receivers:
kafkametrics:
    brokers: localhost:9092
    protocol_version: 2.0.0
    scrapers:
    - brokers
    - topics
    - consumers

exporters:
otlphttp/openobserve:
    endpoint: OPENOBSERVE_ENDPOINT
    headers:
    Authorization: OPENOBSERVE_TOKEN
    stream-name: OPENOBSERVE_STREAM

service:
pipelines:
    metrics:
    receivers: [kafkametrics]
    exporters: [otlphttp/openobserve]

Replace the following with your OpenObserve details:

  • OPENOBSERVE_ENDPOINT → API endpoint (e.g., https://api.openobserve.ai)
  • OPENOBSERVE_TOKEN → API token
  • OPENOBSERVE_STREAM → Stream name
Step 5: Start OpenTelemetry Collector
  1. Start and check status: 
    sudo systemctl start otelcol-contrib
sudo systemctl status otelcol-contrib
  2. Check logs: 
    journalctl -u otelcol-contrib --no-pager -n 50
Step 6: Visualize in OpenObserve
  1. Go to Streams → select metrics. Expand on any metric stream to see the events. Visualize in OpenObserve

Tip

You can use prebuilt dashboards to quickly get started with Kafka monitoring. Browse and import dashboards from: OpenObserve Community Dashboards

Troubleshooting

  • Collector Fails to Start: failed to build pipelines: unknown receiver kafkametrics
    • Cause: Using an older otelcol-contrib version that doesn’t support the kafkametrics receiver.
    • Fix: Upgrade to at least v0.116.0 or newer from GitHub Releases.
  • No Metrics Visible in OpenObserve
    • Double-check values in your config.yaml against your OpenObserve → Data Sources → Custom → Metrics → Otel Collector page.
  • Kafka Connection Errors: dial tcp 127.0.0.1:9092: connect: connection refused
    • Cause: Kafka broker not running or wrong address.
    • Fix: Ensure Kafka is started and reachable. Run: 
      bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092