How to Monitor Apache Airflow Logs and Metrics Using OpenTelemetry
Chaitanya Sistla
January 22, 2025
4 min read
Don’t forget to share!
Apache Airflow is a powerful workflow automation tool used for orchestrating complex data pipelines. It enables users to define, schedule, and monitor workflows as directed acyclic graphs (DAGs). However, monitoring Airflow logs and metrics is crucial to ensure the reliability and performance of workflows.
In this guide, we will explore how to monitor Airflow logs and metrics using OpenTelemetry (OTel) and OpenObserve. We will configure Airflow to emit telemetry data, install the OpenTelemetry Collector, and visualize logs and metrics in OpenObserve.
What is Apache Airflow and Why is it Used?
Apache Airflow is an open-source workflow automation tool that allows users to:
- Define workflows as code using Python.
- Schedule, monitor, and manage task execution.
- Automate ETL (Extract, Transform, Load) processes, machine learning pipelines, and data processing workflows.
- Integrate with a variety of services like AWS, GCP, databases, and messaging systems.
Key Benefits of Airflow
- Extensible and modular architecture.
- Task dependencies and retries are automatically managed.
- Supports parallel execution and scaling.
- Rich UI for monitoring and debugging workflows.
How to Monitor Airflow Logs and Metrics?
Monitoring Apache Airflow involves collecting:
- Logs: Capturing Airflow logs from different components such as the scheduler, workers, and DAG execution.
- Metrics: Gathering telemetry data such as task execution times, DAG runs, queue latencies, and resource utilization.
To achieve this, we configure Airflow to send logs and metrics to an OpenTelemetry (OTel) collector, which forwards them to OpenObserve for visualization.
Step 1: Configure airflow.cfg for OpenTelemetry
Edit the airflow.cfg file to enable OpenTelemetry for Airflow metrics.
[metrics]
otel_on = True
otel_host = localhost
otel_port = 4318
This ensures that Airflow emits OpenTelemetry metrics to an OpenTelemetry Collector running on localhost:4318.
Restart Airflow services after making these changes:
airflow db migrate
airflow scheduler -D
airflow webserver -D
Step 2: Install OpenTelemetry Collector
The OpenTelemetry Collector is responsible for receiving logs and metrics from Airflow and forwarding them to OpenObserve.
Install OpenTelemetry Collector
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/otelcol-linux-amd64
chmod +x otelcol-linux-amd64
sudo mv otelcol-linux-amd64 /usr/local/bin/otelcol
Verify the installation:
otelcol --version
Step 3: Configure OpenTelemetry Collector for Airflow
Create an OpenTelemetry Collector configuration file otel-config.yaml:
receivers:
filelog/std:
include:
- /airflow/logs/*/*.log # change the path
- /airflow/logs/scheduler/*/*/*/*.log # change the path
start_at: beginning
otlp:
protocols:
grpc:
http:
processors:
batch:
extensions:
zpages: {}
exporters:
otlphttp/openobserve:
endpoint: https://example.com/api/default/ # add these configurations from O2 data source section
headers:
Authorization: "Basic cdcfefgqg=" # add these configurations from O2 data source section
stream-name: blog_airflow # you can set this anything
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/openobserve]
logs:
receivers: [filelog/std, otlp]
processors: [batch]
exporters: [otlphttp/openobserve]
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/openobserve]
This configuration does the following:
- You can find auth details by following these steps
- Captures logs from
/airflow/logs/and/airflow/logs/scheduler/. - Collects logs, metrics, and traces using the
otlpreceiver. - Sends data to OpenObserve with a custom stream name (
blog_airflow).
Start the OpenTelemetry Collector with this configuration:
systemctl start otel-collector
To check if it's running:
systemctl status otel-collector
Step 4: Logs Ingested from Airflow
Once the OpenTelemetry Collector is running, it starts ingesting logs from:
- DAG execution logs.
- Scheduler logs.
- Worker logs.
- Task execution logs.
To view logs in OpenObserve, navigate to the logs section and select blog_query:
Step 5: Airflow Metrics Dashboards
Pre-built dashboards for Airflow are available at: OpenObserve Airflow Dashboards
These dashboards provide:
- DAG execution times.
- Task latency monitoring.
- Scheduler performance insights.
- Worker resource utilization.
To import a dashboard into OpenObserve:
- Download the JSON file from the GitHub link.
- Go to OpenObserve > Dashboard > Import.
- Upload the JSON file and start visualizing your Airflow metrics.
Conclusion
Monitoring Airflow logs and metrics is crucial for maintaining workflow health and debugging failures. By integrating OpenTelemetry and OpenObserve, you can:
- Collect logs and metrics from Airflow.
- Visualize Airflow performance with dashboards.
- Detect issues in DAG execution in real-time.
This setup provides a scalable observability solution for Apache Airflow, ensuring efficient workflow automation and monitoring.
