# NVIDIA GPU Monitoring with DCGM Exporter and OpenObserve: Complete Setup Guide
> Monitor NVIDIA H100, H200, and A100 GPUs with DCGM Exporter and OpenObserve. Complete setup guide with dashboards, alerts, and 89% cost savings vs traditional tools.
Source: https://openobserve.ai/blog/how-to-monitor-nvidia-gpu/
Published: 2025-11-06
Authors: Chaitanya Sistla
Category: Engineering
Tags: AI, Observability, Alerts
---
AI-driven infrastructure landscape is evolving and GPU clusters represent one of the most significant capital investments for organizations. Whether you're running large language models, training deep learning models, or processing massive datasets, your NVIDIA GPUs (H100s, H200s, A100s, or L40S) are the workhorses powering your most critical workloads.
But here's the challenge: **how do you know if your GPU infrastructure is performing optimally?**
Traditional monitoring approaches fall short when it comes to GPU infrastructure. System metrics like CPU and memory utilization don't tell you if your GPUs are thermal throttling, experiencing memory bottlenecks, or operating at peak efficiency. You need deep visibility into GPU-specific metrics like utilization, temperature, power consumption, memory usage, and PCIe throughput.
This is where **NVIDIA's Data Center GPU Manager (DCGM) Exporter** combined with **OpenObserve** creates a powerful, cost-effective monitoring solution that gives you real-time insights into your GPU infrastructure.
## Why GPU Monitoring Matters
### The High Cost of GPU Inefficiency
Consider this scenario: You're running an 8x NVIDIA H200 cluster. Each H200 costs approximately $30,000-$40,000, meaning your hardware investment alone is around $240,000-$320,000. Operating costs (power, cooling, infrastructure) can easily add another $50,000-$100,000 annually.
Now imagine:
- **Thermal throttling** reducing performance by 15% due to poor cooling
- **GPU memory leaks** causing jobs to fail silently
- **Underutilization** with GPUs sitting idle 40% of the time
- **Hardware failures** going undetected until complete outage
- **PCIe bottlenecks** limiting data transfer rates
Without proper monitoring, you're flying blind. You might be:
- **Wasting $50,000+ annually** on inefficient GPU utilization
- **Missing critical performance degradation** before it impacts production
- **Unable to justify ROI** on GPU infrastructure to stakeholders
- **Lacking data** for capacity planning and optimization decisions
### What You Need to Monitor
Effective GPU monitoring requires tracking dozens of metrics across multiple dimensions:
**Performance Metrics:**
- GPU compute utilization (%)
- Memory bandwidth utilization (%)
- Tensor Core utilization
- SM (Streaming Multiprocessor) occupancy
**Thermal & Power:**
- GPU temperature (°C)
- Power consumption (W)
- Power limit throttling events
- Thermal throttling events
**Memory:**
- GPU memory usage (MB/GB)
- Memory allocation failures
- ECC (Error Correction Code) errors
- Memory clock speeds
**Interconnect:**
- PCIe throughput (TX/RX)
- NVLink bandwidth
- NVSwitch fabric health
- Data transfer bottlenecks
**Health & Reliability:**
- XID errors (hardware faults)
- Page retirement events
- GPU compute capability
- Driver version compliance
## The Solution: DCGM Exporter + OpenObserve
### What is DCGM Exporter?
NVIDIA's Data Center GPU Manager (DCGM) is a suite of tools for managing and monitoring NVIDIA datacenter GPUs. DCGM Exporter exposes GPU metrics in Prometheus format, making it easy to integrate with modern observability platforms.
You can find more details about DCGM exporter here.
**Key capabilities:**
- Exposes 40+ GPU metrics per device
- Supports all modern NVIDIA datacenter GPUs (A100, H100, H200, L40S)
- Low overhead monitoring (~1% GPU utilization)
- Works with Docker, Kubernetes, and bare metal
- Handles multi-GPU and multi-node deployments
- Provides health diagnostics and error detection
## Complete Setup Guide
### Prerequisites
Before starting, ensure you have:
- GPU-enabled server (cloud or on-premises)
- NVIDIA GPUs installed and recognized by the system
- NVIDIA drivers version 535+ (550+ recommended for H200)
- Docker installed and configured with NVIDIA Container Toolkit
- OpenObserve instance (cloud or self-hosted)
### Step 1: Verify GPU Detection
First, confirm your GPUs are properly detected by the system:
```bash
# Check if GPUs are visible
nvidia-smi
# Expected output: List of GPUs with utilization, temperature, and memory
```
For NVIDIA H200 or multi-GPU systems with NVSwitch, you'll need the NVIDIA Fabric Manager:
```bash
# Install fabric manager (version should match your driver)
sudo apt update
sudo apt install -y nvidia-driver-535 nvidia-fabricmanager-535
# Reboot to load new driver
sudo reboot
# After reboot, start the service
sudo systemctl start nvidia-fabricmanager
sudo systemctl enable nvidia-fabricmanager
# Verify
nvidia-smi # Should now show all GPUs
```
### Step 2: Deploy DCGM Exporter
Deploy DCGM Exporter as a Docker container. This lightweight container exposes GPU metrics on port 9400:
```bash
docker run -d \
--gpus all \
--cap-add SYS_ADMIN \
--network host \
--name dcgm-exporter \
--restart unless-stopped \
nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04
```
**Configuration breakdown:**
- `--gpus all` - Grants access to all GPUs on the host
- `--cap-add SYS_ADMIN` - Required for DCGM to query GPU metrics
- `--network host` - Uses host networking for easier access
- `--restart unless-stopped` - Ensures resilience across reboots
**Verify DCGM is working:**
```bash
# Wait 10 seconds for initialization
sleep 10
# Access metrics from inside the container
docker exec dcgm-exporter curl -s http://localhost:9400/metrics | head -30
# You should see output like:
# DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-xxxx",...} 45.0
# DCGM_FI_DEV_GPU_TEMP{gpu="0",UUID="GPU-xxxx",...} 42.0
```
### Step 3: Configure OpenTelemetry Collector
The OpenTelemetry Collector scrapes metrics from DCGM Exporter and forwards them to OpenObserve. Create the configuration:
```bash
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'dcgm-gpu-metrics'
scrape_interval: 30s
static_configs:
- targets: ['localhost:9400']
metric_relabel_configs:
# Keep only DCGM metrics
- source_labels: [__name__]
regex: 'DCGM_.*'
action: keep
exporters:
otlphttp/openobserve:
endpoint: https://example.openobserve.ai/api/ORG_NAME/
headers:
Authorization: "Basic YOUR_O2_TOKEN"
processors:
batch:
timeout: 10s
send_batch_size: 1024
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [otlphttp/openobserve]
```
**Get your OpenObserve credentials:**
```bash
# For Ingestion token authentication (recommended):
Go to OpenObserve UI → Datasources -> Custom -> Otel Collector
```
Update the `Authorization` header in the config with your base64-encoded credentials.
### Step 4: Deploy OpenTelemetry Collector
```bash
docker run -d \
--network host \
-v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
--name otel-collector \
--restart unless-stopped \
otel/opentelemetry-collector-contrib:latest \
--config=/etc/otel-collector-config.yaml
```
**Check OpenTelemetry Collector:**
```bash
# View collector logs
docker logs otel-collector
# Look for successful scrapes (no error messages)
```
**Check OpenObserve:**
1. Log into OpenObserve UI
2. Navigate to **Metrics** section
3. Search for metrics starting with `DCGM_`
4. Data should appear within 1-2 minutes
### Step 5: Generate GPU Load (Optional)
To verify monitoring is working, generate some GPU activity:
```bash
# Install PyTorch
pip3 install torch
# Create a load test script
cat > gpu_load.py <<'EOF'
import torch
import time
print("Starting GPU load test...")
devices = [torch.device(f'cuda:{i}') for i in range(torch.cuda.device_count())]
tensors = [torch.randn(15000, 15000, device=d) for d in devices]
print(f"Loaded {len(devices)} GPUs")
while True:
for tensor in tensors:
_ = torch.mm(tensor, tensor)
time.sleep(0.5)
EOF
# Run load test
python3 gpu_load.py
```
Watch your metrics in OpenObserve - you should see GPU utilization spike!
## Creating Dashboards in OpenObserve
1. Download the Dashboards from our community repository.
2. **In OpenObserve UI, go to Dashboards → Import -> Drop your files here -> select your json -> Import**
3. Once the dashboard has been imported, you will see the below metrics that were prebuilt and you can always customize the dashboards as needed.
## Setting Up Alerts
Critical alerts to configure in OpenObserve:
### 1. High GPU Temperature
```promql
DCGM_FI_DEV_GPU_TEMP > 85
```
**Severity:** Warning at 85°C, Critical at 90°C
**Action:** Check cooling systems, reduce workload
### 2. GPU Memory Near Capacity
```promql
(DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) > 0.90
```
**Severity:** Warning at 90%, Critical at 95%
**Action:** Optimize memory usage or scale horizontally
### 3. Low GPU Utilization (Waste Detection)
```promql
avg(DCGM_FI_DEV_GPU_UTIL) < 20
```
**Duration:** For 30 minutes
**Action:** Review workload scheduling, consider rightsizing
### 4. GPU Hardware Errors
```promql
increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
```
**Severity:** Critical
**Action:** Immediate investigation, potential RMA
### 5. Thermal Throttling Detected
```promql
increase(DCGM_FI_DEV_THERMAL_VIOLATION[5m]) > 0
```
**Severity:** Warning
**Action:** Improve cooling or reduce ambient temperature
### 6. GPU Offline
```promql
absent(DCGM_FI_DEV_GPU_TEMP)
```
**Duration:** For 2 minutes
**Action:** Check GPU health, driver status, fabric manager
## Traditional Monitoring vs. GPU Monitoring with OpenObserve
| Aspect | Traditional Monitoring (Prometheus/Grafana) | OpenObserve for GPU Monitoring |
|--------|-------------------------------------------|-------------------------------|
| **Setup Complexity** | Requires Prometheus, node exporters, [Grafana](/grafana-alternative/), storage backend, and complex configuration | Single unified platform with built-in visualization |
| **Storage Costs** | High - Prometheus stores all metrics at full resolution, requires expensive SSD storage | **80% lower** - Advanced compression and columnar storage |
| **Multi-tenancy** | Complex setup requiring multiple Prometheus instances or federation | Built-in with organization isolation and access controls |
| **Alerting** | Separate alerting system (Alertmanager), complex routing configuration | Integrated alerting with flexible notification channels |
| **Long-term Retention** | Expensive - requires additional tools like Thanos or Cortex | Native long-term storage with automatic data lifecycle management |
| **GPU-Specific Features** | Generic time-series database, not optimized for GPU metrics | Optimized for high-cardinality workloads like GPU monitoring |
| **Log Correlation** | Separate log management system needed (ELK, Loki) | **Unified logs, metrics, and traces** in one platform |
| **Setup Time** | 4-8 hours (multiple components, configurations, troubleshooting) | **30 minutes** (end-to-end) |
| **Maintenance Overhead** | High - multiple systems to update, monitor, and troubleshoot | Low - single platform with automatic updates |
### ROI Examples
For an 8-GPU H200 cluster worth $320,000:
**Detect thermal throttling early:**
- 15% performance loss = $48,000 annual waste
- Early detection saves this loss
- **ROI: 990% in first year**
**Optimize utilization:**
- Increase from 40% to 70% = 75% more work
- Defer $240,000 expansion by 1 year
- **ROI: 4,900% in first year**
**Prevent downtime:**
- 1 hour downtime = $2,800 revenue loss
- Preventing 5 hours/year = $14,000 saved
- **ROI: 289% in first year**
## Conclusion
GPU monitoring is no longer optional-it's essential infrastructure for any organization running GPU workloads. The combination of DCGM Exporter and OpenObserve provides:
✅ **Complete visibility** into GPU health, performance, and utilization
✅ **Cost optimization** through identifying waste and inefficiencies
✅ **Proactive alerting** to prevent outages and degradation
✅ **Data-driven decisions** for capacity planning and architecture
✅ **89% lower TCO** compared to traditional monitoring stacks
✅ **30-minute setup** vs. days with traditional tools
Whether you're running AI/ML workloads, rendering farms, scientific computing, or GPU-accelerated databases, this monitoring solution delivers immediate ROI while scaling effortlessly as your infrastructure grows.
### Resources
- **DCGM Exporter:** github.com/NVIDIA/dcgm-exporter
- **OpenObserve:** [openobserve.ai](https://openobserve.ai/)
- **OpenObserve Docs:** [openobserve.ai/docs](https://openobserve.ai/docs/)
- **OpenTelemetry Collector:** opentelemetry.io/docs/collector
> #### Get Started with OpenObserve Today!
> Sign up for a [14 day trial](https://cloud.openobserve.ai)
>Check out our GitHub repository for self-hosting and contribution opportunities