# Kubernetes Monitoring Best Practices: A 2026 Checklist

> Kubernetes monitoring best practices for 2026: what to monitor in a cluster, which metrics to alert on, how to cut alert noise, the dashboards to start with, and a checklist you can work through.

Source: https://openobserve.ai/blog/kubernetes-monitoring-best-practices/
Published: 2026-11-18
Authors: Simran Kumari
Category: Engineering
Tags: OpenObserve, Kubernetes, Monitoring, Alerts, Dashboards, SRE

---

**Kubernetes monitoring best practices** are the habits that let you spot problems in a cluster before your users do: collecting the right data from every layer, alerting only on problems that need a person, and keeping logs, metrics, and traces connected so you can find the cause quickly. This guide explains each practice and ends with a checklist you can work through.

## TL;DR

- **Kubernetes monitoring works best when you watch four layers**: the nodes, the control plane, the workloads running on them, and the applications inside those workloads.
- **The most useful Kubernetes metrics are the ones that expose hidden failures**, such as containers that keep restarting or run out of memory, because Kubernetes often recovers from these quietly.
- **Good alerting means paging people only for problems that users feel or that won't fix themselves**, and showing everything else on dashboards.
- **OpenObserve is the best fit for Kubernetes monitoring**, because it brings logs, metrics, traces, and Kubernetes events together with a cluster browser and alerts on any query. You can try it on [OpenObserve Cloud](https://cloud.openobserve.ai).

## What is Kubernetes monitoring?

Kubernetes monitoring is the practice of collecting and analysing data about a Kubernetes cluster and the applications running on it, so you can tell whether everything is healthy, find problems early, and understand their cause. It covers the machines in the cluster (nodes), the system that manages them (the control plane), the containers and pods that run your software, and the applications themselves.

## Why is Kubernetes monitoring different from traditional monitoring?

Kubernetes heals many failures on its own. If a container crashes, Kubernetes restarts it. If a node fails, it moves the pods somewhere else. If a copy of your app disappears, it starts a new one.

That's great for uptime, but it hides problems. A container can restart 50 times an hour while your Deployment still looks healthy, and a node can be running low on memory long before anything visibly breaks. Pods also come and go constantly, so you can't rely on fixed hostnames the way you would with traditional servers. Good Kubernetes monitoring is built to catch these quiet failures.

## What should you monitor in Kubernetes?

Think of a cluster as four layers, each with its own data sources and failure modes:

<div class="not-prose my-8" style="overflow-x:auto">
<svg width="100%" style="min-width:640px" viewBox="0 0 820 250" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Four Kubernetes monitoring layers. Nodes: CPU, memory, disk, node conditions. Control plane: API server errors and latency, certificates. Workloads: restarts, OOMKilled, CrashLoopBackOff, pending pods, replicas. Applications: request rate, errors, latency, traces.">
<title>Four Kubernetes monitoring layers. Nodes: CPU, memory, disk, node conditions. Control plane: API server errors and latency, certificates. Workloads: restarts, OOMKilled, CrashLoopBackOff, pending pods, replicas. Applications: request rate, errors, latency, traces.</title>
<rect x="0.5" y="0.5" width="819" height="249" rx="12" fill="#ffffff" stroke="#E5E7EB"/>
<text x="40" y="36" font-family="Inter, sans-serif" font-size="15" font-weight="600" fill="#1E1E1E" text-anchor="start">The four layers to monitor in Kubernetes</text>
<rect x="40" y="60" width="170" height="152" rx="10" fill="#F9FAFB" stroke="#16A34A" stroke-width="1.5"/>
<rect x="40" y="60" width="170" height="6" rx="3" fill="#16A34A"/>
<text x="56.0" y="92" font-family="Inter, sans-serif" font-size="14" font-weight="600" fill="#1E1E1E" text-anchor="start">Nodes</text>
<text x="56.0" y="110" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#6B7280" text-anchor="start">kubelet, cAdvisor</text>
<circle cx="62" cy="132" r="3.5" fill="#16A34A"/>
<text x="74.0" y="136" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">CPU, memory, disk</text>
<circle cx="62" cy="156" r="3.5" fill="#16A34A"/>
<text x="74.0" y="160" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">NotReady, pressure</text>
<circle cx="62" cy="180" r="3.5" fill="#16A34A"/>
<text x="74.0" y="184" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">Allocatable capacity</text>
<rect x="230" y="60" width="170" height="152" rx="10" fill="#F9FAFB" stroke="#EA580C" stroke-width="1.5"/>
<rect x="230" y="60" width="170" height="6" rx="3" fill="#EA580C"/>
<text x="246.0" y="92" font-family="Inter, sans-serif" font-size="14" font-weight="600" fill="#1E1E1E" text-anchor="start">Control plane</text>
<text x="246.0" y="110" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#6B7280" text-anchor="start">API server</text>
<circle cx="252" cy="132" r="3.5" fill="#EA580C"/>
<text x="264.0" y="136" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">API errors</text>
<circle cx="252" cy="156" r="3.5" fill="#EA580C"/>
<text x="264.0" y="160" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">API latency</text>
<circle cx="252" cy="180" r="3.5" fill="#EA580C"/>
<text x="264.0" y="184" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">Certificate expiry</text>
<rect x="420" y="60" width="170" height="152" rx="10" fill="#F9FAFB" stroke="#7C3AED" stroke-width="1.5"/>
<rect x="420" y="60" width="170" height="6" rx="3" fill="#7C3AED"/>
<text x="436.0" y="92" font-family="Inter, sans-serif" font-size="14" font-weight="600" fill="#1E1E1E" text-anchor="start">Workloads</text>
<text x="436.0" y="110" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#6B7280" text-anchor="start">kube-state-metrics</text>
<circle cx="442" cy="132" r="3.5" fill="#7C3AED"/>
<text x="454.0" y="136" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">Restarts, OOMKilled</text>
<circle cx="442" cy="156" r="3.5" fill="#7C3AED"/>
<text x="454.0" y="160" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">CrashLoopBackOff</text>
<circle cx="442" cy="180" r="3.5" fill="#7C3AED"/>
<text x="454.0" y="184" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">Replicas vs desired</text>
<rect x="610" y="60" width="170" height="152" rx="10" fill="#F9FAFB" stroke="#3B82F6" stroke-width="1.5"/>
<rect x="610" y="60" width="170" height="6" rx="3" fill="#3B82F6"/>
<text x="626.0" y="92" font-family="Inter, sans-serif" font-size="14" font-weight="600" fill="#1E1E1E" text-anchor="start">Applications</text>
<text x="626.0" y="110" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#6B7280" text-anchor="start">OpenTelemetry</text>
<circle cx="632" cy="132" r="3.5" fill="#3B82F6"/>
<text x="644.0" y="136" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">Request rate</text>
<circle cx="632" cy="156" r="3.5" fill="#3B82F6"/>
<text x="644.0" y="160" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">Errors, latency</text>
<circle cx="632" cy="180" r="3.5" fill="#3B82F6"/>
<text x="644.0" y="184" font-family="Inter, sans-serif" font-size="12" font-weight="400" fill="#1E1E1E" text-anchor="start">Traces and logs</text>
</svg>
</div>

- **Nodes:** the machines that run your pods. Watch CPU, memory, and disk use, and whether a node reports itself as not ready or short on memory or disk.
- **Control plane:** the Kubernetes components that schedule and manage everything. If you run your own, watch the API server for errors and slow responses.
- **Workloads:** your pods and containers. Watch restarts, containers killed for running out of memory, pods that can't start, and Deployments running fewer copies than they should.
- **Applications:** the services your users call. Watch how many requests arrive, how many fail, and how long they take.

## 10 Kubernetes monitoring best practices

### 1. Collect data from every layer, not just nodes

Node CPU and memory graphs only tell part of the story. Collect cluster state (from kube-state-metrics), container resource usage (from the kubelet and cAdvisor), container logs, Kubernetes events, and application telemetry. Missing any one layer leaves a blind spot exactly where problems tend to start.

### 2. Use the golden signals for every service

For each service, track the four golden signals: latency (how long requests take), traffic (how many requests arrive), errors (how many fail), and saturation (how close you are to running out of resources). They tell you whether users are actually affected, which is the question that matters during an incident.

### 3. Watch for the failures Kubernetes hides

Because Kubernetes recovers automatically, track the signs that something keeps going wrong:

- **Restarts:** a rising `kube_pod_container_status_restarts_total` for a container.
- **OOMKilled containers:** containers killed because they used more memory than their limit.
- **CrashLoopBackOff and ImagePullBackOff:** containers stuck failing to start, or unable to download their image.
- **Pending pods:** pods that can't be scheduled, usually because the cluster has no room or a rule prevents placement.
- **Missing replicas:** a Deployment or StatefulSet with fewer available copies than it asked for.

### 4. Compare resource usage with requests and limits

In Kubernetes, CPU and memory problems look different from on a regular server. A container that reaches its memory limit is killed. A container that reaches its CPU limit is slowed down (throttled), which shows up as slow responses rather than errors. Track memory use (`container_memory_working_set_bytes`) and CPU throttling (`container_cpu_cfs_throttled_periods_total`) against each container's requests and limits, so you can set limits that fit real usage.

### 5. Collect Kubernetes events alongside logs

Events record what Kubernetes itself is doing: scheduling decisions, image download failures, evictions, and failed health checks. These rarely appear in your application logs, but they often explain why a pod isn't running. Store them with your logs so they're searchable during an incident.

### 6. Label everything consistently

Attach the same fields to every signal: cluster, namespace, workload, pod, and node. Consistent labels let you move from a slow service on a dashboard to that service's logs and traces in one step, and let you group data by team or environment.

### 7. Alert on symptoms, not on every metric

The most common Kubernetes monitoring mistake is alerting on everything. Page someone only when users are affected or when Kubernetes can't recover on its own. Everything else belongs on a dashboard or in a low-priority channel. The next section lists which alerts fall into which group.

### 8. Use SLOs for your most important services

A service level objective (SLO) sets a reliability target, such as 99.9% of checkout requests succeeding. Alerting when you're using up your error budget too quickly catches real user impact and ignores noise. Our guide to [SLO-based alerting](/blog/slo-based-alerting/) shows how to set this up.

### 9. Keep logs, metrics, and traces in one place

Metrics tell you that something is wrong; logs and traces tell you why. When all three live in the same tool with the same labels, you can go from "errors are up in the payments namespace" to the failing request and its log lines without switching tools.

### 10. Control what you keep and for how long

Kubernetes clusters produce a lot of logs, especially at debug level. Set shorter retention for noisy, low-value streams and longer retention for audit and error logs, so your monitoring stays affordable as the cluster grows.

## Which Kubernetes metrics should you alert on?

This table separates problems that need a person now from those that can wait:

| Problem | What to do | Why |
|---|---|---|
| A service is burning through its error budget | Page someone | Users are affected right now |
| A container has been crash-looping for over 10 minutes | Page someone | It isn't recovering on its own |
| A node has been not ready for over 5 minutes | Page someone | The cluster has lost capacity |
| A Deployment has had fewer available copies than desired for over 10 minutes | Page someone | Reduced capacity that isn't fixing itself |
| Pods have been pending for over 10 minutes | Create a ticket | The cluster or its quotas need attention |
| A node's disk is over 85% full | Create a ticket | Kubernetes will start evicting pods soon |
| A single container restarted once | Dashboard only | Kubernetes already handled it |
| A brief CPU spike | Dashboard only | Not a problem unless it slows requests |

The time thresholds are starting points; adjust them to match how your workloads behave.

## How do you reduce alert fatigue in Kubernetes?

Alert fatigue happens when people get so many alerts that they start ignoring them. To avoid it:

1. **Page only for user impact or problems that won't self-heal**, as in the table above.
2. **Wait before alerting.** Require a condition to last a few minutes so brief blips during deployments don't page anyone.
3. **Group related alerts.** One alert per condition across all namespaces is easier to handle than one per namespace.
4. **Combine conditions.** Alert when errors are high *and* latency is high, instead of on each separately.
5. **Review alerts regularly.** If an alert never leads to action, delete it or move it to a dashboard.

## What dashboards do you need for Kubernetes monitoring?

Start with four dashboards and add more when an incident shows you were missing something:

1. **Cluster overview:** how many nodes are ready, how many pods are running, and node CPU and memory use.
2. **Workload health:** restarts, containers killed for running out of memory, and available versus desired copies, by namespace.
3. **Service health:** latency, traffic, errors, and saturation for each service.
4. **Capacity:** how much CPU and memory pods request compared with what each node can provide.

![OpenObserve Kubernetes overview dashboard with healthy scrape targets, ready nodes, running pods, node vCPUs, node CPU and memory usage, and running pods and container memory by namespace](/assets/blog/kube-prometheus-stack-guide/openobserve-kubernetes-dashboard.png)

You don't have to build these from scratch: see [how to import prebuilt Kubernetes dashboards](/blog/import-prebuilt-k8s-dashboards/) into OpenObserve.

When a dashboard shows a problem, the next step is usually the logs. For example, this query counts error logs by namespace to show where problems are concentrated:

![OpenObserve Logs SQL query counting error logs by Kubernetes namespace from a k8s_logs stream](/assets/blog/monitor-kubernetes-logs-at-scale/query-errors-by-namespace.png)

## What are the best Kubernetes monitoring tools?

Most teams choose between three approaches:

- **A single observability platform** that collects logs, metrics, traces, and events together, such as OpenObserve. This is the simplest to run and makes it easiest to connect signals during an incident.
- **A Prometheus-based stack**, usually Prometheus for metrics and Grafana for dashboards, with separate tools for logs and traces. Our [kube-prometheus-stack guide](/blog/kube-prometheus-stack-guide/) covers that approach.
- **A commercial SaaS tool**, typically priced per node or host, which can become expensive as clusters autoscale.

For a detailed comparison of individual tools, see the [top 10 Kubernetes monitoring tools](/blog/top-10-k8s-monitoring-tools/).

## How OpenObserve helps with Kubernetes monitoring

OpenObserve is an open-source (AGPL-3.0) observability platform. The [OpenObserve Collector for Kubernetes](https://openobserve.ai/docs/integration/system/k8s/) gathers container logs, node and container metrics, Kubernetes events, and traces, with the cluster, namespace, and pod attached to every record. Under **Infra**, the Kubernetes cluster browser lists every node, pod, workload, and event, with an interactive cluster map you can colour by CPU, memory, restarts, or status.

It covers every practice in this guide:

- **Collection:** logs, metrics, traces, and events from one collector, or Prometheus remote write if you already run kube-prometheus-stack.
- **Queries:** PromQL for metrics and SQL across every signal.
- **Alerting and reliability:** scheduled, real-time, composite, and anomaly alerts, incident management, and SLOs with error budgets and burn rates.
- **Dashboards and reports:** prebuilt Kubernetes dashboards and scheduled PDF, PNG, or CSV reports.
- **AI:** AI Assistant, SRE Agent for AI root cause analysis, an MCP server, LLM observability and evaluations, anomaly detection, log patterns, and insights.
- **Frontend and synthetics:** RUM, session replay, error tracking, and scheduled browser and protocol checks for the services your cluster runs.
- **Data processing:** pipelines, VRL functions, enrichment tables, and metrics downsampling.
- **Storage and cost:** Parquet on object storage, Bring Your Own Bucket, retention per stream, extended retention, and ingest-based pricing with no per-node or per-user fees, so autoscaling doesn't raise your licence cost.
- **Security and governance:** SSO, RBAC, multi-tenancy, sensitive data redaction, audit trail, cipher keys, and federated search across clusters.
- **Compliance:** SOC 2 Type II, ISO 27001, and GDPR via the EU-Central 1 (Frankfurt) region.
- **Open standards:** OpenTelemetry-native ingestion and PromQL and SQL querying.

## Kubernetes monitoring checklist

Use this as a quick review of your setup:

- [ ] Node CPU, memory, disk, and node conditions are collected for every node.
- [ ] Cluster state from kube-state-metrics is collected.
- [ ] Container CPU throttling and memory use are compared with requests and limits.
- [ ] Container logs and Kubernetes events are stored together and searchable.
- [ ] Every service reports latency, traffic, and errors.
- [ ] All signals share cluster, namespace, workload, pod, and node labels.
- [ ] Restarts, OOMKilled containers, pending pods, and missing replicas are tracked.
- [ ] People are paged only for user impact or problems that won't self-heal.
- [ ] Critical services have SLOs with burn-rate alerts.
- [ ] Cluster overview, workload health, service health, and capacity dashboards exist.
- [ ] Retention is set per log stream.

## Get started with OpenObserve

- Start a free trial of [OpenObserve Cloud](https://cloud.openobserve.ai) and install the [OpenObserve Collector](https://openobserve.ai/docs/integration/system/k8s/) on your cluster.
- Already running kube-prometheus-stack? [Send its metrics to OpenObserve](/blog/send-metrics-using-kube-prometheus-stack-to-openobserve/).
- Related reading: [monitoring Kubernetes logs at scale](/blog/monitor-kubernetes-logs-at-scale/) and [monitoring Kubernetes with OpenTelemetry](/blog/monitoring-kubernetes-with-opentelemetry/).
