Monitoring Kubernetes Cluster using OpenTelemetry Components

Introduction to Kubernetes Monitoring with OpenTelemetry

How do you keep an eye on an ever-growing digital landscape without losing sight of what matters? Kubernetes has changed the way we deploy and manage containerized applications and set a new bar for efficiency and scalability. But with great power comes great responsibility in the form of ‘monitoring’. As your clusters grow, monitoring everything on it can get overwhelming and hard to aggregate. That’s where the K8s Cluster Receiver comes in as a way to aggregate all the important cluster-level metrics and events. By using OpenTelemetry you can standardize your telemetry data collection and see the full picture of how your cluster is running in real-time for both efficiency and correctness.

What is OpenTelemetry?

Definition and Goals

OpenTelemetry is an open-source tool for producing and combining telemetry (metrics, logs, and traces) across diverse technologies and platforms, making observability simple to integrate and deploy.

Components of OpenTelemetry

OpenTelemetry has the following components:

APIs: Instrumentation interfaces
SDKs: Telemetry data collectors
Libraries: Instrumentation for specific languages
Integrations: Built-in components for common libraries and frameworks

What is an OpenTelemetry Collector

This component facilitates the exporting, processing, and aggregating of telemetry data and offers extreme flexibility and compatibility in different operational modes.

Now that you have a good grasp on what OpenTelemetry is and its components, let’s move on to getting it up and running in your Kubernetes cluster.

Installing and Configuring OpenTelemetry to Monitor a Kubernetes Cluster

Requirements

To install OpenTelemetry on a Kubernetes cluster, you should already have:

Kubernetes cluster running
Kubectl (command-line tool for Kubernetes operations) configured for your cluster
Helm (package manager for Kubernetes) installed

Installing and Configuring OpenTelemetry Collector

The OpenTelemetry Collector can be installed with Helm charts. The following example simplifies the installation procedure:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm install my-opentelemetry-collector open-telemetry/opentelemetry-collector

Daemonset vs. Deployment Installations

Daemonset: Runs a collector on each node in the cluster. This should be used if you want to collect data from each node in the cluster.
Deployment: Runs a user-defined amount of collectors. This should be used if you want to aggregate data from the collectors on a cluster level.

Now that you know how to install the OpenTelemetry Collector, let's dive into some key components that will boost your Kubernetes monitoring.

Main OpenTelemetry for Kubernetes Components

1. Kubernetes Attributes Processor

The Kubernetes Attributes Processor adds Kubernetes-related metadata to the telemetry data, like pod name, pod namespace, pod labels, etc. This provides better context to the data collected for later analysis.

2. Kubeletstats Receiver:

The Kubeletstats Receiver can be thought of as a postman gathering useful stats from the Kubelet API server. The Kubelet is a component of every node in a Kubernetes cluster and it manages pods and containers. This receiver collects useful metrics regarding CPU utilization, memory usage, etc. directly from the Kubelet. In essence, the Kubeletstats receiver “delivers'' a health report for each of your cluster nodes so that you are aware of how each node is performing.

3. Kubernetes Cluster Receiver

The K8s Cluster Receiver is where cluster-level metrics and events are aggregated. It collects node status, pod conditions, and cluster events so you can see the overall health of the cluster. Use the K8s Cluster Receiver to get visibility into your cluster and fix issues before they turn into bigger problems.

4. Kubernetes Objects Receiver

This receiver collects Kubernetes objects from the API server so you can see the state and changes of Kubernetes resources like deployments, services, and configuration maps.

5. Prometheus Receiver

Prometheus Receiver supports the Prometheus metrics format so you can ingest metrics from applications already instrumented with Prometheus.

6. Host Metrics Receiver

This component scrapes machine-level metrics from nodes so you can see the full system view of resource usage and performance.

7. Filelog Receiver

Filelog Receiver efficiently tails and parses log files so you can monitor and analyze your application logs.

Ready to put it all together? Let's get those OpenTelemetry components deployed and configured.

Deploying and Configuring OpenTelemetry Components

Deployment Guide

Deploying the OpenTelemetry Collector involves setting up either Daemonset or Deployment instances based on your monitoring needs. Here’s an example configuration for deploying a Daemonset:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector:latest
        ports:
        - containerPort: 55680
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/otel
          name: otel-config
      volumes:
      - name: otel-config
        configMap:
          name: otel-config

Configuring Key Components

Kubernetes Attributes Processor:

processors:
  k8sattributes:
    auth_type: serviceAccount

Kubeletstats Receiver:

receivers:
  kubeletstats:
    endpoint: "https://${KUBELET_HOST}:10250"
    auth_type: "kubeConfig"

Best Practices for Kubernetes RBAC

To collect data securely, configure Kubernetes Role-Based Access Control (RBAC) policies. Define roles and bindings that give OpenTelemetry components the required permissions without over-permission.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector-role
rules:
- apiGroups: [""]
  resources: ["nodes", "pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-collector-role
subjects:
- kind: ServiceAccount
  name: otel-collector
  namespace: default

Creating manifest files

Manifests are required to deploy OpenTelemetry components in your cluster. Manifests define what you want your applications and resources to be. Here’s a manifest for the OpenTelemetry Collector as a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 3
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector:latest
        ports:
        - containerPort: 55680
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/otel
          name: otel-config
      volumes:
      - name: otel-config
        configMap:
          name: otel-config

Great job getting everything set up! Now, let's turn to how you can monitor your cluster's performance and health in real time.

Monitoring Cluster Performance and Health

SigNoz Dashboard

SigNoz has a very simple dashboard for visual monitoring and alerting. It integrates with OpenTelemetry to show real-time cluster performance, node conditions, pod phases, and container restarts.

Collecting and analyzing metrics

Collect metrics for node conditions, pod phases, and container restarts to gauge your cluster health. This data can be used to find patterns and predict problems.

Check out ‘Using Prometheus APM Tools for Asset Performance Management’ from OpenObserve for reference.

How to Create Dashboards for Monitoring the Kubernetes Cluster?

Dashboards are essential for monitoring Kubernetes clusters as they provide a human-readable way to interpret the metrics. With OpenTelemetry metrics, traces, and logs, setting up a dashboard is a guided process. Here’s how you can set up dashboards:

Scope: Start by deciding what metrics you need. Common metrics are CPU usage, memory usage, and I/O operations.
Use Templates: You can use dashboard templates that can be customized as per your needs.
Alerts: Make sure your dashboards show current metrics and also forecast potential issues using predictive analysis. Set up alerts for abnormal patterns or when thresholds are crossed.
Regular Updates: As your cluster changes, so should your dashboards. Update your setups as per the changing needs and metrics of your Kubernetes cluster.

By following these steps you can make sure your Kubernetes monitoring dashboard is complete and up-to-date which is essential for cluster health and performance.

Advanced Monitoring Techniques

Enhancing Telemetry Data

Use OpenTelemetry Processors to pull in telemetry data and add context so you can get better insights.

Authentication and Security Secure the Kubernetes API for telemetry by adding authenticating and encrypting data in transit.
Configure OpenTelemetry Customize OpenTelemetry to your monitoring needs. This might mean tweaking sampling rates, defining custom metrics, and adding in extra receivers and processors.

So, what do we do next?

Using OpenTelemetry for Kubernetes cluster monitoring has many benefits, including visibility, resource utilization, and proactive issue resolution. For more information check out the OpenTelemetry and Kubernetes documentation and try out some additional configurations and advanced components of the OpenTelemetry Collector to tune your monitoring.

By using the K8s Cluster Receiver and other OpenTelemetry components you can have full monitoring and your Kubernetes clusters will run smoothly and efficiently. The K8s Cluster Receiver is super useful for high observability, so you can see all the important metrics and events in your cluster.

Resources