Installing Kubelet Stats Receiver with OpenTelemetry
Your Kubernetes cluster is a complex ecosystem teeming with data. To harness its full potential, you need powerful tools. The Kubelet Stats Receiver is a cornerstone of effective Kubernetes cluster monitoring.
The Kubelet Stats Receiver is a critical component within the OpenTelemetry Collector architecture for gathering metrics from Kubernetes nodes. It interacts directly with the Kubelet, a control plane agent that runs on each node.
Key Functions:
- Metric Collection: Periodically fetches metrics such as CPU, memory, network, and disk utilization from the Kubelet.
- Data Enrichment: Adds context to collected metrics, including pod and container information.
- Data Export: Sends enriched metrics to the OpenTelemetry Collector for further processing and analysis.
Integration with OpenTelemetry:
The Kubelet Stats Receiver is integrated with the OpenTelemetry Collector, which is responsible for receiving, processing, and exporting telemetry data. The collector can then forward the metrics to various backend systems for storage and visualization.
Importance of Kubelet Stats Receiver:
- Real-time monitoring: Provides up-to-date insights into cluster health and resource utilization.
- Capacity planning: Helps in predicting future resource needs.
- Performance optimization: Identifies bottlenecks and optimization opportunities.
- Anomaly detection: Detects abnormal resource usage patterns.
By effectively utilizing the Kubelet Stats Receiver, organizations can gain valuable insights into their Kubernetes clusters and make data-driven decisions. Let’s get started by understanding the deployment and configuration of OpenTelemetry collector.
Deployment and Configuration
To install the Kubelet Stats Receiver with OpenTelemetry in OpenObserve, follow these steps:
Deploying the OpenTelemetry Collector
The OpenTelemetry Collector can be deployed in various ways, including as a Docker container, using Docker Compose, or on Kubernetes. The key steps are:
- Pull the OpenTelemetry Collector Docker image:
|
Bash |
- Run the Collector container, mounting a custom configuration file:
|
Bash |
Configuring the OpenTelemetry Collector
In the OpenTelemetry Collector configuration, add the following receiver and exporter to collect Kubelet metrics:
|
Bash |
Replace https://openobserve.example.com with your OpenObserve cluster URL and your_api_key with a valid API key.
Verifying the Installation
After restarting the OpenTelemetry Collector, you should see Kubelet metrics in OpenObserve within a few minutes. You can verify this by running a query in the OpenObserve UI or using the API:
|
sql |
This query will return all Kubelet-related metrics being collected by OpenTelemetry and sent to OpenObserve.
By following these steps, you can successfully install the Kubelet Stats Receiver with OpenTelemetry and start collecting Kubelet metrics in your OpenObserve observability platform. To know more about how to use OpenObserve, check out the detailed documentation here.
Now, let’s understand the authentication and permissions required.
Authentication and Permissions
Here are the key points about authentication and permissions for the Kubelet Stats Receiver with OpenTelemetry in OpenObserve:
Service Account Authentication
- Use auth_type: "serviceAccount" in the receiver configuration
- Requires the default service account token to authenticate to the kubelet API
- The pod spec should set the node name using the downward API:
|
Bash |
- Then reference the K8S_NODE_NAME environment variable in the endpoint field:
|
Bash |
TLS Authentication
- Use auth_type: "tls" in the receiver configuration
- Requires setting ca_file, key_file, and cert_file fields
- Example configuration:
|
Bash |
Required Permissions
- The Kubelet Stats Receiver needs get permissions on the nodes/stats resources
- If using extra_metadata_labels or request/limit_utilization metrics, it also needs get permissions on nodes/proxy resources
- Example RBAC configuration:
|
Bash |
By following these authentication and permission guidelines, you can successfully configure the Kubelet Stats Receiver to collect metrics from the kubelet API in your OpenObserve setup. Let’s see a sample configuration file and parameters to go with it.
Sample Configuration
Here's a sample configuration for the Kubelet Stats Receiver with OpenTelemetry in OpenObserve:
Example Configuration File
|
Bash |
Configuration Parameters
Mandatory Parameters
- collection_interval: Interval at which to collect metrics (e.g., "10s")
- endpoint: Kubelet API endpoint (e.g., "${K8S_NODE_NAME}:10250")
- auth_type: Authentication type, either "serviceAccount" or "tls"
Optional Parameters
- metric_groups: Specify which metric groups to collect (container, node, pod, volume)
- extra_metadata_labels: Additional labels to include in the metrics (e.g., "kubernetes.io/hostname")
- request_limit_utilization:
- enabled: Enable request/limit utilization metrics
- threshold: Threshold percentage for request/limit utilization (e.g., 80)
- ca_file, key_file, cert_file: Required for "tls" auth_type
In the OpenObserve exporter configuration:
- endpoint: OpenObserve cluster URL (e.g., https://openobserve.example.com)
- api_key: Valid API key for authentication
This sample configuration demonstrates the key parameters for the Kubelet Stats Receiver and OpenObserve exporter. Adjust the values based on your specific requirements and environment. In the next section, we will discuss a few advanced configuration options.
Advanced Configuration Options
Here are the key advanced configuration options for installing the Kubelet Stats Receiver with OpenTelemetry:
Including/excluding specific metrics
You can configure which metric groups to collect by setting the metric_groups option in the receiver configuration. The available groups are:
- node: Node-level metrics
- pod: Pod metrics
- container: Container metrics
- volume: Volume metrics
For example, to collect only node and pod metrics:
|
Bash |
Adding additional metadata attributes
You can add extra metadata labels to the collected metrics by setting the extra_metadata_labels option. This requires get permissions on the nodes/proxy resource in addition to nodes/stats.
|
Bash |
Configuring collection intervals and security settings
The collection_interval option sets how often metrics are collected, defaulting to 10 seconds.
The auth_type option configures how the receiver authenticates to the kubelet API. The available options are:
- none: Use the read-only kubelet endpoint on port 10255 (no auth required)
- tls: Use TLS auth with the specified ca_file, cert_file, and key_file
- serviceAccount: Use the pod's service account token and CA cert
- kubeConfig: Use the specified kubeConfig file to authenticate and proxy through the API server
|
Bash |
The insecure_skip_verify option disables TLS cert verification when set to true.
Role-based access control
The Kubelet Stats Receiver requires get permissions on the nodes/stats resource. If using extra_metadata_labels or request/limit utilization metrics, it also needs access to nodes/proxy.
|
Bash |
By leveraging these advanced configuration options, you can fine-tune the Kubelet Stats Receiver to collect the exact metrics you need, add relevant metadata, and optimize collection intervals and security settings to best fit your observability requirements when using OpenTelemetry.
Monitoring and Metrics
Here are the key points about the available metrics and resource attributes for the Kubelet Stats Receiver:
Available metrics
The Kubelet Stats Receiver collects the following metric groups from the kubelet API:
- Node metrics: CPU, memory, filesystem, network, and volume metrics at the node level
- Pod metrics: CPU, memory, network, and volume metrics at the pod level
- Container metrics: CPU, memory, and filesystem metrics at the container level
- Volume metrics: Capacity, available, used, and inodes metrics for volumes
Resource attributes
The receiver automatically adds the following resource attributes to the collected metrics:
- k8s.cluster.name: Name of the Kubernetes cluster
- k8s.node.name: Name of the node
- k8s.pod.name: Name of the pod
- k8s.container.name: Name of the container
- k8s.volume.name: Name of the volume
You can also configure additional metadata labels to be added using the extra_metadata_labels option, such as:
- node_name
- pod_name
- container_name
This allows enriching the metrics with more context about the origin of the data.
Metric types
The metrics collected are of the following types:
- Counters: Monotonically increasing values, like CPU time
- Gauges: Values that can go up or down, like memory usage
- Histograms: Distributions of values, like latency
This rich set of node, pod, container and volume metrics along with the associated resource attributes can help you gain deep visibility into the performance and health of your Kubernetes cluster using the Kubelet Stats Receiver.
Troubleshooting and Support
Unable to get network stats in OpenShift 4
- If you are running the OpenObserve Collector in OpenShift 4 and unable to get network stats, ensure you have the correct RBAC permissions set up.
- The Kubelet Stats Receiver needs ‘get’ permissions on the nodes/stats resource. If using extra_metadata_labels or request/limit utilization metrics, it also needs access to nodes/proxy.
Metrics not showing up
- Check that the Kubelet Stats Receiver is properly configured in your OpenObserve Collector config.
- Ensure the kubeletstats receiver is included in the receivers section and the metrics pipeline.
- Verify the auth_type is set correctly. If using tls, make sure the ca_file, cert_file, and key_file are specified. For serviceAccount, ensure the pod has the necessary permissions.
Incomplete metrics
- By default, the Kubelet Stats Receiver collects node, pod and container metrics.
- To collect volume metrics as well, set metric_groups to include volume.
Getting support
For general questions and troubleshooting, you can reach out to the OpenObserve community on Github.
By following these troubleshooting steps and leveraging the OpenObserve community support, you should be able to resolve most issues encountered when using the Kubelet Stats Receiver.
Why OpenObserve is the Right Choice
OpenObserve is the ideal choice for analyzing and visualizing the metrics collected by the Kubelet Stats Receiver. With its powerful querying capabilities, customizable dashboards, and alerting features, OpenObserve enables you to:
- Quickly identify performance bottlenecks and anomalies
- Optimize resource utilization and plan for future capacity needs
- Correlate metrics across different layers of your stack
- Collaborate with your team and share insights
Sign up for OpenObserve Today
To get started with OpenObserve, sign up for a free trial. Our team is dedicated to helping you achieve observability and gain actionable insights from your Kubernetes metrics.
Don't miss out on the benefits of comprehensive Kubernetes monitoring with OpenObserve. Sign up today and take your observability to the next level!
