The OpenTelemetry Collector, when configured with a Prometheus receiver, provides an integration with kubelet to scrape Prometheus metrics. The Collector uses the Prometheus Receiver to fetch metrics from the configured path. From there, the metrics are processed and exported to Cloud Observability.
To quickly ingest Kubernetes metrics into Cloud Observability using the OpenTelemetry Operator for Kubernetes, see instructions in the Kubernetes Quick Start.
To complete the integration, you will:
You’ve configured the Collector to export metric data to Cloud Observability.
You may need to expose kubelet Prometheus metrics. How you do that depends on your deployment model and whether or not the default parameters are appropriate in your case. In kubelet, Prometheus metrics are on port 10255 by default.
For more details on ingesting kubelet metrics see the official documentation on Kubernetes system component metrics.
In the Collector configuration file, configure the Prometheus receiver to use the kubelet Prometheus endpoint as a scrape target.
- job_name: 'otel-kubelet'
- targets: ['kubelet.kube-system.svc.cluster.local:10255', 'kubelet.kube-system.svc.cluster.local:10255/metrics/cadvisor']
The OpenTelemetry repo’s README provides additional details about Prometheus receiver configuration.
More details about the Prometheus scrape configuration can be found here.
Once the kubelet receiver is configured, enable it by adding it to one or more pipelines as described in the Collector configuration documentation.
You can validate that metrics are reporting to Cloud Observability on the Metrics details page in Organization settings.
In Cloud Observability, click Organization settings > Metric details.
Search for kubelet metric names.
Kubelet exposes metrics at paths
/metrics/probes. The endpoints have independent lifecycles and may also be configured to include feature gated or hidden metrics. For more details see the official Kubernetes system component documentation.
If needed, click on the metric to edit the description and how the units are displayed in Cloud Observability.
Use the Cloud Observability Terraform Provider to create a dashboard for the metrics.
Learn more about how to Ingest Prometheus metrics using an OpenTelemetry Collector on Kubernetes.
There is a pre-made Cloud Observability dashboard to help you quickly visualize kubelet monitoring.
Updated Dec 1, 2022