OpenTelemetry Kubernetes metrics quickstart

This topic describes how to install an OpenTelemetry Collector in Kubernetes to collect infrastructure and application metrics and send them to Cloud Observability. To simplify the configuration of the OpenTelemetry Collector, a StatefulSet is used with the Target Allocator’s PrometheusCRD functionality enabled. This behavior allows the Collector to dynamically discover new jobs.

Prerequisites

• Local install of Helm.
• A Cloud Observability access token.
• The OpenTelemetry Operator and Cert Manager running in your cluster.

Install the kube-otel-stack chart

1. From the Cloud Observability otel-collector-charts repository, copy the kube-otel-stack folder to your existing directory.

2. Set the shell variable LS_TOKEN to your Cloud Observability access token.

   1
   export LS_TOKEN=”<ACCESS_TOKEN>”
   

3. Disable any Kubernetes Infrastructure Components you do not need

   By default, the values.yaml for the kube-otel-stack has the following components enabled for scraping:

   • kubeApiServer
   • kubelet
   • kubeControllerManager
   • coreDns
   • kubeDns
   • kubeEtcd
   • kubeScheduler
   • kubeProxy
   • kubeStateMetrics

   You can disable any of them by setting enabled: false

4. Install the chart using the kube-otel-stack/values.yaml values.

   1
   2
   3
   4
   kubectl create namespace opentelemetry
   kubectl create secret generic otel-collector-secret -n opentelemetry --from-literal=LS_TOKEN=$LS_TOKEN
   helm dependency update
   helm upgrade lightstep ./charts/kube-otel-stack -f ./charts/kube-otel-stack/values.yaml -n opentelemetry --install
   

5. Verify that the StatefulSet Collector is up and running, You should see three pods in “ready” state in the namespace.

   1
   kubectl get statefulset -n opentelemetry
   

Next steps

Now that you are succesfully scraping Collector, Operator, Target Allocator, and Kubernetes infrastructure metrics, you can use our pre-built dashboards to monitor them. Or you can use the terraform dashboards. If in the future you have applications that emit Prometheus metrics, you can use ServiceMonitors and PodMonitors to scrape them.

See also

Scale Collector with StatefulSet

Performance test and tune the Collector

Updated Oct 21, 2022