Running the OpenTelemetry Collector as a DaemonSet
This topic covers instructions to deploy the OpenTelemetry Collector using Kubernetes DaemonSet to replace Prometheus. A DaemonSet is a type of workload that ensures that every Kubernetes Node has a running instance.
To scale the Collector with a Daemonset, you configure the Collector to only scrape application metrics from its nodes. Optionally, you can run another single Collector in Deployment mode to scrape static targets and infrastructure metrics if needed.
- Local install of Helm.
- A Lightstep Access Token.
- Have the OpenTelemetry Operator and Cert Manager running in your cluster.
This topic covers the steps to deploy two OpenTelemetry Collectors, one in DaemonSet mode and optionally, a second one in Deployment mode, in a Kubernetes Cluster. You will need to (in this order):
- Install the OpenTelemetry Collector DaemonSet
- Configure the OpenTelemetry Collector deployment to scrape infrastructure metrics and static targets, if needed
Install the OpenTelemetry Collector DaemonSet
From the Lightstep prometheus-k8s-opentelemetry-collector repository, copy the
collector_k8sfolder to your existing directory.
- Set the shell variable
LS_TOKENto your Lightstep Access Token.
- Install the OpenTelemetry Collector using the
1 2 3
kubectl create namespace opentelemetry kubectl create secret generic otel-collector-secret -n opentelemetry --from-literal=LS_TOKEN=$LS_TOKEN helm upgrade lightstep ./collector_k8s -f ./collector_k8s/values-daemonset.yaml -n opentelemetry --install
- Verify that the daemonset Collector is up and running, You should see one pod in “ready” state for each node on your cluster.
kubectl get daemonset -n opentelemetry
This Collector will scrape all pods that are annotated with the
prometheus.io/scrape: trueannotation, which is per pod. You can adjust the
prometheus.io/portannotation to scrape a port of your choice instead of the default.
- In Lightstep Observability, use a Notebook to verify that the metric
otelcol_process_uptimeis reporting to your Lightstep project. You can group this metric by
k8s.pod.nameto see all pods that were created. You should expect one pod for each node on your Kubernetes Cluster.
Additionally, verify that your applications are being scraped by the Collector with the metric
scrape_samples_scraped grouped by
service.name. You should see the amount of samples scraped from each application. At this point, you can start querying your app metrics.
If you don’t see this metric, you might not have set your token correctly. Check the logs of your Collector pod for
access token not found errors using:
% kubectl logs -n opentelemetry <collector pod name>.
If you see these errors, make sure that the correct token is saved in your
otel-collector-secret and has write metrics permissions.
Next, you can configure the deployment Collector to scrape your infrastructure metrics.
(Optional) Configure the Deployment Collector to scrape your infrastructure metrics
The DaemonSet Collector deployment has been configured to scrape application metrics from its nodes. In order to scrape static targets and infastructure metrics, run a second OpenTelemetry Collector as a single replica deployment.
Add your additional scrape targerts to the
scrape_configs.yaml. This should contain static targets that are not discovered by the Kubernetes service discovery in the DaemonSet Collector.
- Enable the secondary Collector Deployment by setting
collectorsarray element named as
Once complete, upgrade the Collector’s chart to incorporate the new changes.
helm upgrade lightstep ./collector_k8s -f ./collector_k8s/values-daemonset.yaml -n opentelemetry --install
- Using Notebooks, verify that your applications are being scraped by the Collector with the metric