If you are migrating to Lightstep from Prometheus (or from no prior metric installation) and want to collect infrastructure metrics and application metrics within a Kubernetes cluster environment, we recommend using the OpenTelemetry Collector with the Prometheus receiver.

This topic covers replacing Prometheus with an OpenTelemetry Collector in your Kubernetes cluster and assumes that you are running a single pod Prometheus in a Kubernetes cluster. You need to (in this order):

  1. Install the OpenTelemetry Operator and Cert Manager
  2. Install the OpenTelemetry Collector
  3. Configure the Collector to scrape the metrics you need

These instructions install the OpenTelemetry Collector to a Kubernetes cluster as a single replica Kubernetes Deployment (also called “standalone” mode) using the OpenTelemetry Operator. If you’re interested in running a high-availability Collector (multiple replicas), please contact your Customer Success representative.

You must be able to run ValidatingWebhookConfigurations and MutatingWebhookConfigurations within your Kubernetes cluster; these are used to verify the Collector configuration.

Prerequisites

Install the OpenTelemetry Operator and Cert Manager

Before you install an OpenTelemetry Collector in the cluster, you need to install the OpenTelemetry Operator into the cluster. The OpenTelemetry Operator requires a Cert Manager installation to be present.

You can learn more about the Operator pattern in Kubernetes here.

  1. Configure Helm for installation.
    1
    2
    3
    
    % helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
    % helm repo add jetstack https://charts.jetstack.io
    % helm repo update
    
  2. Install the Cert Manager.
    1
    2
    3
    4
    5
    6
    
    % helm install \
      cert-manager jetstack/cert-manager \
      --namespace cert-manager \
      --create-namespace \
      --version v1.8.0 \
      --set installCRDs=true
    
  3. Install the OpenTelemetry Operator.
    1
    2
    3
    4
    
    % helm install \
      opentelemetry-operator open-telemetry/opentelemetry-operator \
      -n opentelemetry-operator \
      --create-namespace
    
  4. Verify the components have been correctly installed.
    1
    2
    3
    4
    
    ## this should show “cert-manager” and “opentelemetry-operator” installed
    % helm list -A
    ## this will complete when the opentelemetry operator pod is finished
    % kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=opentelemetry-operator -n opentelemetry-operator
    

    Next, you need to install the OpenTelemetry Collector.

Install the OpenTelemetry Collector

Now that you’ve installed Operator, you can install the Collector.

  1. From the Lightstep prometheus-k8s-opentelemetry-collector repository, copy the collector_k8s folder to your existing directory.

  2. Set the shell variable LS_TOKEN to your Lightstep Access Token.
    1
    
    export LS_TOKEN=”<ACCESS_TOKEN>”
    
  3. Install the OpenTelemetry Collector.
    1
    2
    3
    
    kubectl create namespace opentelemetry
    kubectl create secret generic otel-collector-secret -n opentelemetry --from-literal=LS_TOKEN=$LS_TOKEN
    helm upgrade lightstep ./collector_k8s -f ./collector_k8s/values.yaml -n opentelemetry --install
    
  4. In Lightstep Observability, use a Notebook to verify that the metric otelcol_process_uptime is reporting to your Lightstep project. Verifying OpenTelemetry Installation

If you don’t see this metric, you might not have set your token correctly. Check the logs of your Collector pod for access token not found errors using: % kubectl logs -n opentelemetry <collector pod name> If you see these errors, make sure that the token saved in your otel-collector-secret is correct and has write metrics permissions.

Next, you need to configure the collector to scrape the metrics.

Configure the Collector to scrape a subset of metrics

Now that the Collector is available, you run a Lightstep Docker image in the namespace of your Prometheus server to extract and save the scrape_config file.

  1. Check that your Prometheus pod is healthy and all containers are running.
    Replace <namespace> and <prometheus pod name> with your Prometheus server’s namespace and pod name.
    1
    
    % kubectl get pod -n <namespace> <prometheus pod name>
    
  2. Run the following command to identify the pod ip address where your Prometheus server is running.
    1
    
    % kubectl get pods -n <namespace> <prometheus pod name> -o jsonpath='{.status.podIP}'
    
  3. Extract and save your Prometheus configuration into scrape_configs.yaml.
    Replace <namespace> and <pod ip address> with your Prometheus server’s namespace and pod ip address.
    1
    
    % kubectl run --rm  --quiet -i -n <namespace> --image=lightstep/prometheus-config-helper:latest --env="PROMETHEUS_ADDR=<pod ip address>:9090" --restart=Never get-prometheus-scrape-configs > collector_k8s/scrape_configs.yaml
    

    Depending on the state of the Prometheus server, this may fail and leave the scrape_configs.yaml file empty. If it does, you may safely rerun the command.

  4. (Optional) Edit scrape_config.yaml to exclude any scrape targets you want to omit.
    Use # to omit individual lines.
    Once complete, upgrade the Collector’s chart to incorporate the new changes.
    1
    
    % helm upgrade lightstep ./collector_k8s -f ./collector_k8s/values.yaml -n opentelemetry --install
    
  5. Verify your scrape targets are appearing using Notebooks. Verify OpenTelemetry Scrape Targets