Configure the OpenTelemetry Collector to Ingest Prometheus Metrics

Now that you’ve installed the OpenTelemetry Operator for Kubernetes, you can configure the collector with Lightstep’s example Helm chart for a single-replica deployment.

  1. From the Lightstep prometheus-k8s-opentelemetry-collector repository, copy the collector_k8s folder to your existing directory.
    1
    
    git clone https://github.com/lightstep/prometheus-k8s-opentelemetry-collector
    
  2. Set the shell variable LS_TOKEN to your Lightstep Access Token.
    1
    
    export LS_TOKEN="<ACCESS_TOKEN>"
    
  3. Configure a single-replica collector using Lightstep’s example Helm chart for the OpenTelemetry Operator for Kubernetes.
    1
    2
    3
    
    kubectl create namespace opentelemetry
    kubectl create secret generic otel-collector-secret -n opentelemetry --from-literal=LS_TOKEN=$LS_TOKEN
    helm upgrade lightstep ./collector_k8s -f ./collector_k8s/values-deployment.yaml -n opentelemetry --install
    
  4. In Lightstep Observability, use a Notebook to verify that the metric otelcol_process_uptime is reporting to your Lightstep project. Verifying OpenTelemetry Installation

If you don’t see this metric, you might not have set your token correctly. Check the logs of your Collector pod for access token not found errors using: % kubectl logs -n opentelemetry <collector pod name> If you see these errors, make sure that the token saved in your otel-collector-secret is correct and has write metrics permissions.

Troubleshooting

The default OTLP Exporter from a Collector enables gzip compression and TLS. Depending on your network configuration, you may need to enable or disable certain other gRPC features. This page contains a complete list of configuration parameters for the Collector gRPC client.

In the event that you are unable to establish a gRPC connection to the Lightstep Observability platform, you can use the grpcurl tool to ensure connectivity from your network to our public satellites. Run the following command, replacing <YOUR_ACCESS_TOKEN> with your project’s access token:

1
grpcurl -H 'lightstep-access-token:<YOUR_ACCESS_TOKEN>' ingest.lightstep.com:443 list

You should see the following output, or something similar:

1
2
3
4
5
grpc.reflection.v1alpha.ServerReflection
jaeger.api_v2.CollectorService
lightstep.collector.CollectorService
lightstep.egress.CollectorService
opentelemetry.proto.collector.trace.v1.TraceService

If you do not see this output, or the request hangs, then something is blocking gRPC traffic from transiting your network to ours. Please ensure that any proxies are passing through the lightstep-access-token header.

For additional troubleshooting recommendations, see Troubleshooting Missing Data in Lightstep.


What Did we learn?

  • We configured the collector using the Kubernetes Operator for OpenTelemetry