Scrape infrastructure metrics and static targets in Prometheus format

Now that the Collector is available, you run a Lightstep Docker image in the namespace of your Prometheus server to extract and save the scrape_config file. An existing Prometheus server is required for this step.

  1. Check that your Prometheus pod is healthy and all containers are running.
    Replace <namespace> and <prometheus pod name> with your Prometheus server’s namespace and pod name.
    % kubectl get pod -n <namespace> <prometheus pod name>
  2. Run the following command to identify the pod ip address where your Prometheus server is running.
    % kubectl get pods -n <namespace> <prometheus pod name> -o jsonpath='{.status.podIP}'
  3. Extract and save your Prometheus configuration into scrape_configs.yaml.
    Replace <namespace> and <pod ip address> with your Prometheus server’s namespace and pod ip address.
    % kubectl run --rm  --quiet -i -n <namespace> --image=lightstep/prometheus-config-helper:latest --env="PROMETHEUS_ADDR=<pod ip address>:9090" --restart=Never get-prometheus-scrape-configs > collector_k8s/scrape_configs.yaml

    Depending on the state of the Prometheus server, this may fail and leave the scrape_configs.yaml file empty. If it does, you may safely rerun the command.

  4. (Optional) Edit scrape_config.yaml to exclude any scrape targets you want to omit.
    Use # to omit individual lines.
    Once complete, upgrade the Collector’s chart provided by Lightstep’s example respository to incorporate the new changes.
    # collector_k8s/values-deployment.yaml comes from cloned example repository
    % helm upgrade lightstep ./collector_k8s -f ./collector_k8s/values-deployment.yaml -n opentelemetry --install
  5. Verify your scrape targets are appearing using Notebooks. Verify OpenTelemetry Scrape Targets

What Did we learn?

  • We added a dashboard to Lightstep for monitoring the collector