Scrape infrastructure metrics and static targets in Prometheus format

Now that the Collector is available, you run a Lightstep Docker image in the namespace of your Prometheus server to extract and save the scrape_config file. An existing Prometheus server is required for this step.

  1. Check that your Prometheus pod is healthy and all containers are running.
    Replace <namespace> and <prometheus pod name> with your Prometheus server’s namespace and pod name.
    1
    
    % kubectl get pod -n <namespace> <prometheus pod name>
    
  2. Run the following command to identify the pod ip address where your Prometheus server is running.
    1
    
    % kubectl get pods -n <namespace> <prometheus pod name> -o jsonpath='{.status.podIP}'
    
  3. Extract and save your Prometheus configuration into scrape_configs.yaml.
    Replace <namespace> and <pod ip address> with your Prometheus server’s namespace and pod ip address.
    1
    
    % kubectl run --rm  --quiet -i -n <namespace> --image=lightstep/prometheus-config-helper:latest --env="PROMETHEUS_ADDR=<pod ip address>:9090" --restart=Never get-prometheus-scrape-configs > charts/collector_k8s/scrape_configs.yaml
    

    Depending on the state of the Prometheus server, this may fail and leave the scrape_configs.yaml file empty. If it does, you may safely rerun the command.

  4. (Optional) Edit your scrape_config.yaml to exclude any scrape targets you want to omit.
    Use # to omit individual lines.
    Once complete, upgrade the Collector’s chart provided by Lightstep’s example respository to incorporate the new changes.

  5. Verify your scrape targets are appearing using Notebooks. Verify OpenTelemetry Scrape Targets

What Did we learn?

  • We added a dashboard to Lightstep for monitoring the collector