This tutorial demonstrates how to use the OpenTelemetry Collector to send infrastructure metric data to Lightstep Observability.

You will run a local containerized instance of the OpenTelemetry Collector that collects and send host metric data using a Collector Receiver to Lightstep Observability.

otel-collector-example

If you’re deploying the collector in Kubernetes, we recommend using the Kubernetes Operator.

For more on the OpenTelemetry Collector, see the official OpenTelemetry docs.

Pre-Requisites

Configure the Collector to send host metrics to Lightstep

  1. Clone the OpenTelemetry Examples repo.

    1

     git clone https://github.com/lightstep/opentelemetry-examples.git

  2. Edit the OpenTelemetry Collector Config file.

    1

     cd opentelemetry-examples

    Open collector/vanilla/collector.yaml for editing using your favorite editor.

    It’s recommended that you make a copy of this file collector.yaml and save it as otelcol-lightstep.yaml.

    The file looks like this:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

     receivers:
   otlp:
     protocols:
       grpc:
       http:

 exporters:
   logging:
     logLevel: debug
   otlp/ls:
     endpoint: ingest.lightstep.com:443
     headers: 
       "lightstep-access-token": "${LIGHTSTEP_ACCESS_TOKEN}"

 processors:
   batch:

 service:
   pipelines:
     traces:
       receivers: [otlp]
       processors: [batch]
       exporters: [logging, otlp/ls]

     metrics:
         receivers: [otlp]
         processors: [batch]
         exporters: [logging,otlp/ls]

    Replace ${LIGHTSTEP_ACCESS_TOKEN} with your own Lightstep access token, and save the file. The access token tells what Lightstep project to send your telemetry data to.

  3. Add the host metrics receiver to the Collector config

    Update the otelcol-lightstep.yaml file and add a host metrics receiver under the receivers section. The scrapers designate which metrics to collect and from where. In this example we’re telling the scraper to collect metrics from our system’s cpu, load, memory, etc.

    Your receiver snippet should look like this:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

     receivers:
     otlp:
         protocols:
         grpc:
         http:

     hostmetrics:
         collection_interval: 10s
         scrapers:
             cpu:
             load:
             memory:
             disk:
             filesystem:
             network:
             paging:
             processes:

  4. Configure your Collector metrics pipeline.

    Now add the hostmetrics receiver to your Collector metrics pipeline. In the receivers array of the metrics pipeline add hostmetrics next to the existing entry. Your metric pipeline snippet should look like:

    1
2
3
4

     metrics:
     receivers: [hostmetrics,otlp]
     processors: [batch]
     exporters: [logging,otlp/ls]

    Your final otelcol-lightstep.yaml file should look like the following:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

     receivers:
   otlp:
     protocols:
       grpc:
       http:

   hostmetrics:
     collection_interval: 10s
     scrapers:
         cpu:
         load:
         memory:
         disk:
         filesystem:
         network:
         paging:
         processes:

 exporters:
   logging:
   otlp/ls:
     endpoint: ingest.lightstep.com:443
     headers:
         "lightstep-access-token": "${LIGHTSTEP_ACCESS_TOKEN}"

 processors:
   batch:

 service:
   pipelines:
     traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [logging, otlp/ls]
     metrics:
         receivers: [hostmetrics]
         processors: [batch]
         exporters: [logging,otlp/ls]

  5. Launch the Collector.

    Open a new terminal window in the opentelemetry-examples folder and run the following command:

    1
2
3
4

     docker run --rm \
   -v "$(pwd)/collector/vanilla/otelcol-lightstep.yaml":/collector.yaml \
   --name otelcol otel/opentelemetry-collector-contrib:latest \
   --config collector.yaml

    After running the Collector the metrics will start appearing within 5 minutes. Give yourself a well earned short break in the meantime, you deserve it.

  6. See metrics in Lightstep.

    Log into Lightstep.

    To view metrics in your Lightstep project, click notebooks or dashboards in the left navigation bar.

    When using notebooks you can click on any host metrics in the all telemetry dropdown. All operating systems don’t report all host metrics. Check the system.cpu.time metric first, as all operating systems report this.

    otel-infra-metrics-example

    There are also several different pre-built dashboards available for host metrics. Select the dashboard tile in the UI, click Create a pre-built dashboard, and choose the dashboard you’re interested in.

    otel-infra-metrics-dashboard-example

  7. Enrich your host metrics.

    You may notice that the metrics don’t have many relevant attributes. You can enhance your host metrics using a resource detector to capture resource information from your resources and add that to all of your metrics.

    In the processors section, add the resourcedetection and system detector:

    1
2
3
4

     processors:
   batch:
   resourcedetection:
     detectors: [system]

    And in your metric pipeline processor section:

    1
2
3
4

     metrics:
     receivers: [hostmetrics]
     processors: [batch,resourcedetection]
     exporters: [logging,otlp/ls]

  8. Re-launch the Collector.

    Once your metrics are flowing you should see the same metrics in your notebook or dashboard but with more attributes to filter on like host.name, host.id and os.type.

    otel-metrics-enriched-example

    This is just the first step. OpenTelemetry offers dozens of infrastructure receivers you can use and configure out of the box.

    For a full list of the out of the box receivers OpenTelemetry offers see Ingest metrics using the OpenTelemetry Collector.