Already using OpenTelemetry and the Collector?

If you’ve already instrumented for tracing with OpenTelemetry using Collectors, then getting data into Cloud Observability requires only an edit to the YAML config file. You need to configure the Collector to use OTLP to send data to ServiceNow Cloud Observability using your project’s access token.

For sending metrics to Cloud Observability using the OpenTelemetry Collector, see Ingest metrics using the OpenTelemetry Collector.

In the collector-config.yaml file, add an otlp exporter that points to ingest.lightstep.com:443.

Organizations using the EU data center should use ingest.eu.lightstep.com:443.

Start tabs

Public Satellites

1
2
3
4
5
6
7
8

exporters:

    otlp:
      endpoint: ingest.lightstep.com:443 # US data center
      # endpoint: ingest.eu.lightstep.com:443 # EU data center
      headers: {"lightstep-access-token":"<YOUR_ACCESS_TOKEN>"}

    . . .

Developer Mode

1
2
3
4
5
6
7
8
9
10
11
12

exporters:
    prometheus:
        endpoint: "0.0.0.0:8889"

    logging:
        loglevel: debug

    # configuring otlp to developer satellites
    otlp:
      endpoint: localhost:8360

    . . .

End code tabs

You can find a full example of a collector-config.yaml here.

Configuration Details and Troubleshooting

The first thing you should do when troubleshooting collector issues is make sure data from your network can reach Cloud Observability. Your firewall or cloud configuration may be preventing a connection.

The default OTLP Exporter from a Collector enables gzip compression and TLS. Depending on your network configuration, you may need to enable or disable certain other gRPC features. This page contains a complete list of configuration parameters for the Collector gRPC client.

In the event that you are unable to establish a connection to the Cloud Observability platform, you can use curl to verify HTTP/2 connectivity to our collectors. Run the following command, replacing <YOUR_ACCESS_TOKEN> with your project’s access token:

1
2

curl -D- -XPOST --http2-prior-knowledge -H "lightstep-access-token: <YOUR_ACCESS_TOKEN>" https://ingest.lightstep.com/access-test # US data center
# curl -D- -XPOST --http2-prior-knowledge -H "lightstep-access-token: <YOUR_ACCESS_TOKEN>" https://ingest.eu.lightstep.com/access-test # EU data center

You should see the following output, or something similar:

1
2
3
4
5
6
7

HTTP/2 200
content-length: 2
content-type: text/plain
date: Thu, 09 May 2024 15:39:14 GMT
server: envoy

OK

If you do not see this output, or the request hangs, then something is blocking HTTP2 traffic from transiting your network to ours.

If you see HTTP/2 401, your request succeeded, but your token was not accepted. Some things to check:

  • Validity of your access token.
  • Ensure proxies are passing through the lightstep-access-token header.

Alternatively, to exercise the full gRPC request/response cycle, you can try emitting a single span to your project using the otel-cli tool. Refer to this example image and commands for running the CLI tool in Kubernetes and Docker on GitHub. Only send test spans to a non-production project.

For additional troubleshooting recommendations, see Troubleshooting Missing Data in Cloud Observability.

Updated Aug 28, 2020