Already using OpenTelemetry and the Collector?

If you’ve already instrumented for tracing with OpenTelemetry using Collectors, then getting data into Cloud Observability requires only an edit to the YAML config file. You need to configure the Collector to use OTLP to send data to ServiceNow Cloud Observability using your project’s access token.

For sending metrics to Cloud Observability using the OpenTelemetry Collector, see Ingest metrics using the OpenTelemetry Collector.

In the collector-config.yaml file, add an otlp exporter that points to ingest.lightstep.com:443.

Start tabs

Public Satellites

1
2
3
4
5
6
7
exporters:

    otlp:
      endpoint: ingest.lightstep.com:443
      headers: {"lightstep-access-token":"<YOUR_ACCESS_TOKEN>"}

    . . .

Developer Mode

1
2
3
4
5
6
7
8
9
10
11
12
exporters:
    prometheus:
        endpoint: "0.0.0.0:8889"

    logging:
        loglevel: debug

    # configuring otlp to developer satellites
    otlp:
      endpoint: localhost:8360

    . . .

End code tabs

You can find a full example of a collector-config.yaml here.

Configuration Details and Troubleshooting

The first thing you should do when troubleshooting collector issues is make sure data from your network can reach Cloud Observability. Your firewall or cloud configuration may be preventing a connection.

The default OTLP Exporter from a Collector enables gzip compression and TLS. Depending on your network configuration, you may need to enable or disable certain other gRPC features. This page contains a complete list of configuration parameters for the Collector gRPC client.

In the event that you are unable to establish a gRPC connection to the Cloud Observability Observability platform, you can use the grpcurl tool to ensure connectivity from your network to our public satellites. Run the following command, replacing <YOUR_ACCESS_TOKEN> with your project’s access token:

1
grpcurl -H 'lightstep-access-token:<YOUR_ACCESS_TOKEN>' ingest.lightstep.com:443 list

You should see the following output, or something similar:

1
2
3
4
5
grpc.reflection.v1alpha.ServerReflection
jaeger.api_v2.CollectorService
lightstep.collector.CollectorService
lightstep.egress.CollectorService
opentelemetry.proto.collector.trace.v1.TraceService

If you do not see this output, or the request hangs, then something is blocking gRPC traffic from transiting your network to ours. Please ensure that any proxies are passing through the lightstep-access-token header.

Alternately, you can also try emitting a single span to your project using the otel-cli tool. We have an example image and commands for running the CLI tool in Kubernetes and Docker on GitHub. Only send test spans to a non-production project.

For additional troubleshooting recommendations, see Troubleshooting Missing Data in Cloud Observability.

Updated Aug 28, 2020