If you’ve already instrumented for tracing with OpenTelemetry using Collectors, then getting data into Cloud Observability requires only an edit to the YAML config file. You need to configure the Collector to use OTLP to send data to ServiceNow Cloud Observability using your project’s access token.
For sending metrics to Cloud Observability using the OpenTelemetry Collector, see Ingest metrics using the OpenTelemetry Collector.
In the collector-config.yaml
file, add an otlp
exporter that points to ingest.lightstep.com:443
.
Start tabs
Public Satellites
1
2
3
4
5
6
7
exporters:
otlp:
endpoint: ingest.lightstep.com:443
headers: {"lightstep-access-token":"<YOUR_ACCESS_TOKEN>"}
. . .
Developer Mode
1
2
3
4
5
6
7
8
9
10
11
12
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
logging:
loglevel: debug
# configuring otlp to developer satellites
otlp:
endpoint: localhost:8360
. . .
End code tabs
You can find a full example of a collector-config.yaml
here.
The first thing you should do when troubleshooting collector issues is make sure data from your network can reach Cloud Observability. Your firewall or cloud configuration may be preventing a connection.
The default OTLP Exporter from a Collector enables gzip
compression and TLS.
Depending on your network configuration, you may need to enable or disable
certain other gRPC features. This
page
contains a complete list of configuration parameters for the Collector gRPC
client.
In the event that you are unable to establish a gRPC connection to the Cloud Observability
Observability platform, you can use the
grpcurl tool to ensure connectivity
from your network to our public satellites. Run the following command, replacing
<YOUR_ACCESS_TOKEN>
with your project’s access
token:
1
grpcurl -H 'lightstep-access-token:<YOUR_ACCESS_TOKEN>' ingest.lightstep.com:443 list
You should see the following output, or something similar:
1
2
3
4
5
grpc.reflection.v1alpha.ServerReflection
jaeger.api_v2.CollectorService
lightstep.collector.CollectorService
lightstep.egress.CollectorService
opentelemetry.proto.collector.trace.v1.TraceService
If you do not see this output, or the request hangs, then something is blocking
gRPC traffic from transiting your network to ours. Please ensure that any
proxies are passing through the lightstep-access-token
header.
Alternately, you can also try emitting a single span to your project using the otel-cli tool. We have an example image and commands for running the CLI tool in Kubernetes and Docker on GitHub. Only send test spans to a non-production project.
For additional troubleshooting recommendations, see Troubleshooting Missing Data in Cloud Observability.
Updated Aug 28, 2020