This document describes steps to take to verify that you’ve installed and configured your Microsatellites correctly.
Verify Microsatellite log statuses
/var/log/syslog
(on Docker and Debian)/var/log/messages
(Amazon Linux)
The log should contain output from a self-test:
1
2
3
4
5
6
7
8
Satellite Self-Test Results...
Satellite Key authorized for:
1 Organizations [organization name]
5 Projects [project name]
100 Access Tokens [redacted]
Tests: Connected to api-grpc.lightstep.com:8043: success
Require unique port numbers: success
Self-Test Status: SUCCESS
If you don’t see a success message, read below to help troubleshoot.
Verify basic connectivity
- Verify application host to Microsatellite network connectivity. Are you able to ping the Microsatellite from the application host?
- Verify that the Microsatellite host is correctly listening. Open a browser and access the Microsatellite’s diagnostic page:
http://{satellite_ip}:8000}/diagnostics
. The Satellite Diagnostics page should display.
Test connectivity to Microsatellites
The following shows how to test the connectivity to the Microsatellites using either HTTP and a sample payload or gRPC (HTTP/2).
Microsatellites accept telemetry data via the following reporting mechanisms:
- Protocol buffers over HTTP
- Protocol buffers over gRPC (HTTP/2)
- JSON over HTTP
- Thrift over HTTP
Support for Thrift over HTTP will be deprecated!
Trace ingest endpoints
Endpoints accepting application/x-protobuf
- OTLP versions v0.5 - v0.9:
Start tabs
On-Premise Satellites
1
2
3
https://<microsatellite_ip>:<microsatellite_port>/traces/otlp/v0.6
or
https://<microsatellite_ip>:<microsatellite_port>/traces/otlp/v0.9
Public Satellites
1
2
3
https://ingest.lightstep.com:443/traces/otlp/v0.6
or
https://ingest.lightstep.com:443/traces/otlp/v0.9
End code tabs
Endpoints accepting application/json
- OTLP version v0.5:
Start tabs
On-Premise Satellites
1
https://<microsatellite_ip>:<microsatellite_port>/api/v2/otel/trace
Public Satellites
1
https://ingest.lightstep.com:443/api/v2/otel/trace
End code tabs
- OTLP versions v0.6 and above:
Start tabs
On-Premise Satellites
1
2
3
https://<microsatellite_ip>:<microsatellite_port>/traces/otlp/v0.6
or
https://<microsatellite_ip>:<microsatellite_port>/traces/otlp/v0.9
Public Satellites
1
2
3
https://ingest.lightstep.com:443/traces/otlp/v0.6
or
https://ingest.lightstep.com:443/traces/otlp/v0.9
End code tabs
If you are using OTLP v0.5, you must upgrade as Lightstep support for v0.5 will be deprecated!
Metrics ingest endpoints
Endpoints accepting application/x-protobuf
- OTLP versions v0.5 - v0.7:
Start tabs
On-Premise Satellites
1
2
3
https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.5
or
https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.6
Public Satellites
1
2
3
https://ingest.lightstep.com:443/metrics/otlp/v0.5
or
https://ingest.lightstep.com:443/metrics/otlp/v0.6
End code tabs
- OTLP version v0.9:
Start tabs
On-Premise Satellites
1
https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.9
Public Satellites
1
https://ingest.lightstep.com:443/metrics/otlp/v0.9
End code tabs
Endpoints accepting application/json
- OTLP versions v0.5 - v0.7:
Start tabs
On-Premise Satellites
1
2
3
https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.5
or
https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.6
Public Satellites
1
2
3
https://ingest.lightstep.com:443/metrics/otlp/v0.5
or
https://ingest.lightstep.com:443/metrics/otlp/v0.6
End code tabs
- OTLP version v0.9:
Start tabs
On-Premise Satellites
1
https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.9
Public Satellites
1
https://ingest.lightstep.com:443/metrics/otlp/v0.9
End code tabs
If you are using OTLP v0.5, you must upgrade as Lightstep support for v0.5 will be deprecated!
All of Lightstep Observability’s ingestion endpoints are encrypted with TLS.
Test the HTTP ingest path
In order to test connectivity to a public or on-premise microsatellite via its HTTP ingestion points, you need to use a command-line utility such as curl
. You also need your Lightstep access token in order to authenticate the request.
Use the following protobuf JSON payload small_data.json
to represent a test span and run the following curl command to test overall connectivity to the Microsatellites:
small_data.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
{
"resourceSpans": [
{
"resource": {
"attributes": [
{
"key": "service.name",
"value": {
"stringValue": "curl-test-otel-pipeline"
}
}
]
},
"instrumentationLibrarySpans": [
{
"spans": [
{
"traceId": "71699b6fe85982c7c8995ea3d9c95df2",
"spanId": "3c191d03fa8be065",
"name": "test-span",
"kind": 1,
"droppedAttributesCount": 0,
"events": [],
"droppedEventsCount": 0,
"status": {
"code": 1
}
}
],
"instrumentationLibrary": {
"name": "local-curl-example"
}
}
]
}
]
}
Start tabs
On-Premise Satellites
1
2
3
curl -ivL -H "Lightstep-Access-Token: <LIGHTSTEP_ACCESS_TOKEN>" \
-H "Content-Type: application/json" \
http://{microsatellite_ip}:{http_port}/traces/otlp/v0.9 -d @small_data.json
Public Satellites
1
2
3
curl -ivL -H "Lightstep-Access-Token: <LIGHTSTEP_ACCESS_TOKEN>" \
-H "Content-Type: application/json" \
https://ingest.lightstep.com/traces/otlp/v0.9 -d @small_data.json
End code tabs
A successful request should receive a 200 HTTP response, and a successful response will include a receive and transmit timestamp:
{"receiveTimestamp":"2022-04-29T03:57:33.827317232Z","transmitTimestamp":"2022-04-29T03:57:33.864646841Z"}
An HTTP response code of 400 Bad Request
indicates either an issue with the access token in use, or a problem with the payload sent to the public microsatellite pool.
If you receive an HTTP response indicating a request timeout, verify that a web proxy or firewall is not intercepting the traffic destined for the microsatellite.
Testing the gRPC ingest path
Testing against a public or on-premise microsatellite pool’s gRPC ingest path requires a tool called grpcurl
, available here. You can ensure that you have a direct connection to Lightstep Observability by downloading and running the aforementioned tool with your Lightstep access token (for authentication).
You should receive a response listing the RPCs in use by the server:
Start tabs
On-Premise Satellites
1
2
3
4
5
6
7
$ grpcurl -H 'lightstep-access-token:{LS_ACCESS_TOKEN}' {microsatellite_ip}:{grpc_port} list
grpc.reflection.v1alpha.ServerReflection
jaeger.api_v2.CollectorService
lightstep.collector.CollectorService
lightstep.egress.CollectorService
opentelemetry.proto.collector.trace.v1.TraceService
Public Satellites
1
2
3
4
5
6
7
$ grpcurl -H 'lightstep-access-token:{LS_ACCESS_TOKEN}' ingest.lightstep.com:443 list
grpc.reflection.v1alpha.ServerReflection
jaeger.api_v2.CollectorService
lightstep.collector.CollectorService
lightstep.egress.CollectorService
opentelemetry.proto.collector.trace.v1.TraceService
End code tabs
If you receive any errors such as:
Failed to dial target host "ingest.lightstep.com:443": context deadline exceeded
or
Failed to dial target host "ingest.lightstep.com:443": read tcp 192.168.95.2:37948->35.222.219.106:443: read: connection reset by peer
verify that there are no proxies or firewalls in the environment that might be blocking direct access via gRPC to the microsatellite pool in question.
Possible service startup issues
Check your /var/logs/syslog
file on your Microsatellite host for these possible Microsatellite service startup issues:
-
Apr 30 18:18:15 collector-host lightstep-collector[5571]: panic: failed to generate credentials:open /root/certs/mydomain.bundle.pem: permission denied
If you are not using a secure connection between the application and the Microsatellite, comment out thetls_cert_prefix: /root/certs/mydomain
in your Microsatellite Configuration file. If you are using a secure connection, make sure the certificate file is correct. -
Apr 30 18:22:12 collector-host lightstep-collector[5729]: panic: listen tcp :80: bind: permission denied
By default, the Microsatellite uses port80
on the Microsatellite host for inbound non-secure communications with the application clients. Because port80
is a privileged port, you may receive a permission error. You can modify theplain_port:
configuration in your Microsatellite Configuration file to use a non-privileged port (above1024
).
Monitor and tune Microsatellites
Now that you’ve verified that your tracers, Microsatellites, and the Lightstep SaaS are communicating, you need to ensure that the configurations are ideal for the amount of span data that your tracers are collecting from your instrumentation.
Lightstep offers a number of ways to Monitor Lightstep performance. If you see issues, you may need to tune configuration options, add more satellites, or tune your Microsatellite pools and tracers.