Verify and test Microsatellite setup

This document describes steps to take to verify that you’ve installed and configured your Microsatellites correctly.

Verify Microsatellite log statuses

  • /var/log/syslog (on Docker and Debian)
  • /var/log/messages (Amazon Linux)

The log should contain output from a self-test:

1
2
3
4
5
6
7
8

Satellite Self-Test Results...
Satellite Key authorized for:
1 Organizations [organization name]
5 Projects [project name]
100 Access Tokens [redacted]
Tests: Connected to api-grpc.lightstep.com:8043: success
Require unique port numbers: success
Self-Test Status: SUCCESS

If you don’t see a success message, read below to help troubleshoot.

Verify basic connectivity

  1. Verify application host to Microsatellite network connectivity. Are you able to ping the Microsatellite from the application host?
  2. Verify that the Microsatellite host is correctly listening. Open a browser and access the Microsatellite’s diagnostic page: http://{satellite_ip}:8000}/diagnostics. The Satellite Diagnostics page should display.

Test connectivity to Microsatellites

The following shows how to test the connectivity to the Microsatellites using either HTTP and a sample payload or gRPC (HTTP/2).

Protocols and formats accepted by Cloud Observability Microsatellites

Microsatellites accept telemetry data via the following reporting mechanisms:

  • Protocol buffers over HTTP
  • Protocol buffers over gRPC (HTTP/2)
  • JSON over HTTP
  • Thrift over HTTP

Support for Thrift over HTTP will be deprecated!

Expandable end

Cloud Observability Microsatellite endpoints

Trace ingest endpoints

Endpoints accepting application/x-protobuf
  • OTLP versions v0.5 - v0.9:

Start tabs

On-Premise Satellites

1

https://<microsatellite_ip>:<microsatellite_port>/v1/traces

Public Satellites

1

https://ingest.lightstep.com:443/v1/traces

End code tabs

Endpoints accepting application/json
  • OTLP version v0.5:

Start tabs

On-Premise Satellites

1

https://<microsatellite_ip>:<microsatellite_port>/api/v2/otel/trace

Public Satellites

1

https://ingest.lightstep.com:443/api/v2/otel/trace

End code tabs

  • OTLP versions v0.6 and above:

Start tabs

On-Premise Satellites

1

https://<microsatellite_ip>:<microsatellite_port>/v1/traces

Public Satellites

1
2
3

https://ingest.lightstep.com:443/traces/otlp/v0.6
or 
https://ingest.lightstep.com:443/v1/traces

End code tabs

If you are using OTLP v0.5, you must upgrade as Cloud Observability support for v0.5 will be deprecated!

Metrics ingest endpoints

Endpoints accepting application/x-protobuf
  • OTLP versions v0.5 - v0.7:

Start tabs

On-Premise Satellites

1
2
3

https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.5
or
https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.6

Public Satellites

1
2
3

https://ingest.lightstep.com:443/metrics/otlp/v0.5
or
https://ingest.lightstep.com:443/metrics/otlp/v0.6

End code tabs

  • OTLP version v0.9:

Start tabs

On-Premise Satellites

1

https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.9

Public Satellites

1

https://ingest.lightstep.com:443/metrics/otlp/v0.9

End code tabs

Endpoints accepting application/json
  • OTLP versions v0.5 - v0.7:

Start tabs

On-Premise Satellites

1
2
3

https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.5
or
https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.6

Public Satellites

1
2
3

https://ingest.lightstep.com:443/metrics/otlp/v0.5
or
https://ingest.lightstep.com:443/metrics/otlp/v0.6

End code tabs

  • OTLP version v0.9:

Start tabs

On-Premise Satellites

1

https://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.9

Public Satellites

1

https://ingest.lightstep.com:443/metrics/otlp/v0.9

End code tabs

If you are using OTLP v0.5, you must upgrade as Cloud Observability support for v0.5 will be deprecated!

All of Cloud Observability’s ingestion endpoints are encrypted with TLS.

Expandable end

Test the HTTP ingest path

In order to test connectivity to a public or on-premise microsatellite via its HTTP ingestion points, you need to use a command-line utility such as curl. You also need your Cloud Observability access token in order to authenticate the request.

Use the following protobuf JSON payload small_data.json to represent a test span and run the following curl command to test overall connectivity to the Microsatellites:

small_data.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "curl-test-otel-pipeline"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "spans": [
            {
              "traceId": "71699b6fe85982c7c8995ea3d9c95df2",
              "spanId": "3c191d03fa8be065",
              "name": "test-span",
              "kind": 1,
              "droppedAttributesCount": 0,
              "events": [],
              "droppedEventsCount": 0,
              "status": {
                "code": 1
              }
            }
          ],
          "scope": {
            "name": "local-curl-example"
          }
        }
      ]
    }
  ]
 }

Start tabs

On-Premise Satellites

1
2
3

curl -ivL -H "Lightstep-Access-Token: <LIGHTSTEP_ACCESS_TOKEN>" \
-H "Content-Type: application/json" \
http://{microsatellite_ip}:{http_port}/v1/traces -d @small_data.json

Public Satellites

1
2
3

curl -ivL -H "Lightstep-Access-Token: <LIGHTSTEP_ACCESS_TOKEN>" \
-H "Content-Type: application/json" \
https://ingest.lightstep.com/v1/traces -d @small_data.json

End code tabs

A successful request should receive a 200 HTTP response with an empty body.

An HTTP response code of 400 Bad Request indicates either an issue with the access token in use, or a problem with the payload sent to the public microsatellite pool.

If you receive an HTTP response indicating a request timeout, verify that a web proxy or firewall is not intercepting the traffic destined for the microsatellite.

Testing the gRPC ingest path

Testing against a public or on-premise microsatellite pool’s gRPC ingest path requires a tool called grpcurl, available here. You can ensure that you have a direct connection to Cloud Observability by downloading and running the aforementioned tool with your Cloud Observability access token (for authentication).

You should receive a response listing the RPCs in use by the server:

Start tabs

On-Premise Satellites

1
2
3
4
5
6
7

$ grpcurl -H 'lightstep-access-token:{LS_ACCESS_TOKEN}' {microsatellite_ip}:{grpc_port} list

grpc.reflection.v1alpha.ServerReflection
jaeger.api_v2.CollectorService
lightstep.collector.CollectorService
lightstep.egress.CollectorService
opentelemetry.proto.collector.trace.v1.TraceService

Public Satellites

1
2
3
4
5
6
7

$ grpcurl -H 'lightstep-access-token:{LS_ACCESS_TOKEN}' ingest.lightstep.com:443 list

grpc.reflection.v1alpha.ServerReflection
jaeger.api_v2.CollectorService
lightstep.collector.CollectorService
lightstep.egress.CollectorService
opentelemetry.proto.collector.trace.v1.TraceService

End code tabs

If you receive any errors such as:

Failed to dial target host "ingest.lightstep.com:443": context deadline exceeded

or

Failed to dial target host "ingest.lightstep.com:443": read tcp 192.168.95.2:37948->35.222.219.106:443: read: connection reset by peer

verify that there are no proxies or firewalls in the environment that might be blocking direct access via gRPC to the microsatellite pool in question.

Possible service startup issues

Check your /var/logs/syslog file on your Microsatellite host for these possible Microsatellite service startup issues:

  • Apr 30 18:18:15 collector-host lightstep-collector[5571]: panic: failed to generate credentials:open /root/certs/mydomain.bundle.pem: permission denied If you are not using a secure connection between the application and the Microsatellite, comment out the tls_cert_prefix: /root/certs/mydomain in your Microsatellite Configuration file. If you are using a secure connection, make sure the certificate file is correct.

  • Apr 30 18:22:12 collector-host lightstep-collector[5729]: panic: listen tcp :80: bind: permission denied By default, the Microsatellite uses port 80 on the Microsatellite host for inbound non-secure communications with the application clients. Because port 80 is a privileged port, you may receive a permission error. You can modify the plain_port: configuration in your Microsatellite Configuration file to use a non-privileged port (above 1024).

Monitor and tune Microsatellites

Now that you’ve verified that your tracers, Microsatellites, and the Cloud Observability SaaS are communicating, you need to ensure that the configurations are ideal for the amount of span data that your tracers are collecting from your instrumentation.

Cloud Observability offers a number of ways to Monitor Cloud Observability performance. If you see issues, you may need to tune configuration options, add more satellites, or tune your Microsatellite pools and tracers.

See also

Microsatellite configuration parameters

Troubleshooting Missing Data in Cloud Observability

Learn about Microsatellites

Updated Aug 2, 2022