Satellite Configuration Parameters

Installation and configuration of LightStep Satellites vary, depending on if you’re installing to Docker, AWS/AMI, or Debian/Ubuntu, but all share the same configuration parameters. This topic describes each of these parameters. Read these topics for specific installation procedures, examples, and templates that you can use:

If the default port numbers work for you and you don’t require TLS, you need to modify only these parameters:

satellite_key
pool
bytes_per_project

Required parameters and highly recommended parameters are noted; all others are optional. Examples are shown for both YAML files for Debian, Ubuntu, or AMI, and environment variables for Docker.

We prioritize satellite configuration values as follows:
1. Any values found in flags are used if present.
2. Any values found in the file designated by the collector_base_config flag if present.
3. Any values found in the environment are used if present.
4. Any values found in the file designated by the COLLECTOR_BASE_CONFIG environment variable if present.

Authentication

satellite_key

REQUIRED

This value authorizes a Satellite to communicate with LightStep SaaS for span reporting. You can find or generate your Satellite key here. Copy and paste the key as the value for this parameter. Satellite keys expire after one year. You’ll need to replace this value when you generate a new one.

Your yaml file may say api-key instead of satellite-key. That’s fine - either will work.

Debian/Ubuntu/AMI

satellite_key: your-satellite-key

Docker

-e COLLECTOR_SATELLITE_KEY=your-satellite-key

Debugging

pool

HIGHLY RECOMMENDED

A user-defined, human-readable name for this Satellite pool to facilitate troubleshooting. LightStep uses this parameter to segment your traffic from other customers in metrics and logs. It is also displayed in the LightStep UI as the Satellite pool name on the Satellite Pool Report page.

Guidelines:

DO use different names for Satellite pools that serve distinct LightStep projects.
DO use different pool names for Satellites of different machine sizes or configuration settings. Try to keep individual instances within pools uniform.
DO use different names for Satellite pools in different regions.
DO use separate Satellite pool names for separate projects within LightStep. This enables finer-grained tuning and optimization.
DO NOT use a unique pool name per Satellite instance/machine.

Debian/Ubuntu/AMI

pool: production-canary

Docker

-e COLLECTOR_POOL=production-canary

guid

HIGHLY RECOMMENDED

A user-defined, human-readable name for this Satellite instance. This ID is used to assist with matching Satellite information in the LightStep application with instances running in your environment. The GUID allows you to identify individual Satellites within a pool. It’s displayed on the Satellites page when viewing a list of Satellites in a particular pool.

The GUID does not need to be the same as the AWS instance ID.

Guidelines:

DO provide a globally unique identifier.
DO use a value that maps directly to an instance in your environment.

Debian/Ubuntu/AMI

pool: ec2-19-211-35-1.compute-1.amazonaws.com
guid: 999.9.9.9

Docker

-e COLLECTOR_POOL=production-canary
-e COLLECTOR_GUID=999.9.9.9

Monitoring

Satellites can optionally export a set of StatsD-compatible metrics that allow you to monitor key indicators of a Satellite’s health. Read Understand StatsD Reporting Metrics for more info about Satellite monitoring and the metrics you can monitor.

statsd

REQUIRED if enabling monitoring

host

REQUIRED if enabling monitoring

This parameter sets the host for the StatsD agent.

Using localhost may not work. Use 127.0.0.1 instead.

Debian/Ubuntu/AMI

statsd:
  host: 127.0.0.1

Docker

-e COLLECTOR_STATSD_HOST=127.0.0.1

port

REQUIRED if enabling monitoring

Port number for the StatsD agent.

Debian/Ubuntu/AMI

port: 8125

Docker

-e COLLECTOR_STATSD_PORT=8125

export_statsd or export_datadog

REQUIRED if enabling monitoring

You can export metrics to either StatsD or to Datadog. You can only set one of them. The StatsD export generates metrics in basic StatsD format and does not support any tags. The Datadog export generates metrics compatible with Datadog, with one default tag (lightstep_project) and support for additional custom tags.

Debian/Ubuntu/AMI

export_statsd: true
# OR
export_dogstatsd: true

Docker

-e COLLECTOR_STATSD_EXPORT_STATSD=true
# OR
-e COLLECTOR_STATSD_EXPORT_DOGSTATSD=true

dogstatsd_tags

OPTIONAL if using Datadog for monitoring

These are tags used by Datadog when reporting metrics.
All tags must:

Start with a letter
Must only contain ASCII alphanumerics (lowercase recommended), underscores, and periods

Debian/Ubuntu/AMI

dogstats_tags: "pool:us-west-1,canary:true"

Docker

-e COLLECTOR_STATSD_DOGSTATSD_TAGS="pool:us-west-1,canary:true"

prefix

OPTIONAL

Metrics exported by the satellite are named as follows: <prefix>.<satellite_prefix|client_prefix>.<metric_name>

By default, the prefix is empty. You can add a custom prefix that will be prepended to the name of all metrics.

All prefixes must:

Start with a letter
Only contain ASCII alphanumerics (lowercase recommended), underscores, and periods

Best Practice:
Set the prefix to lightstep, so you know that these are the Satellite metrics.

Debian/Ubuntu/AMI

prefix: "lightstep"

Docker

-e COLLECTOR_STATSD_PREFIX=lightstep

satellite_prefix

OPTIONAL

Metrics exported by the satellite are named as follows: <prefix>.<satellite_prefix|client_prefix>.<metric_name>

The satellite_prefix field has a default value of satellite and is included in the name of all metrics that reflect Satellite behavior. You can change that value.

Debian/Ubuntu/AMI

satellite_prefix: "satellite-canary"

Docker

-e COLLECTOR_STATSD_SATELLITE_PREFIX=satellite-canary

client_prefix

OPTIONAL

Metrics exported by the satellite are named as follows: <prefix>.<satellite_prefix|client_prefix>.<metric_name>

The client_prefix field has a default value of client and is included in the name of all metrics that reflect the LightStep tracer behavior (as reported by the Satellite). You can change that value.

Debian/Ubuntu/AMI

client_prefix: "client-via-canary"

Docker

-e COLLECTOR_STATSD_CLIENT_PREFIX=client-via-canary

Ports

Satellites support four reporting mechanisms for instrumented clients to send span data to Satellites:

JSON over HTTP
Thrift over HTTP
Protocol buffers over HTTP
Protocol buffers over gRPC (HTTP/2)

As of March 2019, all ports accept all transports, so you only need to configure the ports under the admin object. These ports will accept incoming traffic from all tracer transport methods.

The exact Satellite configuration mechanism varies based on the distribution type, but generally contains the following variables:

http: May contain subsections secure_port and plain_port
grpc: May contain subsections secure_port and plain_port
plain_port: If set, the port on which to serve plain Thrift connections
secure_port: If set, the port on which to serve secure (TLS) Thrift connections
tls_cert_prefix: Root directory containing certificates for TLS connections

diagnostic_port

REQUIRED

This port serves the /diagnostics endpoint, which provides a status page with diagnostic information about the satellite.

The value is flexible and can be set to a different (unbound) port number as needed. The default is 8000.

Debian/Ubuntu/AMI

diagnostic_port: 8000

Docker

-e COLLECTOR_DIAGNOSTIC_PORT=port -p port:port

admin

REQUIRED

These ports (plain_port and secure_port) are used for health checks and diagnostics and serve the following endpoints:

/_ready: Used by load balancers for health checks. A 200 (OK) response indicates that the Satellite is both responding and able to handle incoming span traffic.
/_live: Used by orchestration tools like Kubernetes for liveness checks. A 200 (OK) response indicates that the Satellite is alive and responding, and you should not terminate the instance. However, it makes no promises about Satellite health. It may be temporarily overloaded and not accepting spans.

plain_port

REQUIRED

This port is the plaintext admin port and must always be set to an open port number for Satellite diagnostics to work properly. This port is also the destination for unencrypted health/readiness checks from a load balancer or orchestration framework.

secure_port

OPTIONAL

This port is only required if sending encrypted health/readiness checks to the Satellite (this is not common). If set to a non-zero value, the tls_cert_prefix parameter must also be set.

Debian/Ubuntu/AMI

admin:
  plain_port: 8180
  secure_port: 9090

Docker

-e COLLECTOR_ADMIN_PLAIN_PORT=port -p port:port
-e COLLECTOR_ADMIN_SECURE_PORT -p port:port

Proto-via-HTTP Ports

These ports accept span traffic encoded as Protocol buffers and sent to satellites over HTTP. The values are flexible and can be set to a different (unbound) port as needed.

These ports are required if using the following LightStep tracer clients handling this encoding/transport combination:

lightstep-tracer-go (if configured as use_http: true)
lightstep-tracer-java (if configured to use Protocol buffers over HTTP)
lightstep-tracer-android (if configured to use Protocol buffers over HTTP)

These ports are optional if you know that your satellites will never need to support these tracers, but it’s better to set at least one of them for maximum flexibility.

If you set either of these ports, you must also configure the corresponding grpc port.

http

plain_port: 8181

REQUIRED FOR TRACERS LISTED ABOVE

Required for the satellite to accept unencrypted Proto-over-HTTP span traffic. This port is also used for encrypted span traffic if TLS termination happens at the load balancer and data is not re-encrypted to the satellite.

secure_port

OPTIONAL Same as above, but for encrypted span traffic where TLS termination happens at the satellite. If you set this to a non-zero value, you must also configure the tls_cert_prefix parameter.

Debian/Ubuntu/AMI

http:
  plain_port: 8181
  secure_port: 9191

Docker

-e COLLECTOR_HTTP_PLAIN_PORT=port -p port:port
-e COLLECTOR_HTTP_SECURE_PORT=port -p port:port

gRPC Ports

At least one of these ports is required to accept span traffic via gRPC. The values are flexible and can be set to a different (unbound) port as needed.

At least one of these ports is also required in order to accept spans encoded as protocol buffers, even if the transport is HTTP.

These ports are required if using the following LightStep tracer clients:

lightstep-tracer-go (if configured to use gRPC)
lightstep-tracer-java (if configured to use gRPC)
lightstep-tracer-android (if configured to use gRPC)
lightstep-tracer-cpp

These ports are optional ONLY if you know that your satellites will never need to support any of these tracers, but it’s better to set at least one of them for maximum flexibility.

gRPC uses HTTP/2 and few HTTP load-balancers support this well, so we are currently recommending using TCP load-balancers while we continue to evaluate other options. With a TCP load-balancer, the Satellite terminates TLS for gRPC connections. More info on load balancing Satellites here.

grpc

plain_port

REQUIRED FOR TRACERS LISTED ABOVE AND IF SPANS ARE ENCODED AS PROTOCOL BUFFERS Allows the Satellite to accept unencrypted gRPC span traffic. This port is also used for secure connections if TLS termination happens at the load balancer and data is not re-encrypted to the Satellite.

secure_port

OPTIONAL Same as above, but for encrypted gRPC span traffic where TLS termination happens at the Satellite. If you set this to a non-zero value, you must also configure the tls_cert_prefix option.

Debian/Ubuntu/AMI

grpc:
  plain_port: 8282
  secure_port: 9292

Docker

-e COLLECTOR_GRPC_PLAIN_PORT -p port:port
-e COLLECTOR_GRPC_SECURE_PORT=port -p port:port

Thrift Ports

You must set at least one of the following ports to handle Thrift span traffic. The values are flexible and can be set to a different (unbound) port as needed.

These ports are required if using the following LightStep tracer clients:

lightstep-tracer-go (if configured as use_thrift: true)
lightstep-tracer-javascript
lightstep-tracer-objc
lightstep-tracer-php
lightstep-tracer-python
lightstep-tracer-ruby

These ports are optional ONLY if you know that your satellites will never need to support any of these tracers.

plain_port

REQUIRED FOR TRACERS LISTED ABOVE This port accepts unencrypted Thrift span traffic. It should also be used for secure connections if TLS termination happens at the load balancer and data is not re-encrypted to the satellite.

By default, the Satellite uses port 80 on the Satellite host for inbound non-secure communications with the application clients. Because port 80 is a privileged port, you may receive a permission error. You can modify the plain_port: configuration to use a non-privileged port (above 1024).

secure_port

This port accepts encrypted Thrift span traffic. If you set this to a non-zero value, you must also configure the tls_cert_prefix option.

Debian/Ubuntu/AMI

plain_port: 8383
secure_port: 9393

Docker

-e COLLECTOR_PLAIN_PORT=port -p port:port
-e COLLECTOR_SECURE_PORT=port -p port:port

Memory

These settings define how much memory the Satellite should dedicate to tracing data for a project. Satellites store data sent from tracers in memory, waiting for the SaaS LightStep Engine to query them for data for display in the UI. For a stream to reliably assemble representative traces, the recall at each Satellite must be at least 5 minutes. For Explorer queries, the Service diagram, and Correlations to return comprehensive results from 100% of the span data in the past certain number of minutes, the recall at each Satellite must be at least that certain number of minutes.

In general, a standard target for recall is 5 - 10 minutes, depending on how much history you want to be able to see in the Explorer view. However, there is no technical limitation to maintaining a longer recall window if you need to query the full set of span data longer into the past.

However, you can’t configure the recall window directly; It is proportional to the amount of span traffic sent from the tracer and the available Satellite memory. Longer recall can be achieved by either reducing the amount of span traffic set on the tracer during initialization) or increasing the available memory of Satellites in the pool (either by increasing the available memory per instance, or the overall number of instances).

Recommended Settings
LightStep recommends allocating 16GB of memory for each Satellite instance. The Satellite then sub-allocates this memory to various functions it needs to perform (span indexing, deserializing reports off of the wire, etc.). Most of this memory is allocated by the Satellite for its internal operation, leaving about one-quarter of the RAM to be used for indexing spans.

For example, if you allocate the recommended 16 GB of memory to a Satellite, 4 GB of memory are available for indexing spans.

You can then allocate the bytes available for indexing to the various projects reporting to the Satellite instance. This allocation is controlled through two configuration settings: bytes_per_project and bytes_per_project_overrides.

reporter

bytes_per_project

REQUIRED
The number of bytes to allocate per project. This is the amount allocated to each of your projects by default. You can override this using the bytes_per_project_overrides parameter.

When a Satellite serves multiple LightStep projects, you can control how the memory in each Satellite is partitioned between those projects. These values should be set deliberately and may require guidance.

Recommended Settings
Set this variable low (at or below100 MB). This prevents accidental out-of-memory errors when a new project is reported to an existing satellite pool.

bytes_per_project_overrides

Use this setting to override the default setting and configure how much memory a specific project is allocated for indexing. Changing the ratio of memory allocated to each project is a way to adjust the desired recall for each project in a pool. The parameter name must match a project’s name.

Two examples are shown, the first showing a LightStep organization with just one project - the my_primary_project. The second example shows a Satellite that serves two projects.

Set an override even if you only have one project.
Setting an override prevents future projects from consuming memory.

The example values below were computed based on our recommended machine size: 16GB RAM + 2 CPUs.

Example 1: Single Project Override

Debian/Ubuntu/AMI

reporter:
  bytes_per_project: 100_000_000  #100 MB (for any *other* project besides my_primary_project)
  bytes_per_project_overrides`:
    my_primary_project: 3_500_000_000  # 3.5 GB (e.g. production)

Docker

-e COLLECTOR_REPORTER_BYTES_PER_PROJECT=1000_000_000
-e COLLECTOR_REPORTER_BYTES_PER_PROJECT_OVERRIDES="{"my_primary_project":3_500_000_000}"

Example 2: Multiple Project Overrides

Debian/Ubuntu/AMI

reporter:
  bytes_per_project: 100_000_000 # 100 MB (for any *other* project besides those listed below)
  bytes_per_project_overrides:
    project_name_1: 3_000_000_000 # 3 GB (e.g. production instance)
    project_name_2: 500_000_000   # 500 MB (e.g. staging instance)

Docker

-e COLLECTOR_REPORTER_BYTES_PER_PROJECT=1000_000_000
-e COLLECTOR_REPORTER_BYTES_PER_PROJECT_OVERRIDES="{"project_name_1":3_000_000_000, "project_name_2":500_000_000}"

You should set the sum of (bytes_per_project + bytes_per_project_overrides) for all projects that report to a Satellite to approximately 1/4 of the available system memory.

When using Kubernetes, a node “limit” size of 16GB should be practical because a Satellite with the above configuration should not exceed the resources of a 16GB machine. However, please be sure to test this thoroughly in your environment, as Kubernetes will automatically shut down a node that exceeds this resource allocation, potentially leading to satellites being unavailable during periods of high load. If this type of shutdown event is a concern, you can omit setting a node limit altogether as a correctly configured Satellite should have no undesirable effect.

Recommended Settings
Once you install and configure your Satellites, it’s important to Monitor LightStep performance to ensure that your Satellite pools are balanced (each Satellite is receiving and sending similar amounts of data) and that neither your Satellites nor tracers are dropping spans due to insufficient memory allocation. If you find that one project is dropping spans, adjust this setting accordingly. Read about load balancing here.

TLS

A TLS value is required if you are using any secure ports to provide TLS termination at the Satellite. When setting a secure port, you must make certificates available to the Satellite. Copy TLS certificates and a key to PATH.bundle.pem and PATH.key.pem and set the TLS certificate prefix to PATH. The prefix may contain /. The bundle file must contain a complete chain of certificates from a root (for example, PATH.bundle.pem is often the concatenation of .crt and .ca-bundle).

tls_cert_prefix

REQUIRED IF USING SECURE PORTS This string gives the file path prefix for certificate files used in serving TLS connections on the Satellite.

This example assumes the certificate is located at /root/certs/mydomain.bundle.pem and /root/certs/mydomain.key.pem

Debian/Ubuntu/AMI

tls_cert_prefix: /root/certs/mydomain

Docker

-e COLLECTOR_TLS_CERT_PREFIX=/root/certs/mydomain

Single Project Mode

If you are only running one project in LightStep, then instead of needing to pass in an access-token, you can set your Satellites to single-project mode and they won’t expect the token.

disable_access_token_checking

OPTIONAL Set this to true if you are running a single project in LightStep and don’t want to have to pass your access token from your tracer to your Satellites.

project_name

REQUIRED IF DISABLE_ACCESS_TOKEN_CHECKING IS SET TO TRUE The name of the project in LightStep.

Debian/Ubuntu/AMI

disable_access_token_checking: true
project_name: "My Project"

Docker

COLLECTOR_DISABLE_ACCESS_TOKEN_CHECKING=true
COLLECTOR_PROJECT_NAME=${PROJECT}

Ingestion Tags

Ingestions tags are static tags added to all spans received by the Satellite.

ingestion_tags

A comma-separated name/value pair list of tags/values. These tags are static - the same tag and value is applied to every span the Satellite receives.

Debian/Ubuntu/AMI

ingestion_tags: myTagValue1,myTagKey2:myTagValue:2

Docker

COLLECTOR_INGESTION_TAGS=myTagValue1,myTagKey2:myTagValue:2

Authentication

satellite_key

Debugging

pool

guid

Monitoring

statsd

host

port

export_statsd or export_datadog

dogstatsd_tags

prefix

satellite_prefix

client_prefix

Ports

diagnostic_port

admin

plain_port

secure_port

Proto-via-HTTP Ports

http

plain_port: 8181

secure_port

gRPC Ports

grpc

plain_port

secure_port

Thrift Ports

plain_port

secure_port

Memory

reporter

bytes_per_project

bytes_per_project_overrides

TLS

tls_cert_prefix

Single Project Mode

disable_access_token_checking

project_name

Ingestion Tags

ingestion_tags

Was this topic helpful?

Read Next

Install and Configure Satellites

Learn About Satellites

Monitor LightStep Performance