This topic is about our legacy Satellites. If you installed Satellites after 4/06/2021, you are probably running Microsatellites.
Installation and configuration of Lightstep Satellites vary, depending on if you’re installing to Docker, AWS/AMI, or Debian/Ubuntu, but all share the same configuration parameters. This topic describes each of these parameters. Read these topics for specific installation procedures, examples, and templates that you can use:
- Docker: Install and Configure Satellites
- Docker with Helm: Install and Configure Satellites
- AWS/AMI: Install and Configure Satellites
- Debian/Ubuntu: Install and Configure Satellites
If the default port numbers work for you and you don’t require TLS, you need to modify only these parameters:
-
satellite_type: flavor
( AWS/AMI only: be sure this is set tofull
) satellite_key
-
pool
bytes_per_project
Required parameters and highly recommended parameters are noted; all others are optional. Examples are shown using YAML files for Debian, Ubuntu, or AMI, and environment variables for Docker without Helm or a YAML file if using a Helm chart.
We prioritize Satellite configuration values as follows:
1. Any values found in flags are used if present.
2. Any values found in the file designated by the collector_base_config
flag if present.
3. Any values found in the environment are used if present.
4. Any values found in the file designated by the COLLECTOR_BASE_CONFIG
environment variable if present.
satellite_type
Determines whether you are installing a Microsatellite or a legacy Satellite.
Start tabs
Debian/Ubuntu/AMI
1
2
satellite_type:
flavor: full
Docker Env Vars
1
-e COLLECTOR_SATELLITE_TYPE_FLAVOR=full
Docker with Helm
1
2
satellite_type:
flavor: full
End code tabs
Authentication
These parameters allow your Satellites to communicate with the Lightstep SaaS platform.
satellite_key
REQUIRED
This value authorizes a Satellite to communicate with the platform for span reporting. You can find or generate your Satellite key here. Copy and paste the key as the value for this parameter. Satellite keys expire after one year. You’ll need to replace this value when you generate a new one.
Your yaml file may say api_key
instead of satellite_key
. That’s fine - either will work.
Start tabs
Debian/Ubuntu/AMI
1
satellite_key: your-satellite-key
Docker Env Vars
1
-e COLLECTOR_SATELLITE_KEY=your-satellite-key
Docker with Helm
1
2
3
4
satelliteKey: "your-satellite-key"
# OR if using a Kubernetes secret key
collector_satellite_key_secret_name: "your-satellite-key"
collector_satellite_key_secret_key: "your-satellite-key-secret"
End code tabs
Debugging
These parameters create IDs for your Satellites and pools for easy debugging.
pool
HIGHLY RECOMMENDED
A user-defined, human-readable name for this Satellite pool to facilitate troubleshooting. Lightstep uses this parameter to segment your traffic from other customers in metrics and logs. It is also displayed in the Lightstep UI as the Satellite pool name on the Satellite Pool Report page.
Guidelines:
- DO use different names for Satellite pools that serve distinct Lightstep projects.
- DO use different pool names for Satellites of different machine sizes or configuration settings. Try to keep individual instances within pools uniform.
- DO use different names for Satellite pools in different regions.
- DO use separate Satellite pool names for separate projects within Lightstep. This enables finer-grained tuning and optimization.
- DO NOT use a unique pool name per Satellite instance/machine.
Start tabs
Debian/Ubuntu/AMI
1
pool: production-canary
Docker Env Vars
1
-e COLLECTOR_POOL=production-canary
Docker with Helm
1
collector_pool: rancher-satellite-pool
End code tabs
guid
HIGHLY RECOMMENDED
A user-defined, human-readable name for this Satellite instance. This ID is used to assist with matching Satellite information in the Lightstep application with instances running in your environment. The GUID allows you to identify individual Satellites within a pool. It’s displayed on the Satellites page when viewing a list of Satellites in a particular pool.
The GUID does not need to be the same as the AWS instance ID.
Guidelines:
- DO provide a globally unique identifier.
- DO use a value that maps directly to an instance in your environment.
Start tabs
Debian/Ubuntu/AMI
1
2
pool: ec2-19-211-35-1.compute-1.amazonaws.com
guid: 999.9.9.9
Docker Env Vars
1
2
-e COLLECTOR_POOL=production-canary
-e COLLECTOR_GUID=999.9.9.9
Docker with Helm
1
2
collector_pool: rancher-satellite-pool
guid: 999.9.9.9
End code tabs
Monitoring
Satellites can optionally export a set of StatsD-compatible metrics that allow you to monitor key indicators of a Satellite’s health. Read Understand StatsD Reporting Metrics for more info about Satellite monitoring and the metrics you can monitor.
statsd
REQUIRED if enabling monitoring
host
REQUIRED if enabling monitoring
This parameter sets the host for the StatsD agent.
Using localhost
may not work. Use 127.0.0.1
instead.
Start tabs
Debian/Ubuntu/AMI
1
2
statsd:
host: 127.0.0.1
Docker Env Vars
1
-e COLLECTOR_STATSD_HOST=127.0.0.1
End code tabs
port
REQUIRED if enabling monitoring
Port number for the StatsD agent.
Start tabs
Debian/Ubuntu/AMI
1
port: 8125
Docker Env Vars
1
-e COLLECTOR_STATSD_PORT=8125
End code tabs
export_statsd or export_dogstatsd
REQUIRED if enabling monitoring
You can export metrics to either StatsD or to Datadog. You can only set one of them. The StatsD export generates metrics in basic StatsD format and does not support any tags. The Datadog export generates metrics compatible with Datadog, with two default tags:
lightstep_project
: Holds the Lightstep project namesatellite_type
: Shows the type of Satellite reporting: eitherfull
ormicro
.
You can also add custom tags.
Start tabs
Debian/Ubuntu/AMI
1
2
3
export_statsd: true
# OR
export_dogstatsd: true
Docker Env Vars
1
2
3
-e COLLECTOR_STATSD_EXPORT_STATSD=true
# OR
-e COLLECTOR_STATSD_EXPORT_DOGSTATSD=true
End code tabs
dogstatsd_tags
OPTIONAL if using Datadog for monitoring
These are tags used by Datadog when reporting metrics.
All tags must:
- Start with a letter
- Must only contain ASCII alphanumerics (lowercase recommended), underscores, and periods
Start tabs
Debian/Ubuntu/AMI
1
dogstats_tags: "pool:us-west-1,canary:true"
Docker Env Vars
1
-e COLLECTOR_STATSD_DOGSTATSD_TAGS="pool:us-west-1,canary:true"
End code tabs
prefix
OPTIONAL
Metrics exported by the Satellite are named as follows: <prefix>.<satellite_prefix|client_prefix>.<metric_name>
By default, the prefix
is empty. You can add this custom prefix to have it prepended to the name of all metrics.
All prefixes must:
- Start with a letter
- Only contain ASCII alphanumerics (lowercase recommended), underscores, and periods
Best Practice:
Set the prefix to lightstep
, so you know that these are the Satellite metrics.
Start tabs
Debian/Ubuntu/AMI
1
prefix: "lightstep"
Docker Env Vars
1
-e COLLECTOR_STATSD_PREFIX=lightstep
End code tabs
satellite_prefix
OPTIONAL
Metrics exported by the Satellite are named as follows: <prefix>.<satellite_prefix|client_prefix>.<metric_name>
The satellite_prefix
field has a default value of satellite
and is included in the name of all metrics that reflect Satellite behavior. You can change that value.
Start tabs
Debian/Ubuntu/AMI
1
satellite_prefix: "satellite-canary"
Docker Env Vars
1
-e COLLECTOR_STATSD_SATELLITE_PREFIX=satellite-canary
End code tabs
client_prefix
OPTIONAL
Metrics exported by the Satellite are named as follows: <prefix>.<satellite_prefix|client_prefix>.<metric_name>
The client_prefix
field is used to identify the tracer and has a default value of client
. It’s included in the name of all metrics that reflect the Lightstep tracer behavior (as reported by the Satellite). You can change that value.
Start tabs
Debian/Ubuntu/AMI
1
client_prefix: "client-via-canary"
Docker Env Vars
1
-e COLLECTOR_STATSD_CLIENT_PREFIX=client-via-canary
End code tabs
Ports
These parameters configure the ports your tracers use to communicate with your Satellites.
Ports can’t be shared and must be unique.
Satellites support four reporting mechanisms:
- JSON over HTTP
- Thrift over HTTP
- Protocol buffers over HTTP
- Protocol buffers over gRPC (HTTP/2)
span ingest
REQUIRED
Set at least one of these ports to handle incoming span traffic.
plain_port
This port accepts unencrypted span traffic.
Start tabs
Debian/Ubuntu/AMI
1
plain_port: 8383
Docker Env Vars
1
-e COLLECTOR_PLAIN_PORT=port -p port:port
Docker with Helm
1
plain_port: 8383
End code tabs
secure_port
This port accepts encrypted span traffic.
Start tabs
Debian/Ubuntu/AMI
1
secure_port: 9393
Docker Env Vars
1
-e COLLECTOR_SECURE_PORT=port -p port:port
Docker with Helm
1
secure_port: 9393
End code tabs
diagnostic_port
REQUIRED
This port serves the /diagnostics
endpoint, which provides a status page with diagnostic information about the Satellite.
The value is flexible and can be set to a different (unbound) port number as needed. The default is 8000
.
Start tabs
Debian/Ubuntu/AMI
1
diagnostic_port: 8000
Docker Env Vars
1
-e COLLECTOR_DIAGNOSTIC_PORT=port -p port:port
Docker with Helm
1
diagnostic_port: 8000
End code tabs
admin
REQUIRED
These ports (plain_port
and secure_port
) are used for health checks and diagnostics and serve the following endpoints:
/_ready
: Used by load balancers for health checks. A200
(OK) response indicates that the Satellite is both responding and able to handle incoming span traffic./_live
: Used by orchestration tools like Kubernetes for liveness checks. A200
(OK) response indicates that the Satellite is alive and responding, and you should not terminate the instance. However, it makes no promises about Satellite health. It may be temporarily overloaded and not accepting spans.
plain_port
REQUIRED
This port is the plaintext admin port and must always be set to an open port number for Satellite diagnostics to work properly. This port is also the destination for unencrypted health/readiness checks from a load balancer or orchestration framework.
secure_port
OPTIONAL
This port is only required if sending encrypted health/readiness checks to the Satellite (this is not common). If set to a non-zero value, the tls_cert_prefix parameter must also be set.
Start tabs
Debian/Ubuntu/AMI
1
2
3
admin:
plain_port: 8180
secure_port: 9090
Docker Env Vars
1
2
-e COLLECTOR_ADMIN_PLAIN_PORT=port -p port:port
-e COLLECTOR_ADMIN_SECURE_PORT -p port:port
Docker with Helm
1
2
admin_plain_port: 8180
admin_secure_port: 9090
End code tabs
Memory
These parameters define how much memory the Satellite should dedicate to tracing data for a project. Satellites store data sent from tracers in memory, waiting for the platform to query them for data for display in the UI. For a stream to reliably assemble representative traces, the recall at each Satellite must be at least 5 minutes. For Explorer queries, the Service diagram, and Correlations to return comprehensive results from 100% of the span data in the past certain number of minutes, the recall at each Satellite must be at least that certain number of minutes.
In general, a standard target for recall is 5 - 10 minutes, depending on how much history you want to be able to see in the Explorer view. However, there is no technical limitation to maintaining a longer recall window if you need to query the full set of span data longer into the past.
However, you can’t configure the recall window directly; It is proportional to the amount of span traffic sent from the tracer and the available Satellite memory. Longer recall can be achieved by either reducing the amount of span traffic set on the tracer during initialization) or increasing the available memory of Satellites in the pool (either by increasing the available memory per instance, or the overall number of instances).
Recommended Settings
Lightstep recommends allocating 16GB of memory for each Satellite instance. The Satellite then sub-allocates this memory to various functions it needs to perform (span indexing, deserializing reports off of the wire, etc.). Most of this memory is allocated by the Satellite for its internal operation, leaving about one-quarter of the RAM to be used for indexing spans.
For example, if you allocate the recommended 16 GB of memory to a Satellite, 4 GB of memory are available for indexing spans.
You can then allocate the bytes available for indexing to the various projects reporting to the Satellite instance. This allocation is controlled through two configuration settings: bytes_per_project
and bytes_per_project_overrides
.
reporter
bytes_per_project
REQUIRED
The number of bytes to allocate per project. This is the amount allocated to each of your projects by default. You can override this for a particular project using the bytes_per_project_overrides
parameter.
When a Satellite serves multiple Lightstep projects, you can control how the memory in each Satellite is partitioned between those projects. These values should be set deliberately and may require guidance.
Recommended Settings
Set this variable to be less than the bytes_per_project_overrides
parameter. This prevents accidental out-of-memory errors when a new project is reported to an existing Satellite pool.
bytes_per_project_overrides
Use this setting to override the default setting and configure how much memory a specific project is allocated for indexing. Changing the ratio of memory allocated to each project is a way to adjust the desired recall for each project in a pool. The parameter name must match a project’s name.
Two examples are shown, the first showing a Lightstep organization with just one project - the my_primary_project
. The second example shows a Satellite that serves two projects.
Set an override even if you only have one project.
Setting an override prevents future projects from consuming memory.
The example values below were computed based on our recommended machine size: 16GB RAM + 2 CPUs.
Example 1: Single Project Override
Start tabs
Debian/Ubuntu/AMI
1
2
3
4
reporter:
bytes_per_project: 100000000 #100 MB (for any *other* project besides my_primary_project)
bytes_per_project_overrides`:
my_primary_project: 3500000000 # 3.5 GB (e.g. production)
Docker Env Vars
1
2
-e COLLECTOR_REPORTER_BYTES_PER_PROJECT=1000000000
-e COLLECTOR_REPORTER_BYTES_PER_PROJECT_OVERRIDES="{"my_primary_project":3500000000}"
Docker with Helm
1
2
bytes_per_project: "1000000000"
bytes_per_project_override:"3500000000"
End code tabs
Example 2: Multiple Project Overrides
Start tabs
Debian/Ubuntu/AMI
1
2
3
4
5
reporter:
bytes_per_project: 100000000 # 100 MB (for any *other* project besides those listed below)
bytes_per_project_overrides:
project_name_1: 3000000000 # 3 GB (e.g. production instance)
project_name_2: 500000000 # 500 MB (e.g. staging instance)
Docker Env Vars
1
2
-e COLLECTOR_REPORTER_BYTES_PER_PROJECT=1000000000
-e COLLECTOR_REPORTER_BYTES_PER_PROJECT_OVERRIDES="{"project_name_1":3000000000, "project_name_2":500000000}"
Docker with Helm
1
2
3
4
bytes_per_project: "1000000000"
bytes_per_project_override:
project_name_1: "3000000000" # 3 GB (e.g. production instance)
project_name_2: "500000000" # 500 MB (e.g. staging instance)
End code tabs
You should set the sum of (bytes_per_project
+ bytes_per_project_overrides
) for all projects that report to a Satellite to approximately 1/4 of the available system memory.
When using Kubernetes, a node “limit” size of 16GB should be practical because a Satellite with the above configuration should not exceed the resources of a 16GB machine. However, please be sure to test this thoroughly in your environment, as Kubernetes will automatically shut down a node that exceeds this resource allocation, potentially leading to Satellites being unavailable during periods of high load. If this type of shutdown event is a concern, you can omit setting a node limit altogether as a correctly configured Satellite should have no undesirable effect.
Recommended Settings
Once you install and configure your Satellites, it’s important to Monitor Lightstep performance to ensure that your Satellite pools are balanced (each Satellite is receiving and sending similar amounts of data) and that neither your Satellites nor tracers are dropping spans due to insufficient memory allocation. If you find that one project is dropping spans, adjust this setting accordingly. Read about load balancing here.
TLS
A TLS value is required if you are using any secure ports to provide TLS termination at the Satellite. When setting a secure port, you must make certificates available to the Satellite. Copy TLS certificates and a key to PATH.bundle.pem
and PATH.key.pem
and set the TLS certificate prefix to PATH
. The prefix may contain /
. The bundle file must contain a complete chain of certificates from a root (for example, PATH.bundle.pem
is often the concatenation of .crt
and .ca-bundle
).
tls_cert_prefix
REQUIRED IF USING SECURE PORTS
This string gives the file path prefix for certificate files used in serving TLS connections on the Satellite.
This example assumes the certificate is located at /root/certs/mydomain.bundle.pem
and /root/certs/mydomain.key.pem
Start tabs
Debian/Ubuntu/AMI
1
tls_cert_prefix: /root/certs/mydomain
Docker Env Vars
1
-e COLLECTOR_TLS_CERT_PREFIX=/root/certs/mydomain
Docker with Helm
1
tls_cert_prefix: "/root/certs/mydomain"
End code tabs
Single Project Mode
If you are only running one project in Lightstep, then instead of needing to pass in an access token, you can set your Satellites to single-project mode and they won’t expect the token.
disable_access_token_checking
OPTIONAL
Set this to true
if you are running a single project in Lightstep and don’t want to have to pass your access token from your tracer to your Satellites.
project_name
REQUIRED IF DISABLE_ACCESS_TOKEN_CHECKING IS SET TO TRUE
The name of the project in Lightstep.
Start tabs
Debian/Ubuntu/AMI
1
2
disable_access_token_checking: true
project_name: "My Project"
Docker Env Vars
1
2
COLLECTOR_DISABLE_ACCESS_TOKEN_CHECKING=true
COLLECTOR_PROJECT_NAME=${PROJECT}
Docker with Helm
1
2
disable_access_token_checking: true
project_name: "My Project"
End code tabs
Ingestion Tags
Ingestions tags are static tags added to all spans received by the Satellite.
ingestion_tags
OPTIONAL
A comma-separated name/value pair list of tags/values. These tags are static - the same tag and value is applied to every span the Satellite receives.
Start tabs
Debian/Ubuntu/AMI
1
ingestion_tags: myTagKey1:myTagValue1,myTagKey2:myTagValue2
Docker Env Vars
1
COLLECTOR_INGESTION_TAGS=myTagKey1:myTagValue1,myTagKey2:myTagValue2
Docker with Helm
1
collector_ingestion_tags: "myTagKey1:myTagValue1,myTagKey2:myTagValue2"
End code tabs
Receivers
This feature requires a Satellite upgrade to the June 2020 release.
When you use an OpenTracing auto-installer for instrumentation, you may end up with non human-readable operation names, or operation names that don’t make sense. This can happen because the auto-installer is getting the name from a parameter in Datadog (the original owner of the installer) that might not be appropriate for your language. You can set that parameter to different values to see if that results in better operation names.
You can use either the resource
or the name
parameter, or both.
Which to use (or if using the both, the order to use) depends on the language of the installer. Refer to the Datadog docs for more info.
operation_name_extractors
OPTIONAL
Set the parameter to either resource
, name
, or both. If you set both, the order of the values matters. Lightstep tries to extract a name from the first variable value. If one isn’t found, it looks for the second value and uses that.
This parameter can only be used with YAML configuration.
Start tabs
Debian/Ubuntu/AMI
1
2
3
4
5
receivers:
datadog:
operation_name_extractors:
- resource
- name
End code tabs
Sampling
By default, Satellites collect 100% of the trace data emitted by your instrumentation. If you wish to reduce the number of spans sent to the platform for cost reasons, you can configure your Satellites to sample data.
If you decide to use this parameter, you will lose data. Satellites base sampling on trace_id
to ensure that only full traces are sent. Spans in the same trace will either all be sent or all be dropped. Stream statistics remain accurate.
sample_one_in_n
OPTIONAL (should be used only when necessary)
This parameter allows you to send only a percentage of span data to the platform. When this parameter is set to a value greater than 0, the Satellite will send 1 out of every N spans. For example, if sample_one_in_n
is set to 10
, the Satellite will send only 1 out of every 10 spans, or 10% of all spans.
This setting must be the same on all Satellites in the pool to avoid broken traces.
Start tabs
Debian/Ubuntu/AMI
1
sample_one_in_n: 10
Docker Env Vars
1
COLLECTOR_SAMPLE_ONE_IN_N=10
Docker with Helm
1
sample_one_in_n: 10
End code tabs