LightStep

LightStep [𝑥]PM Documentation

Welcome to the LightStep developer hub. You'll find comprehensive guides and documentation to help you start working with LightStep [𝑥]PM as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    

Satellite Setup

See also:

What is the Satellite?

The LightStep [x]PM data processing pipeline has two major components: the Satellites, which are standalone software appliances running within a customer VPC, and the LightStep Engine, the SaaS component. All spans generated by instrumented clients and servers are sent to a pool of satellites where they are processed and temporarily stored during trace assembly. The LightStep Engine queries the satellites, records aggregate information about spans, directs the assembly process, and stores traces durably.

Satellites are straightforward to deploy using a Docker image, AWS AMI, or Debian package.

Setup and Configuration

The LightStep Satellite is generally run in a Docker container or in a dedicated virtual machine instance. We recommend provisioning memory optimized instances, such as r4.large, n1-highmem-2, or E2 v3 for AWS, GCP, and Azure respectively. However, smaller instance types may be appropriate for development or testing environments. You must allow outbound traffic from the satellites on TCP port 443 to api-all.lightstep.com. Satellites have no storage requirements.

You must include an API Key as part of your Satellite configuration. These keys can be generated and managed through the account settings page in LightStep [x]PM, and a single key can be used across many Satellites. The mechanism used to configure a Satellite depends on the platform on which it's deployed;

Always use the most recent version of the Satellite available, and update your Satellites according to the schedule provided in your contract. Please also note that your Satellite API key will expire every 12 months (for security purposes).

Configuration Template

This is a self-documenting template file with comments explaining the various Satellite configuration parameters.

A variant of this file in its entirety (with appropriately configured parameters) can placed directly as /etc/lightstep/collector.yaml with the Debian package installs, or the individual parameter keys can be configured as User Data with the AMI, or the equivalent environment variables can be passed to the Docker image (usually with some minor translation - see the examples in the Docker section).

# Satellite Configuration
#
# This file describes all the required and optional fields to customize your satellite configuration.
#
# If you are happy with the default port numbers and no TLS, you only need to modify the following settings:
#     api_key
#     pool
#     bytes_per_project
#
# These settings are specified in different ways depending on the satellite distribution type:
#
#     Debian/Ubuntu:
#         This file should already be present on the filesystem and can be modified in place.
#
#         /etc/lightstep/collector.yaml
#
#     AMI:
#         Copy the contents of this file into the Instance User Data in EC2
#
#     Docker:
#         Instead of using this file format, set the corresponding environment variables when launching the satellite.
#         See [Docker] tags throughout this file to find the environment variable names.
#         Include all the settings that are marked as REQUIRED below, and expose the corresponding ports to the host.
#
#         Example:
#
#         docker run \
#            -e COLLECTOR_API_KEY={your_API_key}             \
#            -e COLLECTOR_POOL={your_pool_name}              \
#            -e COLLECTOR_BABYSITTER_PORT=8000  -p 8000:8000 \
#            -e COLLECTOR_ADMIN_PLAIN_PORT=8080 -p 8080:8080 \
#            -e COLLECTOR_HTTP_PLAIN_PORT=8181  -p 8181:8181 \
#            -e COLLECTOR_GRPC_PLAIN_PORT=8282  -p 8282:8282 \
#            -e COLLECTOR_PLAIN_PORT=8383       -p 8383:8383 \
#            -e COLLECTOR_REPORTER_BYTES_PER_PROJECT={see below} \
#            -e COLLECTOR_REPORTER_BYTES_PER_PROJECT_OVERRIDES={see below} \
#            lightstep/collector:{version}
#
# For more information see:
# https://docs.lightstep.com/docs/satellite-setup

# ----------------------- AUTHENTICATION ---------------------

# REQUIRED: Satellite API Key
# This value authorizes a satellite to communicate with LightStep SaaS for span reporting.
# Satellite API Keys can be viewed and created by account administrators: https://app.lightstep.com/<project>/account
# Copy the raw text of the API key below.
#
# [Docker]: -e COLLECTOR_API_KEY=key
api_key: 

# ----------------------- DEBUGGING ---------------------

# HIGHLY RECOMMENDED: Satellite Pool Name
# A user-defined, human-readable name for this satellite pool to facilitate troubleshooting.
# It will be used by LightStep support to segment your traffic from other customers in metrics and logs.
#
# Guidelines:
# - DO use different names for satellite pools that serve distinct LightStep projects.
# - DO use different names for satellites that have different machine sizes or configuration settings.
# - DO use different names for satellites in different regions.
#
# - DO NOT use a unique name per satellite instance/machine.
#
# Examples: 
#   pool: production-canary
#   pool: myorganization-dev
#   pool: myproduct-staging-us-east-1
#
# [Docker]: -e COLLECTOR_POOL=pool_name
pool: 

# ----------------------- MONITORING ---------------------
# OPTIONAL: Export statsd metrics for Satellite
# The satellite can optionally export a set of statsd-compatible metrics that allow you to monitor key indicators.
# To enable satellite metrics, uncomment the `statsd` header and relevant fields below.
#
# statsd:
#
#   REQUIRED: Host address for the statsd agent
#     Note: "localhost" may not work, use 127.0.0.1 instead
#     Example:
#       host: 127.0.0.1
#     [Docker]: COLLECTOR_STATSD_HOST
#     Uncomment the line below to set the statsd agent host address
#   host:
#
#   REQUIRED: Port number for the statsd agent
#     Example:
#       port: 8125
#     [Docker]: COLLECTOR_STATSD_PORT
#     Uncomment the line below to set the port number for your statsd agent
#   port:
#
#   REQUIRED: Choose between statsd and dogstatsd metrics format
#     The following settings are mutually exclusive and you must set exactly one of them.
#     - export_statsd generates metrics in basic statsd format, and does not support any tags
#     - export_dogstatsd generates metrics that are compatible with DataDog,
#       with one default tag (lightstep_project) and support for additional customer-provided tags
#     [Docker]: COLLECTOR_STATSD_EXPORT_STATSD
#     [Docker]: COLLECTOR_STATSD_EXPORT_DOGSTATSD
#     Uncomment one of the following lines (or set the corresponding env variable above) to enable the desired format for metrics
#   export_statsd: true
#   export_dogstatsd: true
#
#   OPTIONAL: Add custom tags for dogstatsd metrics
#     This setting is only available when export_dogstatsd: true
#     Example:
#       dogstats_tags: "pool:us-west-1,canary:true"
#     [Docker]: COLLECTOR_STATSD_DOGSTATSD_TAGS
#     Uncomment the following line to add one or more custom tags for dogstatsd metrics
#   dogstatsd_tags:
#
#
#   Note: Metrics exported by the satellite are named as follows:
#       <prefix>.<satellite_prefix|client_prefix>.<metric_name>
#   You can use the set of optional fields below to customize the metric names, if desired.
#
#     All prefixes and tag names below should follow these rules, unless your metrics service provider supports additional characters:
#     - must start with a letter
#     - must only contain ASCII alphanumerics (lowercase recommended), underscores, and periods
#
#   OPTIONAL: Set a custom metric name prefix
#     The prefix is empty by default.
#     If set to a non-empty value, the prefix will be prepended as a dot-separated prefix to the name of all metrics
#     Example:
#       prefix: "lightstep"
#     [Docker]: COLLECTOR_STATSD_PREFIX
#     Uncomment the following line to set a custom metrics prefix
#   prefix:
#
#   OPTIONAL: Rename the satellite and/or client metrics
#     The satellite_prefix field has a default value of "satellite", and will be included in the name of all
#     metrics that reflect Satellite behavior.
#     Example:
#       prefix: "satellite"
#     [Docker]: COLLECTOR_STATSD_SATELLITE_PREFIX
#     Uncomment the following line to set a custom prefix for satellite metrics
#   satellite_prefix:
#
#     The client_prefix field has a default value of "client", and will be included as part of the name for all metrics
#     that reflect the behavior of the LightStep tracer client (these metrics are still exported via the Satellite).
#     Example:
#       prefix: "client"
#     [Docker]: COLLECTOR_STATSD_CLIENT_PREFIX
#     Uncomment the following line to set a custom prefix for client metrics
#   client_prefix:

# ----------------------- PORTS ---------------------

# Ports cannot be shared. All port numbers below must be unique.

# REQUIRED: Debug Port ("Babysitter")
# This port serves the following endpoint, which provides a status page with diagnostic information about the satellite.
#     /diagnostics
#
# The value is flexible and can be set to a different (unbound) port number as needed.
#
# [Docker]: -e COLLECTOR_BABYSITTER_PORT=port -p port:port
babysitter_port: 8000


# REQUIRED: Admin Ports: Health Checks / Diagnostics
# These ports serve the following endpoints:
#     /_ready    <== Load Balancers: use for health checks
#                        A 200 (OK) response indicates that the satellite is both responding AND healthy
#                        (able to handle incoming span traffic)
#     /_live     <== Orchestration tools (e.g. Kubernetes): use for liveness checks
#                        A 200 (OK) response indicates that the satellite is alive and responding, and
#                        the instance should not be terminated. However, it makes no promises about
#                        satellite health (it could be temporarily overloaded and not accepting spans)
#
# The values are flexible and can be set to different (unbound) port numbers as needed.
admin:

  # REQUIRED: Plaintext Admin Port
  # This value must always be set to an open port number in order for satellite diagnostics to work properly.
  # This is also the destination port for unencrypted health/readiness checks from a load balancer or orchestration framework.
  #
  # [Docker]: -e COLLECTOR_ADMIN_PLAIN_PORT=port -p port:port
  plain_port: 8180

  # OPTIONAL: Secure Admin Port
  # This port is only required if sending *encrypted* health/readiness checks to the satellite (this is not common).
  # If set to a non-zero value, the tls_cert_prefix option must also be configured (see below)
  #
  # [Docker]: -e COLLECTOR_ADMIN_SECURE_PORT -p port:port
  #
  # Uncomment the line below to enable the secure health check port.
  # secure_port: 9090


# REQUIRED: Proto-via-HTTP Ports
# These ports accept span traffic encoded as Protocol Buffers and sent to satellites via HTTP.
# The values are flexible and can be set to a different (unbound) port as needed.
#
# These ports are required in order to support the following LightStep tracer clients:
# - lightstep-tracer-go          (if configured as use_http: true)
# - lightstep-tracer-java        (if configured to use Protocol Buffers over HTTP)
# - lightstep-tracer-android     (if configured to use Protocol Buffers over HTTP)
#
# At least one of the following ports must be set in order to handle this encoding/transport combination.
# These ports are optional if you know that your satellites will never need to support these tracers,
# but it's better to set at least one of them for maximum flexibility.
#
# If either of these ports is set, the corresponding port in the 'grpc' section must also be configured.
http:

  # RECOMMENDED: Plaintext Proto-via-HTTP Port
  # Required for the satellite to accept *unencrypted* Proto-over-HTTP span traffic.
  # This port will also be used for encrypted span traffic if TLS termination happens at the load balancer
  # and data is not re-encrypted to the satellite.
  #
  # [Docker]: -e COLLECTOR_HTTP_PLAIN_PORT=port -p port:port
  plain_port: 8181

  # OPTIONAL: Secure Proto-via-HTTP Port
  # Same as above, but for *encrypted* span traffic where TLS termination happens at the satellite.
  # If set to a non-zero value, the tls_cert_prefix option must also be configured (see below)
  #
  # [Docker]: -e COLLECTOR_HTTP_SECURE_PORT=port -p port:port
  #
  # Uncomment the line below to enable the secure Proto-over-HTTP port.
  # secure_port: 9191


# REQUIRED: gRPC Ports
# At least one of these ports is required in order to accept span traffic via gRPC.
# The values are flexible and can be set to a different (unbound) port as needed.
#
# At least one of these ports is also REQUIRED in order to accept spans encoded as Protocol Buffers,
# even if the transport is HTTP (as above).
#
# These ports are required in order to support the following LightStep tracer clients:
# - lightstep-tracer-go          (if configured as use_grpc: true)
# - lightstep-tracer-java        (if configured to use gRPC)
# - lightstep-tracer-android     (if configured to use gRPC)
# - lightstep-tracer-cpp
#
# These ports are optional ONLY if you know that your satellites will never need to support any of these tracers,
# but it's better to set at least one of them for maximum flexibility.
grpc:

  # RECOMMENDED: Plaintext gRPC Port
  # Required for the satellite to accept *unencrypted* gRPC span traffic.
  # This port should also be used for secure connections if TLS termination happens at the load balancer
  # and data is not re-encrypted to the satellite.
  #
  # [Docker]: -e COLLECTOR_GRPC_PLAIN_PORT -p port:port
  plain_port: 8282

  # OPTIONAL: Secure gRPC Port
  # Same as above, but for *encrypted* gRPC span traffic where TLS termination happens at the satellite.
  # If set to a non-zero value, the tls_cert_prefix option must also be configured (see below)
  #
  # [Docker]: -e COLLECTOR_GRPC_SECURE_PORT=port -p port:port
  #
  # Uncomment the line below to enable the secure gRPC port.
  # secure_port: 9292


# REQUIRED: Thrift ports
# At least one of the following ports must be set in order to handle Thrift span traffic.
# The values are flexible and can be set to a different (unbound) port as needed.
#
# These ports are required in order to support the following LightStep tracer clients:
# - lightstep-tracer-go          (if configured as use_thrift: true)
# - lightstep-tracer-javascript
# - lightstep-tracer-objc
# - lightstep-tracer-php
# - lightstep-tracer-python
# - lightstep-tracer-ruby
#
# These ports are optional ONLY if you know that your satellites will never need to support any of these tracers.

# REQUIRED: Plaintext Thrift Port
# This port accepts *unencrypted* Thrift span traffic. It should also be used for secure connections if TLS termination
# happens at the load balancer and data is not re-encrypted to the satellite.
#
# [Docker]: -e COLLECTOR_PLAIN_PORT=port -p port:port
plain_port: 8383

# OPTIONAL: Secure Thrift Port
# This port accepts *encrypted* Thrift span traffic.
# If set to a non-zero value, the tls_cert_prefix option must also be configured (see below)
#
# [Docker]: -e COLLECTOR_SECURE_PORT=port -p port:port
#
# Uncomment the line below to enable the secure Thrift port.
# secure_port: 9393

# ----------------------- MEMORY ---------------------

# REQUIRED: Memory settings
# These settings define how much memory the satellite should dedicate to tracing data for each customer project.
# These settings directly affect satellite recall. More memory for a project => longer recall.
# 
# Note: The sum of (bytes_per_project + bytes_per_project_overrides) for all projects that report to a satellite
# should be set to approximately 1/4 of the available system memory.
#
# The example values below were computed based on our recommended machine size: 16GB RAM + 2 CPUs
#
# Example 1: Single Project
# When a satellite is only used for a single LightStep project, these settings should be configured as follows:
#
#   bytes_per_project: 100_000_000       # 100 MB (for any *other* project besides my_primary_project)
#   bytes_per_project_overrides:
#     my_primary_project: 3_500_000_000  # 3.5 GB (e.g. production)
#
# Example 2: Multiple Projects
# When a satellite serves multiple LightStep projects, these options control how the memory in each satellite
# is partitioned between those projects. These values should be set deliberately and may require guidance.
# 
#   bytes_per_project: 100_000_000       # 100 MB (for any *other* project besides those listed below)
#   bytes_per_project_overrides:
#     project_name_1: 3_000_000_000      #   3 GB (e.g. production instance)
#     project_name_2:   500_000_000      # 500 MB (e.g. staging instance)
#
reporter:
  # [Docker]: -e COLLECTOR_REPORTER_BYTES_PER_PROJECT=number
  bytes_per_project: 100_000_000

  # [Docker]: -e COLLECTOR_REPORTER_BYTES_PER_PROJECT_OVERRIDES={"project_name_1":number, "project_name_2":number}
  bytes_per_project_overrides:
  #  project_name_1: 3_000_000_000
  #  project_name_2:   500_000_000

# ----------------------- TLS ---------------------

# OPTIONAL: TLS certificate path
# Required if using any of the SecurePorts in order to provide TLS termination at the satellite.
# This string gives the file path prefix for certificate files to be used in serving TLS connections on the satellite.
# Example:
#   for certificate files located at /root/certs/mydomain.bundle.pem and /root/certs/mydomain.key.pem,
#   set the path prefix to `/root/certs/mydomain`.
#
# [Docker]: -e COLLECTOR_TLS_CERT_PREFIX=path
#
# Uncomment the line below (and set appropriately) if using any of the secure ports above.
# tls_cert_prefix: /root/certs/mydomain

Docker Image

A public repository on Docker Hub offers the latest versions of the Satellite. Configuration is provided using environment variables. Please see the configuration template above for a full listing and expanded explanations of variables and recommended settings.

docker run -e COLLECTOR_API_KEY={your_API_key} [-p PORT:PORT...] [-e ENV_VARIABLE ...] lightstep/collector:latest

After startup, check the diagnostics page (http://{satellite_host}:8000/diagnostics) to view the Satellite configuration. When starting from command line as in the above example, the configuration will also be directed to stdout, which can be inspected for troubleshooting if the diagnostics page isn't reachable.

AWS AMI

The LightStep satellite is available as an AMI in several AWS regions:

The AMI will start the Satellite binary automatically when an instance boots.

When creating instances, set user data to configure Satellites. If you are using the console, user data can be found under “Configure Instance Details” and “Advanced Details.” User data should be entered into the console as YAML-formatted text such as:

api_key:{your_API_key}
pool: us-east-1
[other required settings per the Configuration Template]
api_key: {your_API_key}
tls_cert_prefix: /root/certs/mydomain
grpc:
  secure_port: 443
[other required settings per the Configuration Template]

(This assumes that the Satellite can find certificates and a key at /root/certs/mydomain.bundle.pem and /root/certs/mydomain.key.pem.)

Ubuntu Package

The LightStep Satellite is available in Debian package format for Ubuntu 16.04 Xenial users. You may obtain that Debian package here

Presently, Ubuntu packages are not distributed via an apt-get repository. The LightStep Satellite is signed and will be verified automatically by dpkg

To install the package:

$ dpkg -i lightstep-collector_{version}_amd64.deb

The package places a configuration file in /etc/lightstep/collector.yaml with the same format described for AWS user data above. That file also contains a description of how to set the Satellite variables.

To start the server manually:

The LightStep Satellite is configured to run at startup via an upstart config:
/etc/init/lightstep-collector.conf

If needed, the satellite can also be started manually:

$ service lightstep-collector start
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQENBFkU0fIBCACyB6QycEWl0xPrQYwAHgWfzODbp1Kf3HsRq+3OC+eM6QUaj2jT
niVmeHzfxkfvgK8kVaNfoxJDorUwGb8iltaR7JnFSTPw9AcGc56ZVmTtg7V6YYCg
uakNM1uh2ue/jbFfoL6/4UbvFINWyVAmTWA9/UELHPAAeMosDUW4ltRdLmtl3W8W
q7VTSM76TA/hfY+slr8jDJcN4mOD9Hx8LoGJP7t8fhpEIlo9pW+seFQwo6lvk6sK
5PHF72BuaWOivvNjvJbJpbOimDET54sNJf+rVVWoljszCRvTahAj3LpBHGOzeORr
L6ux3WOrDUW1FN9vJKeL0GLKirf54cby4oltABEBAAG0KUxpZ2h0U3RlcCBTdXBw
b3J0IDxzdXBwb3J0QGxpZ2h0c3RlcC5jb20+iQE4BBMBAgAiBQJZFNHyAhsDBgsJ
CAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDpq4dbJcMAWAgvCACue3+qR1cavJDc
31MVVNNOZblMlUd+aC1u9s9chG3As28V0vn5O+6tWSE93r+k0FUj5GDiFXKv9g4E
NpKRPJor0CKsa28qsBESNyNrgAPsr5I6WptYveLn39MQfDZGHEDrasLDGxkeLklB
UCD4zqRy4jNnk39IqVtnNZY4i/3asGfKexfv8YqJTNn+oMZ9zXDraOuoWsivbtt0
8/4RTEGY2U44TmxWjDr4/tdjkCRgi60h6q5y+C2++bntvrAGgiuWb/b9u5acbqm+
MWwLV6ltgb7bR0/t2HkZBAe43NkMrYw0jE7dc9TTI+Ik7NPZF1fuWTbMdRUcWp+K
iBL4QqicuQENBFkU0fIBCADNSWsIn2Z8F3QmI8ZqjD6768Lr03ahJO2oQ5rFXm17
inDsMLByDccRcdlZzjUowOBbunw+RMG4BknotqNo9i7sUQ1D3oPhNMR4TBeJqeYb
2BsURq0AJaofDiDKwJvmdaG9zW4ZVoYNZ3tM980uH+R9mIuoopQvmTGk+H0Ro9M7
+dtj1slZ4T5+HS0NY6vuyFQSg3JBKkt0zzHewGDKC56I8Fg0ijjoe1xGkkiJ951S
HoMWrDgFkN9xcrEKwCYO6rSgHDR3rNkf7vQSMpejFT1f+430qG/tPwUlWlESJf4q
zdRdCDS9WD7TmWZLD3fv0IZHUMePPZRg20zW+7aWbGCTABEBAAGJAR8EGAECAAkF
AlkU0fICGwwACgkQ6auHWyXDAFihHwf8D/y3PI5P8wUdvRtFIifwThYtzVxj5HYV
sIG9lOGlU9LKQPwMAPJrZ+frkg2/cr34MbRRPcuhnFnQMhCLT/3NwjZHDS7n6YJg
wXXfy0wJ71bejtD0w00iYyBCRoBztoEGZcyFex6EWfowZuR/VJLVuDRUvOJ0o8dv
96mG57Tp5SQVETtCWBrSWlF/OpRdunXTSE32uXw/teYafb5u7ximdcImc7l/PbdX
UrXlENGvdGDHIELLnGebrpEK4AhJB/dctWfHvPO9HrXOfGWDcDC9nVuRW6ESFNBh
ECQOgPL1icpZm5YBfXOXmOszpU1ZCtOyrQo8RCkrU95vEEzbl5WIOA==
=qtrh
-----END PGP PUBLIC KEY BLOCK-----

Configure Tracers

LightStep client libraries must be configured with the location of your Satellites. If you are using a single Satellite, provide the DNS name or IP address of that Satellite. Otherwise, use the name or address of the load balancer you've deployed above. By default, client libraries use secure connections, so you must also disable that feature.

Client configuration is platform-specific. Two examples are shown below, but see the LightStep client documentation for a complete list of platforms.

To use the Go tracer with an on-prem pool, fill in your access token and the Satellite host as shown here:

lightstepTracer := lightstep.NewTracer(lightstep.Options{
   AccessToken: "{your_access_token}",
   Collector: lightstep.Endpoint{
       Host:      "{your_load_balancer_DNS_name_or_IP_address}",
       Plaintext: true,
   },
})

Similarly, fill in those values below to configure the Node.js tracer.

Tracer.initGlobalTracer(LightStep.tracer({
   access_token         : '{your_access_token}',
   collector_host       : '{your_load_balancer_DNS_name_or_IP_address}',
   collector_encryption : 'none',
}));