2020-09-24_05-22-16Z

This release includes the following changes:

  • OTLP ingest now populates the span.kind field.
  • Fixes Trace ID truncation in OTLP ingest. Truncation was causing propagation issues for services instrumented with a mix of OpenTelemetry and legacy tracer libraries.
  • Adds proxying for infrastructure metrics. You can now send metrics through the satellites instead of using ingest.lightstep.com, simplifying network access configuration.
  • Adds support for hex encoded Trace and Span IDs in JSON-encoded OTLP.
  • Adds support for Datadog V5 trace format.
  • Fixes bug where dashes (-) were not accepted characters in StatsD prefixes.

2020-07-22_17-02-58Z

This release includes the following changes:

  • Errors from Zipkin now use the error:true tag. The original Zipkin error tag (error_zipkin) has been renamed to error_zipkin_original.
  • Added support for errors in OpenTelemetry by setting the error tag to true when span’s status code is non-0.

  • Upgraded to OTLP version 0.4.0.

    This upgrade is not backwards-compatible with earlier versions. If your services use an earlier version of OTLP, you should not upgrade your Satellites until you upgrade OTLP.

2020-06-23_19-47-23Z

This release fixes a stability issue found in the previous release
(2020-06-01_20-22-15Z).
There are no new features.

2020-06-01_20-22-15Z

This release includes the following changes:

  • Data from the Satellites to the Lightstep Hypothesis Engine now uses better compression.
  • If you use our auto-installers for auto-instrumentation, you can now configure how you want to handle conversion of the value used for an operation’s name. You can use the resource or name parameter, or both. If you set both fields, Lightstep uses the first one with a value.
  • Satellites now support trace data sent using the OpenTelemetry Protocol (OTLP).
  • Updated Go to version 1.14.3

2020-03-03_06-00-29Z

This release includes the following changes:

  • Support for the lightstep.guid tag as the reporter ID for Datadog and Zipkin tracer integrations. The Distinct service instances metric is now accurate for these integrations.

  • Now using span.resource as the operation name (previously span.name) for Datadog integrations. Operations reporting through Datadog will have different names.

    If you previously used span.resource to create Streams, consider creating a query to include the new span.resource tag.
    For example, if you had:
    service IN ("web") AND operation IN ("/api/get-store")
    Create a new stream based on this:
    service IN ("web") AND operation IN ("/api/get-store", "get-store-api")

    The new stream will not include historical data.

  • Support for the Rollout Analysis feature that allows you to view performance for partial releases (like canary or A/B tests), in the Service Health view.

  • Updated Go to version 1.14.0.

2019-10-25_22-47-11Z

This release includes the following changes:

  • api_key in the configuration has been deprecated and replaced with satellite_key to make it more clear which key to set.
  • Zipkin Integration: Satellites can now ingest data from Zipkin tracer libraries
  • Datadog Integration: Satellites can now ingest data from Datadog tracer libraries

2019-08-05_21-29-55Z

This release includes the following changes:

  • Jaeger Integration : The satellite is now capable of ingesting data from the Jaeger agent.

2019-06-12_16-06-52Z

This release includes the following changes:

  • Satellites now support access-token free reporting for a single project. Two arguments are required: COLLECTOR_DISABLE_ACCESS_TOKEN_CHECKING=true COLLECTOR_PROJECT_NAME=${PROJECT}

Combined with any Satellite API key, this functionality allows dedicated reporting to any one project in the organization.

2019-03-04_19-38-40Z

This release includes the following changes:

  • Ingestion tags are available for applying static tags to spans via Satellite configuration. The environment variable COLLECTOR_INGESTION_TAGS (Docker env variable) may be set to a comma-separated tag:value list to add ingestion tags to all spans received by the Satellite. These tags are applied to all Spans ingested by the Satellite and are are visible in traces. Yaml: collector.ingestion_tags. Example: COLLECTOR_INGESTION_TAGS=myTagKey1:myTagValue1,myTagKey2:myTagValue:2
  • All Satellite ports that previously accepted transport-specific report requests now support all transports. This means you can now send both both gRPC-encoded and Thrift-encoded spans to the same port. In the future, we may consolidate the number of ports. Note that if you previously needed to use an L4 balancer for gRPC traffic, that will not change.

2019-02-01_21-17-58Z

This release includes the following changes:

New configuration option: guid. To assist with matching Satellite information in the Lightstep application with instances running in your environment. Specifically, this will be displayed on the Satellites page when viewing a list of Satellites in a particular pool. It is recommended that the identifier you choose be globally unique identifier. As with all configuration options, it may be specified via yaml, environment variable or flag.

  • Example flag: -e COLLECTOR_GUID=$HOSTNAME
  • Example environment variable: export COLLECTOR_GUID=$HOSTNAME
  • Example yaml: guid: xxx.x.x.x

2018-11-19_17-15-06Z

This release includes the following changes:

  • Stability and troubleshooting improvements.
  • Results from the Power-On Self-Test now always appear in the local logs.
  • New metric for monitoring failed Satellite reports due to invalid Access Token:
    • StatsD: <prefix>.<satellite_prefix>.access_tokens.invalid
    • Datadog: <prefix>.<satellite_prefix>.access_tokens.invalid
    • Definition: Number of reports (i.e., batches of spans) that have been rejected by the satellite due to an invalid access token.
      Example: org_name.satellite.access_tokens.invalid
    • Type: Count

  • Logging to syslog has been added as a beta feature. It is disabled by default, and can be enabled by the following in the YAML config file:

    1
2
3

    logging:
  syslog_config:
    enabled: true

    or by the following Docker environment variable: COLLECTOR_LOGGING_SYSLOG_CONFIG_ENABLED=true

  • Deprecated config fields: grpc.on_secure_port and grpc.insecure. This option allowed hosting grpc traffic on the secure port and used grpc.insecure to determine whether that traffic should use TLS. Now, configuration will need to explicitly set the grpc.secure_port or the grpc.plain_port. To upgrade, use the grpc.insecure value to determine which port must be set (plain or secure) and remove the secure_port config option and replace it with the appropriate grpc port option. See the table below for actions necessary.
  grpc.on_secure_port = true grpc.on_secure_port = false
grpc.insecure = true Move the config value for secure_port to grpc.plain_port. For secure traffic, direct that traffic at the grpc.secure_port; for insecure traffic, direct that traffic at grpc.plain_port.
grpc.insecure = false Move the config value for secure_port to grpc.secure_port. No action required.

2018-10-09_20-00-32Z

This release supports several new metrics-related features:

  • Plain statsd support
  • Support for custom metric prefixes
  • Support for custom DataDog tags

This release requires customers to make several changes in how they enable metrics, and changes some of the metric names.

Action Required: Enable Satellite Metrics

Customers who use the old Satellite DataDog metrics will need to make changes in their satellite config file or Docker environment variables per the chart below.

Prefixes

  • The prefix field is empty by default, but will be applied as a prefix to the name of all metrics if set to a non-empty value
  • The satellite_prefix field has a default value of “satellite”, and will be included in the name of all Satellite metrics
  • The client_prefix field has a default value of “client”, and will be included as part of the name for all client metrics

Naming Rules

Optional prefixes and tags should follow these rules, unless your metrics service provider supports additional characters:

  • Start with a letter
  • Must only contain ASCII alphanumerics (lowercase recommended), underscores, and periods
CurrentNew (for DataDog users)New (for Statsd users)

AWS or Debian / Ubuntu

datadog:
        host: 127.0.0.1
        port: 8125

AWS or Debian / Ubuntu

statsd:
        # Required
        host: 127.0.0.1
        port: 8125
        export_dogstatsd: true

        # Recommended
        prefix: lightstep

        # Optional
        satellite_prefix: <string>
        client_prefix: <string>
        dogstatsd_tags: "K1:V1,K2:V2"

AWS or Debian / Ubuntu

statsd:
      # Required
      host: 127.0.0.1
      port: 8125
      export_statsd: true

      # Recommended
      prefix: lightstep

      # Optional
      satellite_prefix: <string>
      client_prefix: <string>

Docker

COLLECTOR_DATADOG_HOST=127.0.0.1
      COLLECTOR_DATADOG_PORT=8125

Docker

# Required
      COLLECTOR_STATSD_HOST=127.0.0.1
      COLLECTOR_STATSD_PORT=8125
      COLLECTOR_STATSD_EXPORT_DOGSTATSD=true

      # Recommended
      COLLECTOR_STATSD_PREFIX=lightstep

      # Optional
      COLLECTOR_STATSD_SATELLITE_PREFIX=<string>
      COLLECTOR_STATSD_CLIENT_PREFIX=<string>
      COLLECTOR_STATSD_DOGSTATSD_TAGS="K1:V1,K2:V2"

Docker

# Required
      COLLECTOR_STATSD_HOST=127.0.0.1
      COLLECTOR_STATSD_PORT=8125
      COLLECTOR_STATSD_EXPORT_STATSD=true

      # Recommended
      COLLECTOR_STATSD_PREFIX=lightstep

      # Optional
      COLLECTOR_STATSD_SATELLITE_PREFIX=<string>
      COLLECTOR_STATSD_CLIENT_PREFIX=<string>

Some Examples:

ExampleDataDogStatsd

Customize all prefixes and tags

statsd:
      host: 127.0.0.1
      port: 8125
      export_dogstatsd: true

      prefix: "lightstep"
      satellite_prefix: "satellite"
      client_prefix: "client"
      dogstatsd_tags:
      "env:prod,pool:us-west-1,canary:true"
      
statsd:
      host: 127.0.0.1
      port: 8125
      export_statsd: true

      prefix: "lightstep.prod.us-west-1"
      satellite_prefix: satellite-canary
      client_prefix: client-via-canary

Customize the basic prefix and tags

statsd:
      host: 127.0.0.1
      port: 8125
      export_dogstatsd: true

      prefix: "lightstep"
      dogstatsd_tags: "env:prod,pool:east"
      
statsd:
      host: 127.0.0.1
      port: 8125
      export_statsd: true

      prefix: "lightstep.prod.east"

Action Required: Update Monitoring for new Metric Names

The names of client and satellite metrics have changed in this release, as shown in the chart below.

Note on lightstep_project:

  • Many of these metrics are automatically labeled with a Lightstep project name, so the resulting timeseries can be grouped by project, if desired.
  • For basic statsd metrics, the lightstep_project becomes part of the metric name itself, e.g. satellite.spans.received.my_lightstep_project_name
  • For DataDog metrics, the project name is attached via a tag called lightstep_project on the relevant metrics. The syntax to indicate a tag is {tag_name}
Current NameNew NameDefinition

client.spans.dropped

Statsd

<prefix>.<client_prefix>.
      spans.dropped.
      <lightstep_project>

DataDog

<prefix>.<client_prefix>.
      spans.dropped
      {lightstep_project}

Number of spans that are dropped at the client because the outgoing queue is full, still trying to send earlier spans to a satellite.

Values are cumulative and can be aggregated across satellites and projects.

Type: Count

collector.reports.inbound.thrift.bytes

<prefix>.<satellite_prefix>.
      bytes.received.thrift

Total bytes of Thrift span traffic received over the network by the satellite.

Values are cumulative and can be aggregated across satellites and projects.

Type: Count

collector.reports.inbound.grpc.bytes

<prefix>.<satellite_prefix>.
      bytes.received.grpc

Total bytes of gRPC span traffic received over the network by the satellite.

Values are cumulative and can be aggregated across satellites and projects.

Type: Count

collector.spans.received

Statsd

<prefix>.<satellite_prefix>.
      spans.received.
      <lightstep_project>
 

DataDog

<prefix>.<satellite_prefix>.
      spans.received
      {lightstep_project}

Total number of spans that have been received and decoded by the satellite. This value includes any spans that may yet be dropped by the Satellite due to insufficient resources. See <satellite_prefix>.spans.dropped for more information.

 

Values are cumulative and can be aggregated across satellites and projects.

Type: Count

collector.spans.dropped

Statsd

<prefix>.<satellite_prefix>.
      spans.dropped.
      <lightstep_project>

DataDog

<prefix>.<satellite_prefix>.
      spans.dropped.
      {lightstep_project}

Total number of spans that are dropped by the satellite due to insufficient resources, after being received and decoded by the satellite. These spans are not indexed or added to the statistics for streams.

If this value is consistently above zero, first verify that the bytes_per_project_override settings match the recommended values here. If the problem persists, try adding more satellites.

Values are cumulative and can be aggregated across satellites and projects.

Type: Count

collector.reports.queued

Statsd

<prefix>.<satellite_prefix>.
      index.queue.length.
      <lightstep_project>

DataDog

<prefix>.<satellite_prefix>.
      index.queue.length.
      {lightstep_project}

Number of reports (i.e., batches of spans) that have been read from the network and are currently waiting to be indexed.

This value is instantaneous (non-cumulative).

Type: Gauge

collector.reports.bytes.queued

Statsd

<prefix>.<satellite_prefix>.
      index.queue.bytes.
      <lightstep_project>

DataDog

<prefix>.<satellite_prefix>.
      index.queue.bytes.
      {lightstep_project}

Number of bytes worth of reports that are currently waiting to be indexed (size of index.queue.length in bytes).

This value is instantaneous (non-cumulative).

Type: Gauge

collector.spans.indexed

Statsd

<prefix>.<satellite_prefix>.
      spans.indexed.
      <lightstep_project>

DataDog

 

<prefix>.<satellite_prefix>.
spans.indexed.
{lightstep_project}

 

Number of spans that have been indexed by the satellite and added to stream statistics.

Values are cumulative and can be aggregated across instances and projects.

Type: Count

collector.buffer.recall.seconds

Statsd

<prefix>.<satellite_prefix>.
      current.recall.seconds.
      <lightstep_project>

DataDog

<prefix>.<satellite_prefix>.
      current.recall.seconds.
      {lightstep_project}

Number of seconds between now and the oldest span that is still indexed in satellite memory. This indicates how much history is currently available to facilitate trace assembly and Live View.

If this value is consistently below 5 minutes, or sometimes dips below 3 minutes, first verify that your bytes_per_project_overrides settings match the recommended values here, then check whether the recall number is consistent across your satellites (if not, check your load balancing settings). If the problem persists, try adding more satellites.

Values are instantaneous (non-cumulative) and aggregation across instances and/or projects is only meaningful with a "minimum" operator.

Type: Gauge

Users who already have dashboards or alerts configured can keep a few of the old metric names by not setting prefix, and by setting satellite_prefix to “collector”. However this will only work for the subset of metrics whose names have not changed.

2018-08-09_04-56-31Z

This release focuses on bug fixes and improving stability and reliability.