Lightstep can ingest traces from the Datadog Agent natively. You can choose to “tee” your traces to both Lightstep and Datadog, or send them only to Lightstep. This document covers the configuration for each use case.

Requirements

Lightstep supports both Datadog Agent major versions 6 and 7 tracks.
The following minimum versions are required:

  • Major version 6: Minimum version of 6.18.0 (released March 16, 2020)
  • Major version 7: Minimum version of 7.18.0 (released March 16, 2020)

Major version 5 of the Datadog Agent is not supported.

Prerequisites

  • The Lightstep access token for the project you want to report traces to.
  • (Optional, if you want to send traces to both Datadog and Lightstep) A Datadog API key.
  • Configure required and recommended attributes for Datadog tracing clients.

    Lightstep’s datadog trace ingest requires the lightstep.service_name attribute to be set on all incoming spans. We also recommend setting the lightstep.service_version attribute. Lightstep does not currently support setting these tags on the Datadog Agent. Instead they must be configured at the tracing client level (e.g. dd-trace-py, etc). Each tracing client has various in-code configuration options, but the most consistent and straightforward way to set these is using the DD_TRACE_GLOBAL_TAGS environment variable.

1
  DD_TRACE_GLOBAL_TAGS="lightstep.service_name:<service_name>,lightstep.service_version:<service_version>"

DD_TRACE_GLOBAL_TAGS is to be set on the Datadog tracing clients, not the agent.

Send Traces to Both Lightstep and Datadog

To send tracing data to Lightstep and to Datadog, you need to add or update the additional_endpoints property. It takes a key value pair, where the key is the endpoint to your Microsatellite environment (where the data is collected) and the value is the Lightstep access token. You can configure this using either YAML or environment variables.

Use YAML to Configure the Agent

  1. In your datadog.yml file, add the additional_endpoints property below the apm_config section.
    Replace <EXISTING_DATADOG_API_KEY>with your value.

If your environment uses public Microsatellites, use the values shown in the example for the trace endpoint. For on-premise Microsatellites, use the Microsatellite’s IP address.

Start tabs

Public Microsatellites

1
2
3
4
5
6
api_key: <EXISTING_DATADOG_API_KEY>

apm_config:
  additional_endpoints:
    'https://ingest.lightstep.com':
      - <LIGHTSTEP_ACCESS_TOKEN>

On-Premise Microsatellites

1
2
3
4
5
6
api_key: <EXISTING_DATADOG_API_KEY>

apm_config:
  additional_endpoints:
    '<MICROSATELLITE_ENDPOINT>':
      - <LIGHTSTEP_ACCESS_TOKEN>

End code tabs

  1. Redeploy the configuration file and restart the Datadog Agent.
    At startup, look for a log message containing Forwarder started;. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep).

Use Environment Variables to Configure the Agent

To send traffic to Lightstep from the Datadog Agent using environment variables, you need to set the DD_APM_ADDITIONAL_ENDPOINTS environment variable. The environment variable value should be formatted as a JSON object with the endpoint as the key and the Lightstep access token as a string.

Replace <LIGHTSTEP_ACCESS_TOKEN> with your value.

  1. Set the DD_APM_ADDITIONAL_ENDPOINTS variable with the appropriate endpoint for your Microsatellite environment. If your environment uses public Microsatellites, use the values shown in the example. For on-premise Microsatellites, use the Microsatellite’s IP address.
    Replace <LIGHTSTEP_ACCESS_TOKEN> with your value.

Start tabs

Public Microsatellites

1
export DD_APM_ADDITIONAL_ENDPOINTS='{"https://ingest.lightstep.com": ["<LIGHTSTEP_ACCESS_TOKEN>"]}'

On-Premise Microsatellites

1
export DD_APM_ADDITIONAL_ENDPOINTS='{"<MICROSATELLITE_ENDPOINT>": ["<LIGHTSTEP_ACCESS_TOKEN>"]}'

End code tabs

  1. Restart the Datadog Agent.
    At startup, look for a log message containing Forwarder started;. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep).

Configure the Datadog Agent to Send Traces to Lightstep Only

If you want to send traces only to Lightstep (and not Datadog), set the api_key variable to your Lightstep access token and update the dd_url and apm_config.apm_dd_url based on your Microsatellite environment, and disable the process agent. If your environment uses public Microsatellites, use the values shown in the example. For on-premise Microsatellites, use the Microsatellite’s IP address.

Setting the dd_url to Lightstep sends metric data to Lightstep instead of Datadog. If not set to a Lightstep Microsatellite endpoint, the data along with your Lightstep access token is sent to Datadog, which is strongly discouraged.

  1. In your Datadog configuration file, replace api_key and dd_url and apm_config.apm_dd_url as follows:

Start tabs

Public Microsatellites

1
2
3
4
5
6
api_key: <YOUR LIGHTSTEP_ACCESS_TOKEN>
dd_url: 'https://metricingest.lightstep.com'
apm_config:
  apm_dd_url: 'https://ingest.lightstep.com'
process_config:
  enabled: disabled

On-Premise Microsatellites

1
2
3
4
5
6
api_key: <YOUR LIGHTSTEP_ACCESS_TOKEN>
dd_url: '<MICROSATELLITE_ENDPOINT>'
apm_config:
  apm_dd_url: '<MICROSATELLITE_ENDPOINT>'
process_config:
  enabled: disabled

End code tabs

  1. Redeploy the configuration file and restart the Datadog Agent.