Lightstep Observability can ingest traces from the Datadog Agent natively. You can choose to “tee” your traces to both Lightstep Observability and Datadog, or send them only to Lightstep Observability. This document covers the configuration for each use case.

Requirements

Lightstep Observability supports both Datadog Agent major versions 6 and 7 tracks.
The following minimum versions are required:

  • Major version 6: Minimum version of 6.18.0 (released March 16, 2020)
  • Major version 7: Minimum version of 7.18.0 (released March 16, 2020)

Major version 5 of the Datadog Agent is not supported.

Prerequisites

  • The Lightstep Observability access token for the project you want to report traces to.
  • (Optional, if you want to send traces to both Datadog and Lightstep Observability) A Datadog API key.

  • Configure required and recommended attributes for Datadog tracing clients.

    Lightstep Observability’s datadog trace ingest requires the lightstep.service_name attribute to be set on all incoming spans. We also recommend setting the lightstep.service_version attribute. Lightstep Observability does not currently support setting these tags on the Datadog Agent. Instead they must be configured at the tracing client level (e.g. dd-trace-py, etc). Each tracing client has various in-code configuration options, but the most consistent and straightforward way to set these is using the DD_TRACE_GLOBAL_TAGS environment variable.

1

  DD_TRACE_GLOBAL_TAGS="lightstep.service_name:<service_name>,lightstep.service_version:<service_version>"

DD_TRACE_GLOBAL_TAGS is to be set on the Datadog tracing clients, not the agent.

Send traces to both Lightstep Observability and Datadog

To send tracing data to Lightstep Observability and to Datadog, you need to add or update the additional_endpoints property. It takes a key value pair, where the key is the endpoint to your Microsatellite environment (where the data is collected) and the value is the Lightstep Observability access token. You can configure this using either YAML or environment variables.

Use YAML to configure the Agent

  1. In your datadog.yaml file, add the additional_endpoints property in the Basic Configuration section, and configure it with the URL for Lightstep metrics ingest and your Lightstep access token.

If your environment uses public Microsatellites, use the values shown in the example for the trace endpoint. For on-premise Microsatellites, use the Microsatellite’s IP address.

Start tabs

Public Microsatellites

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#########################
## Basic Configuration ##
#########################

api_key: <EXISTING_DATADOG_API_KEY>


####################################
## Trace Collection Configuration ##
####################################

apm_config:
  additional_endpoints:
    'https://ingest.lightstep.com':
      - <LIGHTSTEP_ACCESS_TOKEN>

On-Premise Microsatellites

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#########################
## Basic Configuration ##
#########################

api_key: <EXISTING_DATADOG_API_KEY>


####################################
## Trace Collection Configuration ##
####################################

apm_config:
  additional_endpoints:
    '<MICROSATELLITE_ENDPOINT>':
      - <LIGHTSTEP_ACCESS_TOKEN>

End code tabs

  1. Redeploy the configuration file and restart the Datadog Agent.
    At startup, look for a log message containing Forwarder started;. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep Observability).

Use environment variables to configure the Agent

The Datadog Docker Agent, the containerized version of the Datadog Agent, supports configuration via environment variables.

  1. Set the DD_APM_ADDITIONAL_ENDPOINTS variable with the appropriate endpoint for your Microsatellite environment. If your environment uses public Microsatellites, use the values shown in the example below. For on-premise Microsatellites, use the Microsatellite’s IP address.
    Replace <LIGHTSTEP_ACCESS_TOKEN> with your value.

Start tabs

Docker Run

1

-e DD_APM_ADDITIONAL_ENDPOINTS='{"https://ingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'

Kubernetes

1
2
3

env:
  - name: DD_APM_ADDITIONAL_ENDPOINTS
    value: '{"https://ingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'

Docker Compose

1
2
3
4

services:
  datadog:
    environment:
      - DD_APM_ADDITIONAL_ENDPOINTS='{"https://ingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'

End code tabs

  1. Restart the Datadog Agent.
    At startup, look for a log message containing Forwarder started;. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep Observability).

Configure the Datadog Agent to send traces to Lightstep Observability only

If you want to send traces only to Lightstep Observability (and not Datadog), set the api_key variable to your Lightstep Observability access token and update the dd_url and apm_config.apm_dd_url based on your Microsatellite environment, and disable the process agent. If your environment uses public Microsatellites, use the values shown in the example. For on-premise Microsatellites, use the Microsatellite’s IP address.

Setting the dd_url to Lightstep Observability sends metric data to Lightstep Observability instead of Datadog. If not set to a Microsatellite endpoint, the data along with your Lightstep Observability access token is sent to Datadog, which is strongly discouraged.

  1. In your Datadog configuration file, replace api_key and dd_url and apm_config.apm_dd_url as follows:

Start tabs

Public Microsatellites

1
2
3
4
5
6

api_key: <YOUR LIGHTSTEP_ACCESS_TOKEN>
dd_url: 'https://metricingest.lightstep.com'
apm_config:
  apm_dd_url: 'https://ingest.lightstep.com'
process_config:
  enabled: disabled

On-Premise Microsatellites

1
2
3
4
5
6

api_key: <YOUR LIGHTSTEP_ACCESS_TOKEN>
dd_url: '<MICROSATELLITE_ENDPOINT>'
apm_config:
  apm_dd_url: '<MICROSATELLITE_ENDPOINT>'
process_config:
  enabled: disabled

End code tabs

  1. Redeploy the configuration file and restart the Datadog Agent.