Lightstep Observability can ingest traces from the Datadog Agent natively. You can choose to “tee” your traces to both Lightstep Observability and Datadog, or send them only to Lightstep Observability. This document covers the configuration for each use case.
Requirements
Lightstep Observability supports both Datadog Agent major versions 6 and 7 tracks.
The following minimum versions are required:
- Major version 6: Minimum version of 6.18.0 (released March 16, 2020)
- Major version 7: Minimum version of 7.18.0 (released March 16, 2020)
Major version 5 of the Datadog Agent is not supported.
Prerequisites
- The Lightstep Observability access token for the project you want to report traces to.
- (Optional, if you want to send traces to both Datadog and Lightstep Observability) A Datadog API key.
-
Configure required and recommended attributes for Datadog tracing clients.
Lightstep Observability’s datadog trace ingest requires the
lightstep.service_name
attribute to be set on all incoming spans. We also recommend setting thelightstep.service_version
attribute. Lightstep Observability does not currently support setting these tags on the Datadog Agent. Instead they must be configured at the tracing client level (e.g.dd-trace-py
, etc). Each tracing client has various in-code configuration options, but the most consistent and straightforward way to set these is using theDD_TRACE_GLOBAL_TAGS
environment variable.
1
DD_TRACE_GLOBAL_TAGS="lightstep.service_name:<service_name>,lightstep.service_version:<service_version>"
DD_TRACE_GLOBAL_TAGS
is to be set on the Datadog tracing clients, not the agent.
Send traces to both Lightstep Observability and Datadog
To send tracing data to Lightstep Observability and to Datadog, you need to add or update the additional_endpoints
property. It takes a key value pair, where the key is the endpoint to your Microsatellite environment (where the data is collected) and the value is the Lightstep Observability access token. You can configure this using either YAML or environment variables.
Use YAML to configure the Agent
- In your
datadog.yaml
file, add theadditional_endpoints
property in the Basic Configuration section, and configure it with the URL for Lightstep metrics ingest and your Lightstep access token.
If your environment uses public Microsatellites, use the values shown in the example for the trace endpoint. For on-premise Microsatellites, use the Microsatellite’s IP address.
Start tabs
Public Microsatellites
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#########################
## Basic Configuration ##
#########################
api_key: <EXISTING_DATADOG_API_KEY>
####################################
## Trace Collection Configuration ##
####################################
apm_config:
additional_endpoints:
'https://ingest.lightstep.com':
- <LIGHTSTEP_ACCESS_TOKEN>
On-Premise Microsatellites
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#########################
## Basic Configuration ##
#########################
api_key: <EXISTING_DATADOG_API_KEY>
####################################
## Trace Collection Configuration ##
####################################
apm_config:
additional_endpoints:
'<MICROSATELLITE_ENDPOINT>':
- <LIGHTSTEP_ACCESS_TOKEN>
End code tabs
- Redeploy the configuration file and restart the Datadog Agent.
At startup, look for a log message containingForwarder started;
. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep Observability).
Use environment variables to configure the Agent
The Datadog Docker Agent, the containerized version of the Datadog Agent, supports configuration via environment variables.
- Set the
DD_APM_ADDITIONAL_ENDPOINTS
variable with the appropriate endpoint for your Microsatellite environment. If your environment uses public Microsatellites, use the values shown in the example below. For on-premise Microsatellites, use the Microsatellite’s IP address.
Replace<LIGHTSTEP_ACCESS_TOKEN>
with your value.
Start tabs
Docker Run
1
-e DD_APM_ADDITIONAL_ENDPOINTS='{"https://ingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'
Kubernetes
1
2
3
env:
- name: DD_APM_ADDITIONAL_ENDPOINTS
value: '{"https://ingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'
Docker Compose
1
2
3
4
services:
datadog:
environment:
- DD_APM_ADDITIONAL_ENDPOINTS='{"https://ingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'
End code tabs
- Restart the Datadog Agent.
At startup, look for a log message containingForwarder started;
. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep Observability).
Configure the Datadog Agent to send traces to Lightstep Observability only
If you want to send traces only to Lightstep Observability (and not Datadog), set the api_key
variable to your Lightstep Observability access token and update the dd_url
and apm_config.apm_dd_url
based on your Microsatellite environment, and disable the process agent. If your environment uses public Microsatellites, use the values shown in the example. For on-premise Microsatellites, use the Microsatellite’s IP address.
Setting the dd_url
to Lightstep Observability sends metric data to Lightstep Observability instead of Datadog. If not set to a Microsatellite endpoint, the data along with your Lightstep Observability access token is sent to Datadog, which is strongly discouraged.
- In your Datadog configuration file, replace
api_key
anddd_url
andapm_config.apm_dd_url
as follows:
Start tabs
Public Microsatellites
1
2
3
4
5
6
api_key: <YOUR LIGHTSTEP_ACCESS_TOKEN>
dd_url: 'https://metricingest.lightstep.com'
apm_config:
apm_dd_url: 'https://ingest.lightstep.com'
process_config:
enabled: disabled
On-Premise Microsatellites
1
2
3
4
5
6
api_key: <YOUR LIGHTSTEP_ACCESS_TOKEN>
dd_url: '<MICROSATELLITE_ENDPOINT>'
apm_config:
apm_dd_url: '<MICROSATELLITE_ENDPOINT>'
process_config:
enabled: disabled
End code tabs
- Redeploy the configuration file and restart the Datadog Agent.