To bring metric data into Lightstep Observability, you configure a Datadog agent to “tee” traffic to both Lightstep and Datadog. The traffic is duplicated and sent to both destinations. If you have existing dashboards that you want to import into Lightstep Observability, you can use a script to export them and then send them to Lightstep to be imported.
Importing dashboards to Lightstep Observability will not affect your existing dashboards in Datadog.
If you’re just starting with Datadog (and don’t need to import dashboards or send data to Lightstep Observability), you can send metrics directly to Lightstep.
Requirements
Lightstep Observability supports both Datadog Agent major versions 6 and 7 tracks.
The following minimum versions are required:
- Major version 6: Minimum version of 6.18.0 (released March 16, 2020)
- Major version 7: Minimum version of 7.18.0 (released March 16, 2020)
Major version 5 of the Datadog agent is not supported. Version 2 of the Datadog API is also not supported.
Prerequisites
You’ll need the following values to complete the steps:
- An organization-level Datadog API key
- An organization-level Datadog App key
- A Lightstep access token for the project to report metrics to.
- Download the datadog-fetch.sh script to a local directory.
Send metric data from Datadog to Lightstep Observability
To send metrics to Lightstep Observability you need to add the Lightstep endpoint to your Datadog agent configuration. It takes a key for the Lightstep metrics ingest domain (https://metricingest.lightstep.com
) and a value of a Lightstep access token. This allows the data to be accepted by Lightstep. You can do this using the Datadog YAML configuration file.
Use YAML to configure the Agent
- In your
datadog.yaml
file, add theadditional_endpoints
property in the Basic Configuration section, and configure it with the URL for Lightstep metrics ingest and your Lightstep access token. If you are currently using v2 of the Datadog API, then you must also set that to false.
1
2
3
4
5
6
7
8
9
10
11
#########################
## Basic Configuration ##
#########################
api_key: <EXISTING_DATADOG_API_KEY>
additional_endpoints:
'https://metricingest.lightstep.com':
- <LIGHTSTEP_ACCESS_TOKEN>
use_v2_api.series: false
- Redeploy the configuration file and restart the Datadog Agent.
At startup, look for a log message containingForwarder started;
. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep Observability).
Use environment variables to configure the Agent
The Datadog Docker Agent, the containerized version of the Datadog Agent, supports configuration via environment variables. If you are currently using v2 of the Datadog API, then you must also set that to false.
If you are currently using v2 of the Datadog API, then you must also set the DD_USE_V2_API_SERIES
environment variable to false.
- Set the
DD_ADDITIONAL_ENDPOINTS
environment variable with values for the Lightstep metrics ingest domain (https://metricingest.lightstep.com
) and Lightstep access token as serialized JSON.
Start tabs
Docker Run
1
-e DD_ADDITIONAL_ENDPOINTS='{"https://metricingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'
Kubernetes
1
2
3
env:
- name: DD_ADDITIONAL_ENDPOINTS
value: '{"https://metricingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'
Docker Compose
1
2
3
4
services:
datadog:
environment:
- DD_ADDITIONAL_ENDPOINTS='{"https://metricingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'
End code tabs
- Restart the Datadog Agent.
At startup, look for a log message containingForwarder started;
. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep Observability).
You can validate that metrics are reporting to Lightstep on the Metrics details page in Project settings.
Import Datadog metric dashboards and data into Lightstep Observability
To import Datadog dashboards into Lightstep Observability, you run a script to fetch the dashboards, or you can grab the JSON from an individual dashboard in the Datadog GUI. When you run the script, you can export all or a subset. When you send them to Lightstep, they can transform them into Lightstep Observability Metrics Dashboards.
Export Datadog dashboard JSON from GUI
You can follow these instructions to export an individual dashboard’s JSON.
Download and test the script
-
Set your Datadog APP and API keys as environment variables.
Replace<YOUR-APPLICATION-KEY>
and<YOUR-API-KEY>
with your values.1 2
export DD_CLIENT_APP_KEY=<YOUR-APPLICATION-KEY> export DD_CLIENT_API_KEY=<YOUR-API-KEY>
You will need organization-level app and API keys from Datadog.
-
Navigate to the directory where you downloaded the import script, then make the script executable.
1 2 3
cd ~/Downloads/ chmod +x ./datadog-fetch.sh
-
To confirm the script is working, run:
./datadog-fetch.sh help
The script is successful if you see this message:
1
preflight checks passed
Run the script
You can export all your dashboards and metrics or pick a subset. Both dashboards and metrics are saved to their own file named after the ID (for example, datadog/dashboards/$DASHBOARD_ID.json
or datadog/metrics/$METRIC_NAME.json
) in the directory where you ran the script.
Fetch all dashboards and metric data
Run this in your terminal to fetch all dashboards:
./datadog-fetch.sh -all dashboards
Run the following in your terminal to fetch all metric metadata. There is no bulk API endpoint so this will make an API call for every active metric that reported during the last 2 weeks. Expect that this will take awhile and grab a cup of tea!
./datadog-fetch.sh -all metrics
Fetch a selection of dashboards and metrics by ID
To export dashboards correctly Lightstep Observability needs to fetch not only the dashboards, but also the metric kinds associated with the queries.
Identify the the IDs of dashboards you are interested in exporting. You can find these in the URL when viewing the dashboard in the DataDog UI, for example app.datadog.com/dashboard/DASHBOARD_ID/DASHBOARD_TITLE
.
Run the script to export by ID in your terminal:
./datadog-fetch.sh -dashboard-ids DASHBOARD_ID, DASHBOARD_ID...
The ID of a metric is the metric name. For each metric that is charted on your dashboards find the metric name which you can find by looking at Metric Explorer, Metric Summary, or reviewing the query fields for your dashboards. For example, for avg:kubernetes.memory.limits{$kube_environment,kube_app:api-gateway}
the metric ID is kubernetes_memory_limits
.
Run the script to export by ID in your terminal:
./datadog-fetch.sh -metric-ids METRIC_NAME, METRIC_NAME...
Fetch a selection based on other criteria
If you don’t want to click through to find all the IDs of the dashboards, you can use other filter criteria.
To do that, run the script and export all dashboards and then use a JSON parsing tool to filter what you want to keep. You can filter by the presence of a word in a title, a tag, or any of the fields present on a dashboard.
Dashboards by title:
1
2
3
4
5
6
7
8
#!/bin/bash
for file in ./datadog/dashboards/*
do
if [[ $(cat $file | jq '. | select(.title | contains("WORD-TO-MATCH")) | .id') ]]
then echo "keeping ${file}"
else echo rm $file && rm $file
fi
done
Confirm the export was successful
You can spot check the individual dashboard and metric files to ensure the export was successful or run these commands.
Confirm the dashboard export was successful
Confirm the export worked by running this command:
ls ./datadog/dashboards
This command outputs the files that were exported, for example, $DASHBOARD_ID.json
.
Confirm the metric metadata export was successful
Confirm the export was successful by running this command. It should output several files named like $METRIC_NAME.json
.
ls ./datadog/metrics
Send Lightstep Observability exported dashboards
Compress and save the exported directory by running this command:
tar -czvf CUSTOMER_NAME_metadata.tar.gz ./datadog/
Email the file to your technical account manager or contact Customer Success.
Lightstep will turn eligible dashboards into Lightstep Observability metrics dashboards. You’ll be notified when the migration is complete and you can start experimenting with them.
Send only Datadog metrics just to Lightstep Observability
If you just want to send metrics to Lightstep Observability, and not import any dashboards or also send metrics to Datadog, you need to replace the default Datadog configuration.
In your Datadog configuration file, replace api_key
and dd_url
with the following in the Basic Configuration section of the datadog.yaml
file:
1
2
3
4
5
6
7
#########################
## Basic Configuration ##
#########################
api_key: <LIGHTSTEP_ACCESS_TOKEN>
dd_url: "https://metricingest.lightstep.com"
use_v2_api.series: false
Redeploy the configuration file and restart the Datadog agent.
You can validate that metrics are reporting to Lightstep on the Metrics details page in Project settings.