To bring metric data into Lightstep, you configure a Datadog agent to “tee” traffic to both Lightstep and Datadog. The traffic is duplicated and sent to both destinations. If you have existing dashboards that you want to import into Lightstep, you can use a script to export them and then send them to Lightstep to be imported.
Importing dashboards to Lightstep will not affect your existing dashboards in Datadog.
If you’re just starting with Datadog (and don’t need to import dashboards or send data to Lightstep), you can send metrics directly to Lightstep.
Requirements
Lightstep supports both Datadog Agent major versions 6 and 7 tracks.
The following minimum versions are required:
- Major version 6: Minimum version of 6.18.0 (released March 16, 2020)
- Major version 7: Minimum version of 7.18.0 (released March 16, 2020)
Major version 5 of the Datadog agent is not supported.
Prerequisites
- You’ll need the following values to complete the steps:
- Your Datadog API key
- A Datadog App key
- A Lightstep access token for the project to report metrics to.
- Download the datadog-fetch.sh script to a local directory.
Send Metric Data and Import Dashboards from Datadog to Lightstep
To send metrics to Lightstep you need to add the additional_endpoints
configuration field to your Datadog agent configuration. You can do this using either the YAML configuration file or by setting environment variables.
Use YAML to Configure Datadog
To tee traffic to Lightstep, you need to add the additional_endpoints
configuration field to your Datadog agent configuration. It takes a key for the Lightstep metrics ingest domain (https://metricingest.lightstep.com
) and a value of a Lightstep access token. This allows the data to be accepted by Lightstep.
Replace <EXISTING_DATADOG_API_KEY>
and <LIGHTSTEP_ACCESS_TOKEN>
with your values.
1
2
3
4
api_key: <EXISTING_DATADOG_API_KEY>
additional_endpoints:
'https://metricingest.lightstep.com':
- <LIGHTSTEP_ACCESS_TOKEN>
Redeploy the configuration file and restart the Datadog agent.
At startup, look for a log message containing Forwarder started;
. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep).
Use Environment Variables to Configure Datadog
To tee traffic to Lightstep from Datadog using environment variables, you need to set the DD_ADDITIONAL_ENDPOINTS
environment variable with values for the Lightstep metrics ingest domain (https://metricingest.lightstep.com
) and Lightstep access token as serialized JSON.
Replace <LIGHTSTEP_ACCESS_TOKEN>
with your value.
1
export DD_ADDITIONAL_ENDPOINTS='{"https://metricingest.lightstep.com": ["<LIGHTSTEP_ACCESS_TOKEN>"]}'
Restart the Datadog agent.
At startup, look for a log message containing Forwarder started;
. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep).
Import Datadog Metric Dashboards and Data into Lightstep
To import Datadog dashboards into Lightstep, you run a script to fetch the dashboards. When you run it, you can export all or a subset. When you send them to Lightstep, they can transform them into Lightstep Metrics Dashboards.
Download and Test the Script
-
Set your Datadog APP and API keys as environment variables.
Replace<YOUR-APPLICATION-KEY>
and<YOUR-API-KEY>
with your values.1 2
export DD_CLIENT_APP_KEY=<YOUR-APPLICATION-KEY> export DD_CLIENT_API_KEY=<YOUR-API-KEY>
-
Navigate to the directory where you downloaded the import script, then make the script executable.
1 2 3
cd ~/Downloads/ chmod +x ./datadog-fetch.sh
-
To confirm the script is working, run:
./datadog-fetch.sh help
The script is successful if you see this message:
1
preflight checks passed
Run the Script
You can export all your dashboards and metrics or pick a subset. Both dashboards and metrics are saved to their own file named after the ID (for example, datadog/dashboards/$DASHBOARD_ID.json
or datadog/metrics/$METRIC_NAME.json
) in the directory where you ran the script.
Fetch All Dashboards and Metric Data
Run this in your terminal to fetch all dashboards:
./datadog-fetch.sh -all dashboards
Run the following in your terminal to fetch all metric metadata. There is no bulk API endpoint so this will make an API call for every active metric that reported during the last 2 weeks. Expect that this will take awhile and grab a cup of tea!
./datadog-fetch.sh -all metrics
Fetch a Selection of Dashboards and Metrics by ID
To export dashboards correctly Lightstep needs to fetch not only the dashboards, but also the metric types associated with the queries.
Identify the the IDs of dashboards you are interested in exporting. You can find these in the URL when viewing the dashboard in the DataDog UI, for example app.datadog.com/dashboard/DASHBOARD_ID/DASHBOARD_TITLE
.
Run the script to export by ID in your terminal:
./datadog-fetch.sh -dashboard-ids DASHBOARD_ID, DASHBOARD_ID...
The ID of a metric is the metric name. For each metric that is charted on your dashboards find the metric name which you can find by looking at Metric Explorer, Metric Summary, or reviewing the query fields for your dashboards. For example, for avg:kubernetes.memory.limits{$kube_environment,kube_app:api-gateway}
the metric ID is kubernetes_memory_limits
.
Run the script to export by ID in your terminal:
./datadog-fetch.sh -metric-ids METRIC_NAME, METRIC_NAME...
Fetch a Selection Based on Other Criteria
If you don’t want to click through to find all the IDs of the dashboards, you can use other filter criteria.
To do that, run the script and export all dashboards and then use a JSON parsing tool to filter what you want to keep. You can filter by the presence of a word in a title, a tag, or any of the fields present on a dashboard.
Dashboards by title:
1
2
3
4
5
6
7
8
#!/bin/bash
for file in ./datadog/dashboards/*
do
if [[ $(cat $file | jq '. | select(.title | contains("WORD-TO-MATCH")) | .id') ]]
then echo "keeping ${file}"
else echo rm $file && rm $file
fi
done
Confirm the Export was Successful
You can spot check the individual dashboard and metric files to ensure the export was successful or run these commands.
Confirm the Dashboard Export was Successful
Confirm the export worked by running this command:
ls ./datadog/dashboards
This command outputs the files that were exported, for example, $DASHBOARD_ID.json
.
Confirm the Metric Metadata Export was Successful
Confirm the export was successful by running this command. It should output several files named like $METRIC_NAME.json
.
ls ./datadog/metrics
Send Lightstep Exported Dashboards
Compress and save the exported directory by running this command:
tar -czvf CUSTOMER_NAME_metadata.tar.gz ./datadog/
Email the file to your technical account manager or contact Customer Success. Lightstep will turn eligible dashboards into Lightstep metrics dashboards. You’ll be notified when the migration is complete and you can start experimenting with them.
Send Only Datadog Metrics Just to Lightstep
If you just want to send metrics to Lightstep, and not import any dashboards or also send metrics to Datadog, you need to replace the default Datadog configuration.
In your Datadog configuration file, replace apiKey
and dd_url
with the following:
1
2
api_key: <YOUR LIGHTSTEP_ACCESS_TOKEN>
dd_url: "https://metricingest.lightstep.com"
Redeploy the configuration file and restart the Datadog agent.