To bring metric data into Lightstep, you configure a Datadog agent to “tee” traffic to both Lightstep and Datadog. The traffic is duplicated and sent to both destinations. If you have existing dashboards that you want to import into Lightstep, you can use a script to export them and then send them to Lightstep to be imported.

Importing dashboards to Lightstep will not affect your existing dashboards in Datadog.

If you’re just starting with Datadog (and don’t need to import dashboards or send data to Lightstep), you can send metrics directly to Lightstep.

Requirements

Lightstep supports both Datadog Agent major versions 6 and 7 tracks.
The following minimum versions are required:

  • Major version 6: Minimum version of 6.18.0 (released March 16, 2020)
  • Major version 7: Minimum version of 7.18.0 (released March 16, 2020)

Major version 5 of the Datadog agent is not supported.

Prerequisites

Send Metric Data and Import Dashboards from Datadog to Lightstep

To send metrics to Lightstep you need to add the additional_endpoints configuration field to your Datadog agent configuration. You can do this using either the YAML configuration file or by setting environment variables.

Use YAML to Configure Datadog

To tee traffic to Lightstep, you need to add the additional_endpoints configuration field to your Datadog agent configuration. It takes a key for the Lightstep metrics ingest domain (https://metricingest.lightstep.com) and a value of a Lightstep access token. This allows the data to be accepted by Lightstep.

Replace <EXISTING_DATADOG_API_KEY> and <LIGHTSTEP_ACCESS_TOKEN> with your values.

1
2
3
4

api_key: <EXISTING_DATADOG_API_KEY>
additional_endpoints:
  'https://metricingest.lightstep.com':
    - <LIGHTSTEP_ACCESS_TOKEN>

Redeploy the configuration file and restart the Datadog agent.
At startup, look for a log message containing Forwarder started;. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep).

Use Environment Variables to Configure Datadog

To tee traffic to Lightstep from Datadog using environment variables, you need to set the DD_ADDITIONAL_ENDPOINTS environment variable with values for the Lightstep metrics ingest domain (https://metricingest.lightstep.com) and Lightstep access token as serialized JSON.

Replace <LIGHTSTEP_ACCESS_TOKEN> with your value.

1

export DD_ADDITIONAL_ENDPOINTS='{"https://metricingest.lightstep.com": ["<LIGHTSTEP_ACCESS_TOKEN>"]}'

Restart the Datadog agent.
At startup, look for a log message containing Forwarder started;. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep).

Import Datadog Metric Dashboards and Data into Lightstep

To import Datadog dashboards into Lightstep, you run a script to fetch the dashboards. When you run it, you can export all or a subset. When you send them to Lightstep, they can transform them into Lightstep Metrics Dashboards.

Download and Test the Script

  1. Set your Datadog APP and API keys as environment variables.
    Replace <YOUR-APPLICATION-KEY> and <YOUR-API-KEY> with your values.

    1
2

     export DD_CLIENT_APP_KEY=<YOUR-APPLICATION-KEY>
 export DD_CLIENT_API_KEY=<YOUR-API-KEY>

  2. Navigate to the directory where you downloaded the import script, then make the script executable.

    1
2
3

     cd ~/Downloads/

 chmod +x ./datadog-fetch.sh

  3. To confirm the script is working, run:
    ./datadog-fetch.sh help

    The script is successful if you see this message:

    1

     preflight checks passed

Run the Script

You can export all your dashboards and metrics or pick a subset. Both dashboards and metrics are saved to their own file named after the ID (for example, datadog/dashboards/$DASHBOARD_ID.json or datadog/metrics/$METRIC_NAME.json) in the directory where you ran the script.

Fetch All Dashboards and Metric Data

Run this in your terminal to fetch all dashboards:
./datadog-fetch.sh -all dashboards

Run the following in your terminal to fetch all metric metadata. There is no bulk API endpoint so this will make an API call for every active metric that reported during the last 2 weeks. Expect that this will take awhile and grab a cup of tea!

./datadog-fetch.sh -all metrics

Fetch a Selection of Dashboards and Metrics by ID

To export dashboards correctly Lightstep needs to fetch not only the dashboards, but also the metric types associated with the queries.

Identify the the IDs of dashboards you are interested in exporting. You can find these in the URL when viewing the dashboard in the DataDog UI, for example app.datadog.com/dashboard/DASHBOARD_ID/DASHBOARD_TITLE.

Run the script to export by ID in your terminal:
./datadog-fetch.sh -dashboard-ids DASHBOARD_ID, DASHBOARD_ID...

The ID of a metric is the metric name. For each metric that is charted on your dashboards find the metric name which you can find by looking at Metric Explorer, Metric Summary, or reviewing the query fields for your dashboards. For example, for avg:kubernetes.memory.limits{$kube_environment,kube_app:api-gateway} the metric ID is kubernetes_memory_limits.

Run the script to export by ID in your terminal:
./datadog-fetch.sh -metric-ids METRIC_NAME, METRIC_NAME...

Fetch a Selection Based on Other Criteria

If you don’t want to click through to find all the IDs of the dashboards, you can use other filter criteria.
To do that, run the script and export all dashboards and then use a JSON parsing tool to filter what you want to keep. You can filter by the presence of a word in a title, a tag, or any of the fields present on a dashboard.

Dashboards by title:

1
2
3
4
5
6
7
8

#!/bin/bash
for file in ./datadog/dashboards/*
do
 if [[ $(cat $file | jq '. | select(.title | contains("WORD-TO-MATCH")) | .id') ]]
   then echo "keeping ${file}"
   else echo rm $file && rm $file
fi
done

Confirm the Export was Successful

You can spot check the individual dashboard and metric files to ensure the export was successful or run these commands.

Confirm the Dashboard Export was Successful

Confirm the export worked by running this command:
ls ./datadog/dashboards

This command outputs the files that were exported, for example, $DASHBOARD_ID.json.

Confirm the Metric Metadata Export was Successful

Confirm the export was successful by running this command. It should output several files named like $METRIC_NAME.json.
ls ./datadog/metrics

Send Lightstep Exported Dashboards

Compress and save the exported directory by running this command:

tar -czvf CUSTOMER_NAME_metadata.tar.gz ./datadog/

Email the file to your technical account manager or contact Customer Success. Lightstep will turn eligible dashboards into Lightstep metrics dashboards. You’ll be notified when the migration is complete and you can start experimenting with them.

Send Only Datadog Metrics Just to Lightstep

If you just want to send metrics to Lightstep, and not import any dashboards or also send metrics to Datadog, you need to replace the default Datadog configuration.

In your Datadog configuration file, replace apiKey and dd_url with the following:

1
2

api_key: <YOUR LIGHTSTEP_ACCESS_TOKEN>
dd_url: "https://metricingest.lightstep.com"

Redeploy the configuration file and restart the Datadog agent.