To bring metric data into Lightstep Observability, you configure a Datadog agent to “tee” traffic to both Lightstep and Datadog. The traffic is duplicated and sent to both destinations. If you have existing dashboards that you want to import into Lightstep Observability, you can use a script to export them and then send them to Lightstep to be imported.

Importing dashboards to Lightstep Observability will not affect your existing dashboards in Datadog.

If you’re just starting with Datadog (and don’t need to import dashboards or send data to Lightstep Observability), you can send metrics directly to Lightstep.

Requirements

Lightstep Observability supports both Datadog Agent major versions 6 and 7 tracks.
The following minimum versions are required:

  • Major version 6: Minimum version of 6.18.0 (released March 16, 2020)
  • Major version 7: Minimum version of 7.18.0 (released March 16, 2020)

Major version 5 of the Datadog agent is not supported. Version 2 of the Datadog API is also not supported.

Prerequisites

You’ll need the following values to complete the steps:

Send metric data from Datadog to Lightstep Observability

To send metrics to Lightstep Observability you need to add the Lightstep endpoint to your Datadog agent configuration. It takes a key for the Lightstep metrics ingest domain (https://metricingest.lightstep.com) and a value of a Lightstep access token. This allows the data to be accepted by Lightstep. You can do this using the Datadog YAML configuration file.

Use YAML to configure the Agent

  1. In your datadog.yaml file, add the additional_endpoints property in the Basic Configuration section, and configure it with the URL for Lightstep metrics ingest and your Lightstep access token. If you are currently using v2 of the Datadog API, then you must also set that to false.
1
2
3
4
5
6
7
8
9
10
11

#########################
## Basic Configuration ##
#########################

api_key: <EXISTING_DATADOG_API_KEY>

additional_endpoints:
  'https://metricingest.lightstep.com':
    - <LIGHTSTEP_ACCESS_TOKEN>

use_v2_api.series: false
  1. Redeploy the configuration file and restart the Datadog Agent.
    At startup, look for a log message containing Forwarder started;. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep Observability).

Use environment variables to configure the Agent

The Datadog Docker Agent, the containerized version of the Datadog Agent, supports configuration via environment variables. If you are currently using v2 of the Datadog API, then you must also set that to false.

If you are currently using v2 of the Datadog API, then you must also set the DD_USE_V2_API_SERIES environment variable to false.

  1. Set the DD_ADDITIONAL_ENDPOINTS environment variable with values for the Lightstep metrics ingest domain (https://metricingest.lightstep.com) and Lightstep access token as serialized JSON.

Start tabs

Docker Run

1

-e DD_ADDITIONAL_ENDPOINTS='{"https://metricingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'

Kubernetes

1
2
3

env:
  - name: DD_ADDITIONAL_ENDPOINTS
    value: '{"https://metricingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'

Docker Compose

1
2
3
4

services:
  datadog:
    environment:
      - DD_ADDITIONAL_ENDPOINTS='{"https://metricingest.lightstep.com": ["LIGHTSTEP_ACCESS_TOKEN"]}'

End code tabs

  1. Restart the Datadog Agent.
    At startup, look for a log message containing Forwarder started;. This lists the destinations the agent is sending to (which should include both Datadog and Lightstep Observability).

You can validate that metrics are reporting to Lightstep on the Metrics details page in Project settings. Expand icon

Import Datadog metric dashboards and data into Lightstep Observability

To import Datadog dashboards into Lightstep Observability, you run a script to fetch the dashboards, or you can grab the JSON from an individual dashboard in the Datadog GUI. When you run the script, you can export all or a subset. When you send them to Lightstep, they can transform them into Lightstep Observability Metrics Dashboards.

Export Datadog dashboard JSON from GUI

You can follow these instructions to export an individual dashboard’s JSON.

Download and test the script

  1. Set your Datadog APP and API keys as environment variables.
    Replace <YOUR-APPLICATION-KEY> and <YOUR-API-KEY> with your values.

    1
2

     export DD_CLIENT_APP_KEY=<YOUR-APPLICATION-KEY>
 export DD_CLIENT_API_KEY=<YOUR-API-KEY>

You will need organization-level app and API keys from Datadog.

  1. Navigate to the directory where you downloaded the import script, then make the script executable.

    1
2
3

     cd ~/Downloads/

 chmod +x ./datadog-fetch.sh

  2. To confirm the script is working, run:
    ./datadog-fetch.sh help

    The script is successful if you see this message:

    1

     preflight checks passed

Run the script

You can export all your dashboards and metrics or pick a subset. Both dashboards and metrics are saved to their own file named after the ID (for example, datadog/dashboards/$DASHBOARD_ID.json or datadog/metrics/$METRIC_NAME.json) in the directory where you ran the script.

Fetch all dashboards and metric data

Run this in your terminal to fetch all dashboards:
./datadog-fetch.sh -all dashboards

Run the following in your terminal to fetch all metric metadata. There is no bulk API endpoint so this will make an API call for every active metric that reported during the last 2 weeks. Expect that this will take awhile and grab a cup of tea!

./datadog-fetch.sh -all metrics

Fetch a selection of dashboards and metrics by ID

To export dashboards correctly Lightstep Observability needs to fetch not only the dashboards, but also the metric kinds associated with the queries.

Identify the the IDs of dashboards you are interested in exporting. You can find these in the URL when viewing the dashboard in the DataDog UI, for example app.datadog.com/dashboard/DASHBOARD_ID/DASHBOARD_TITLE.

Run the script to export by ID in your terminal:
./datadog-fetch.sh -dashboard-ids DASHBOARD_ID, DASHBOARD_ID...

The ID of a metric is the metric name. For each metric that is charted on your dashboards find the metric name which you can find by looking at Metric Explorer, Metric Summary, or reviewing the query fields for your dashboards. For example, for avg:kubernetes.memory.limits{$kube_environment,kube_app:api-gateway} the metric ID is kubernetes_memory_limits.

Run the script to export by ID in your terminal:
./datadog-fetch.sh -metric-ids METRIC_NAME, METRIC_NAME...

Fetch a selection based on other criteria

If you don’t want to click through to find all the IDs of the dashboards, you can use other filter criteria.
To do that, run the script and export all dashboards and then use a JSON parsing tool to filter what you want to keep. You can filter by the presence of a word in a title, a tag, or any of the fields present on a dashboard.

Dashboards by title:

1
2
3
4
5
6
7
8

#!/bin/bash
for file in ./datadog/dashboards/*
do
 if [[ $(cat $file | jq '. | select(.title | contains("WORD-TO-MATCH")) | .id') ]]
   then echo "keeping ${file}"
   else echo rm $file && rm $file
fi
done

Confirm the export was successful

You can spot check the individual dashboard and metric files to ensure the export was successful or run these commands.

Confirm the dashboard export was successful

Confirm the export worked by running this command:
ls ./datadog/dashboards

This command outputs the files that were exported, for example, $DASHBOARD_ID.json.

Confirm the metric metadata export was successful

Confirm the export was successful by running this command. It should output several files named like $METRIC_NAME.json.
ls ./datadog/metrics

Send Lightstep Observability exported dashboards

Compress and save the exported directory by running this command:

tar -czvf CUSTOMER_NAME_metadata.tar.gz ./datadog/

Email the file to your technical account manager or contact Customer Success.

Lightstep will turn eligible dashboards into Lightstep Observability metrics dashboards. You’ll be notified when the migration is complete and you can start experimenting with them.

Send only Datadog metrics just to Lightstep Observability

If you just want to send metrics to Lightstep Observability, and not import any dashboards or also send metrics to Datadog, you need to replace the default Datadog configuration.

In your Datadog configuration file, replace api_key and dd_url with the following in the Basic Configuration section of the datadog.yaml file:

1
2
3
4
5
6
7

#########################
## Basic Configuration ##
#########################

api_key: <LIGHTSTEP_ACCESS_TOKEN>
dd_url: "https://metricingest.lightstep.com"
use_v2_api.series: false

Redeploy the configuration file and restart the Datadog agent.

You can validate that metrics are reporting to Lightstep on the Metrics details page in Project settings. Expand icon