A destination defines the means by which an alert is sent and who it is sent to. Lightstep has built-in capability to set PagerDuty and Slack channels as destinations. You can also use webhooks to send alerts to other third-party solutions.

Once you create a destination, you can test it by sending a test alert. You can also edit webhook destinations and delete any destinations.

You can also create and manage destinations using our Public API.

Create a PagerDuty Destination

PagerDuty alerts create a new incident in PagerDuty, and the resolution of the alert in Lightstep resolves the PagerDuty incident.

To create a PagerDuty destination:

  1. From the navigation bar, click Monitoring.

    The Monitoring view shows a list of all current conditions.

  2. Click the Destination tab, click New Message Destination and choose PagerDuty.

    You’re redirected to PagerDuty. PagerDuty asks you to authorize Lightstep to trigger, acknowledge, and resolve incidents.

  3. Enter your credentials and click Authorized Integration. Select an existing service with a Lightstep integration or create a new PagerDuty service with an integration for Lightstep. Follow the prompts to configure PagerDuty.

Once configured in PagerDuty, the PagerDuty service is listed as a destination in Lightstep.

You can test that destination is correctly configured by sending a test alert.

You can now create conditions and rules.

Create a Slack Destination

When you define Slack as a destination, the alert is sent to a specific channel or person in a Slack workspace. If you want all your workspace’s Slack channels to be enabled for destinations, then you can integrate Slack to provide that authorization.

The Slack message shows the project and stream the alert is for (with links to both) and the condition name and description, which are links to the Stream. There is also a link to the Explorer view, which will run the Stream’s query as the page loads. Both alerts of violated conditions and notifications of resolutions are sent to the Slack channel.

You must have Admin privileges for your Slack workspace to create a Slack destination.

To create a Slack destination:

  1. From the navigation bar, click Monitoring.

    The Monitoring view shows a list of all current conditions.

  2. Click the Destination tab, click New Message Destination and choose Slack.

    You’re redirected to Slack. Slack asks you to sign into the workspace you want to post the alert to.

  3. Enter your workspace name and click Continue.

    An authorization page displays asking to authorize Lightstep to post to your Slack Workspace.

  4. Select the channel that should receive the alert and click Install.

The channel is added as a destination. You can test that your destination is correctly configured by sending a test alert.

You can now create conditions and rules.

Add a Slack workspace integration

If you are using the Public API to create a destination, or you want to send Lightstep alerts to multiple Slack channels in your workspace without creating a destination per channel, you must first enable the Slack workspace integration.

To integrate your Slack workspace:

  1. In Lightstep, in the left-hand navigation bar, click Account and choose Account Settings.
  2. Scroll to the Slack integration table. In the Send alerts and warnings to Slack, click the Enable Slack button.

Be sure to select the first integration in the table. The second integration allows you to display previews when sharing Lightstep links in Slack.

To remove the integration, click the Revoke button.

Create a Webhook Destination

You can integrate with other alerting services using webhooks. Webhooks POST requests to the given URL with the alert payload.

To create a webhook for a destination:

  1. From the navigation bar, click Monitoring.

    The Monitoring view shows a list of all current conditions.

  2. Click the Destination tab, click New Message Destination and choose Webhook.

  3. Enter a name for the webhook and the HTTP or HTTPS URL where Lightstep can send POST requests with the alert payload. You can create any number of header key/value pairs to be included in all POST requests (often needed to handle authentication or custom messaging).

    You can add custom data to your alerts. This additional data will be included along with the base alert payload.

Example JSON Payload for Webhook ALERT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
{
  "grouping-id": "ls-2017010304045",
  "timestamp": "2017-07-01T16:14:12Z00:00",
  "title": "New Latency Alert",
  "project": {
    "text": "ls-project",
    "href": "https://app.lightstep.com/ls-project/"
  },
  "saved-search": {
    "text": "customer latency",
    "href": "https://app.lightstep.com/ls-project/operations/3cver6A"
  },
  "payloads": [
    {
      "type": "condition-violated",
      "subtitle": {
        "text": "Condition Name",
        "href": "https://app.lightstep.com/ls-project/operations/3cver6A"
      },
      "exemplars": [
        {
          "start": "2017-07-01T16:14:12.00003030Z00",
          "end": "2017-07-01T16:14:12.00003050Z00",
          "is-error": false,
          "href": "https://app.lightstep.com/ls-project/trace?span-guid=abcde"
        },
        {
          "start": "2017-07-01T16:14:12.00003030Z00",
          "end": "2017-07-01T16:14:12.00003050Z00",
          "is-error": false,
          "href": "https://app.lightstep.com/ls-project/trace?span-guid=abcde"
        }
      ],
      "expression": "p99.9 > 500ms",
      "actual-value": {
        "latency-ms": 521
      },
      "custom-data": {
        "custom-key-1": "custom-value-1",
        "custom-key-2": ["list", "of", "values"],
        "custom-key-5": { "nested": "custom values" }
      }
    }
  ]
}

Edit a Webhook Destination

You edit webhook destinations from the Destinations view.

You can’t edit Slack or PagerDuty destinations.
You need to delete, then recreate them if you want to change them.

To edit a webhook destination:

  1. From the navigation bar, click Monitoring and click the Destinations tab.

  2. Click Edit for the Destination.

  3. You can change the name, URL, and add or delete headers for the webhook.

Test a Destination

Once you create a destination, you can test that it’s configured properly. Lightstep sends a test message without any specific alert information.

To test a destination:

  1. From the navigation bar, click Monitoring and click the Destinations tab.
  2. In the list of destinations, click Test for the appropriate destination.

    Lightstep sends a test message.

Delete a Destintation

You can delete destination from the Destinations view.

To delete a destination:

  1. From the navigation bar, click Monitoring and click the Destinations tab.
  2. Click Delete for the appropriate destination.