A notification destination defines the means by which an alert is sent and who it is sent to. Lightstep has built-in capability to set PagerDuty and Slack channels as destinations. You can also use webhooks to send alerts to other third-party solutions.

Once you create a notification destination, you can test it by sending a test alert. You can also edit webhook destinations and delete any destinations.

You can also create and manage notification destinations using our Public API.

Create a PagerDuty Notification Destination

PagerDuty alerts create a new incident in PagerDuty, and the resolution of the alert in Lightstep resolves the PagerDuty incident.

To create a PagerDuty destination:

  1. From the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

    Alerts tab

  2. Click the Notification Destinations tab, click Create a notification destination and choose PagerDuty.

    You’re redirected to PagerDuty. PagerDuty asks you to authorize Lightstep to trigger, acknowledge, and resolve incidents.

  3. Enter your credentials and click Authorized Integration. Select an existing service with a Lightstep integration or create a new PagerDuty service with an integration for Lightstep. Follow the prompts to configure PagerDuty.

Once configured in PagerDuty, the PagerDuty service is listed as a destination in Lightstep.

You can test that destination is correctly configured by sending a test alert.

Create a Slack Notification Destination

When you define Slack as a destination, the alert is sent to a specific channel or person in a Slack workspace. If you want all your workspace’s Slack channels to be enabled for destinations, then you can integrate Slack to provide that authorization.

The Slack message shows the project and stream the alert is for (with links to both) and the alert name and description, which are links to the Stream. There is also a link to the Explorer view, which will run the Stream’s query as the page loads. Both alerts of violated thresholds and notifications of resolutions are sent to the Slack channel.

You must have Admin privileges for your Slack workspace to create a Slack destination.

To create a Slack notification destination:

  1. From the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

    Destination tab

  2. Click the Notification Destinations tab, click Create a notification destination and choose Slack.

    You’re redirected to Slack. Slack asks you to sign into the workspace you want to post the alert to.

  3. Enter your workspace name and click Continue.

    An authorization page displays asking to authorize Lightstep to post to your Slack Workspace.

  4. Select the channel that should receive the alert and click Install.

The channel is added as a notification destination. You can test that your destination is correctly configured by sending a test alert.

Add a Slack workspace integration

If you are using the Public API to create a notification destination, or you want to send Lightstep alerts to multiple Slack channels in your workspace without creating a destination per channel, you must first enable the Slack workspace integration.

To integrate your Slack workspace:

  1. In Lightstep, in the left-hand navigation bar, click Account and choose Account Settings.
  2. Scroll to the Slack integration table. In the Send alerts and warnings to Slack, click the Enable Slack button.

Be sure to select the first integration in the table. The second integration allows you to display previews when sharing Lightstep links in Slack.

To remove the integration, click the Revoke button.

Create a Webhook Notification Destination

You can integrate with other alerting services using webhooks. Webhooks POST requests to the given URL with the alert payload.

To create a webhook for a notification destination:

  1. From the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

    Destination tab

  2. Click the Notification Destinations tab, click Create a notification destination and choose Webhook.

  3. Enter a name for the webhook and the HTTP or HTTPS URL where Lightstep can send POST requests with the alert payload. You can create any number of header key/value pairs to be included in all POST requests (often needed to handle authentication or custom messaging).

    You can add custom data to your alerts. This additional data will be included along with the base alert payload.

Example JSON Payload for Webhook ALERT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
{
  "grouping-id": "21a2_1030_2020-11-06T17:53:01Z",
  "timestamp": "2020-11-06T17:53:01.434454188Z",
  "title": "Alert Triggered",
  "project": {
    "text": "demo",
    "href": "https://app.lightstep.com/demo"
    },
  "saved-search": {
    "text": "component:\"android\" operation:\"/api/make-payment\"",
    "href": "https://app.lightstep.com/demo/stream/VvF37s1J"
    },
  "live-view": {
    "text": "Go to Explorer",
    "href": "https://app.lightstep.com/demo/explorer?q=component%3A%22android%22+operation%3A%22%2Fapi%2Fmake-payment%22"
    },
  "payloads": [
    {
      "type": "condition-violated",
      "subtitle": {
        "text": "lat(95) > 4s",
        "href": "https://app.lightstep.com/demo/stream/VvF37s1J?selected_condition_id=cBTPjq85"
        },
      "exemplars": [
        {
          "start": "2020-11-06T17:51:16.650381Z",
          "end": "2020-11-06T17:51:21.440972Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1604685081440972&selected_span_guid=6c9f6f0da1070105&trace_handle=CIWCnIja4dvPbBIGNDY4NzUxKAIyDAiUmpb9BRDIjZC2AjoOc3RyZWFtc190cmFjZXM%3D&utm_source=webhook"
        },
        {
          "start": "2020-11-06T17:51:14.338421Z",
          "end": "2020-11-06T17:51:19.803192Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1604685079803192&selected_span_guid=5eb482849f71b789&trace_handle=CInvxvvJ0KDaXhIGNDY4NzUxKAIyDAiSmpb9BRCIyq-hAToOc3RyZWFtc190cmFjZXM%3D&utm_source=webhook"
        }
      ],
    "ls-info": {
      "Actual Value": {
        "ActualValue": "4.218855s"
      },
      "Expression": "p95 is above 4.00s over the last 2.0m"
    },
    "custom-data": {}
    }
  ]
}

Edit a Webhook Notification Destination

You edit webhook notification destinations from the Notification Destinations tab.

You can’t edit Slack or PagerDuty destinations.
You need to delete, then recreate them if you want to change them.

To edit a webhook notification destination:

  1. From the navigation bar, click Alerts and click the Notification Destinations tab.

  2. Click Edit for the Destination.

  3. You can change the name, URL, and add or delete headers for the webhook.

Test a Notification Destination

Once you create a notification destination, you can test that it’s configured properly. Lightstep sends a test message without any specific alert information.

To test a notification destination:

  1. From the navigation bar, click Alerts and click the Notification Destinations tab.
  2. In the list of destinations, click Test for the appropriate destination.

    Lightstep sends a test message.

Delete a Notification Destination

You can delete notification destination from the Notification Destinations view.

To delete a notification destination:

  1. From the navigation bar, click Monitoring and click the Destinations tab.
  2. Click Delete for the appropriate destination.