A notification destination defines the means by which an alert is sent and who it is sent to. Lightstep has built-in capability to set the following integrations destinations:

Once you create a notification destination, you can test it by sending a test alert.

You can also create and manage notification destinations using our Public API.

Create a PagerDuty Notification Destination

PagerDuty alerts create a new incident in PagerDuty, and the resolution of the alert in Lightstep resolves the PagerDuty incident.

To create a PagerDuty destination:

  1. From the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

  2. Click the Notification destinations tab, click Create a destination and choose PagerDuty.

    You’re redirected to PagerDuty. PagerDuty asks you to authorize Lightstep to trigger, acknowledge, and resolve incidents.

  3. Enter your credentials and click Authorize Integration. Select an existing service with a Lightstep integration or create a new PagerDuty service with an integration for Lightstep. Follow the prompts to configure PagerDuty.

Once configured in PagerDuty, the PagerDuty service is listed as a destination in Lightstep.

You can test that destination is correctly configured by sending a test alert.

Create a Slack Notification Destination

When you define Slack as a destination, the alert is sent to a specific channel or person in a Slack workspace. If you want all your workspace’s Slack channels to be enabled for destinations, then you can integrate Slack to provide that authorization.

The Slack message shows the project and stream the alert is for (with links to both) and the alert name and description, which are links to the Stream. There is also a link to the Explorer view, which will run the Stream’s query as the page loads. Both alerts of violated thresholds and notifications of resolutions are sent to the Slack channel.

You must have Admin privileges for your Slack workspace to create a Slack destination.

To create a Slack notification destination:

  1. From the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

  2. Click the Notification destinations tab, click Create a destination and choose Slack.

    You’re redirected to Slack. Slack asks you to sign into the workspace you want to post the alert to.

  3. Enter your workspace name and click Continue.

    An authorization page displays asking to authorize Lightstep to post to your Slack Workspace.

  4. Select the channel that should receive the alert and click Install.

The channel is added as a notification destination. You can test that your destination is correctly configured by sending a test alert.

Add a Slack workspace integration

If you are using the Public API to create a notification destination, or you want to send Lightstep alerts to multiple Slack channels in your workspace without creating a destination per channel, you must first enable the Slack workspace integration.

To integrate your Slack workspace:

  1. In Lightstep, in the left-hand navigation bar, click Account and choose Account Settings.
  2. Scroll to the Slack integration table. In the Send alerts and warnings to Slack, click the Enable Slack button.

Be sure to select the first integration in the table. The second integration allows you to display previews when sharing Lightstep links in Slack.

To remove the integration, click the Revoke button.

Create a BigPanda Destination

Alerts with a BigPanda destination result in a new incident in BigPanda. When the alert is resolved, the incident is moved to the Resolved folder in BigPanda.

To create a BigPanda destination, you’ll need a BigPanda Bearer token and App key, generated from the Integrations tab in BigPanda.

To generate the BigPanda App key and Bearer token:

  1. In BigPanda, click the Integrations tab and click New Integration. Add integration in BigPanda

  2. On the Create a New Integration page, choose REST API. REST API integration

  3. For Step 1, enter a name for the integration and click Generate App Key. Generate App key

Your App key and Bearer token are generated. You’ll need these to configure the destination in Lightstep. App ken and Bearer token

To create a BigPanda destination:

  1. In Lightstep, from the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

  2. Click the Notification destinations tab, click Create a destination and choose BigPanda.

  3. Enter a name for the destination and paste in the Bearer token and App key. You can add optional header content if needed. Click Save. Add BigPanda

BigPanda is listed as a destination in Lightstep.

You can test that destination is correctly configured by sending a test alert.

Create a Lightstep Incident Response Destination

Alerts with a Lightstep Incident Response destination send the alert to Incident Response, where it acts as a handler, routing the alert based on escalation policies and on-call schedules. When the alert is resolved in Lightstep, it’s also resolved in Incident Response. The integration is created using a webhook.

You need to first generate the webhook in Incident Response and then in Lightstep, configure Incident Response as the destination.

To generate an Incident Response webhook:

  1. In Incident Response, click Integrations and choose Lightstep Observability. Incident Response integrations

  2. Enter the following on the Integrations page:
    • Name: Enter a name for the integration
    • Status: You don’t need to configure this - it will be sent to Enabled once the webhook is created.
    • Description: Enter a description
    • Integration URL: Enter the URL to the Lightstep project for this integration.
      For example https://app.lightstep.com/my-project
    • Service: Enter the Incident Response service that this alert is associated with.
  3. Click Generate Webhook.
    The webhook displays in the Webhook field. Copy and paste this webhook somewhere safe, as you’ll need it to finish the configuration in Lightstep.Webhook generated

To configure the destination:

  1. In Lightstep, from the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

  2. Click the Notification destinations tab, click Create a destination and choose Webhook. Webhook destination

  3. Enter a name for the destination and in the URL field, paste the webhook URL. Enter webhook in Lightstep

  4. Click Create.

The Incident Response destination is now available for alerts. Incident Response destination

You can test that destination is correctly configured by sending a test alert.

Create a Webhook Notification Destination

You can integrate with other alerting services using webhooks. Webhooks POST requests to the given URL with the alert payload.

To create a webhook for a notification destination:

  1. From the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

  2. Click the Notification destinations tab, click Create a destination and choose Webhook.

  3. Enter a name for the webhook and the HTTP or HTTPS URL where Lightstep can send POST requests with the alert payload. You can create any number of header key/value pairs to be included in all POST requests (often needed to handle authentication or custom messaging).

    You can add custom data to your alerts. This additional data will be included along with the base alert payload.

Example JSON Payload for Webhook ALERT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
{
  "grouping-id": "21a2_1030_2020-11-06T17:53:01Z",
  "timestamp": "2020-11-06T17:53:01.434454188Z",
  "title": "Alert Triggered",
  "project": {
    "text": "demo",
    "href": "https://app.lightstep.com/demo"
    },
  "saved-search": {
    "text": "component:\"android\" operation:\"/api/make-payment\"",
    "href": "https://app.lightstep.com/demo/stream/VvF37s1J"
    },
  "live-view": {
    "text": "Go to Explorer",
    "href": "https://app.lightstep.com/demo/explorer?q=component%3A%22android%22+operation%3A%22%2Fapi%2Fmake-payment%22"
    },
  "payloads": [
    {
      "type": "condition-violated",
      "subtitle": {
        "text": "lat(95) > 4s",
        "href": "https://app.lightstep.com/demo/stream/VvF37s1J?selected_condition_id=cBTPjq85"
        },
      "exemplars": [
        {
          "start": "2020-11-06T17:51:16.650381Z",
          "end": "2020-11-06T17:51:21.440972Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1604685081440972&selected_span_guid=6c9f6f0da1070105&trace_handle=CIWCnIja4dvPbBIGNDY4NzUxKAIyDAiUmpb9BRDIjZC2AjoOc3RyZWFtc190cmFjZXM%3D&utm_source=webhook"
        },
        {
          "start": "2020-11-06T17:51:14.338421Z",
          "end": "2020-11-06T17:51:19.803192Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1604685079803192&selected_span_guid=5eb482849f71b789&trace_handle=CInvxvvJ0KDaXhIGNDY4NzUxKAIyDAiSmpb9BRCIyq-hAToOc3RyZWFtc190cmFjZXM%3D&utm_source=webhook"
        }
      ],
    "ls-info": {
      "Actual Value": {
        "ActualValue": "4.218855s"
      },
      "Expression": "p95 is above 4.00s over the last 2.0m",
      "Query": "service IN (\"android\") AND operation IN (\"/api/make-payment/\")"
    },
    "custom-data": {}
    }
  ]
}

Edit a Webhook Notification Destination

You edit webhook notification destinations from the Notification destinations tab.

You can’t edit Slack or PagerDuty destinations.
You need to delete, then recreate them if you want to change them.

To edit a webhook notification destination:

  1. From the navigation bar, click Alerts and click the Notification destinations tab.

  2. Click Edit for the destination.

  3. You can change the name, URL, and add or delete headers for the webhook.

Test a Notification Destination

Once you create a notification destination, you can test that it’s configured properly. Lightstep sends a test message without any specific alert information.

To test a notification destination:

  1. From the navigation bar, click Alerts and click the Notification destinations tab.
  2. In the list of destinations, click Test for the appropriate destination.

    Lightstep sends a test message.

Delete a Notification Destination

You can delete notification destination from the Notification destinations tab.

To delete a notification destination:

  1. From the navigation bar, click Monitoring and click the Notification destinations tab.
  2. Click Delete for the appropriate destination.