You can use webhooks to send alert notifications to third-party apps. Lightstep offers a number of prebuilt payload templates for popular apps that you can use out of the box or edit to fit your workflow. Or you can build your own from scratch. The templates are JSON-based Go templates and you add variables to pass information in the payload from Lightstep to the third-party app.

If you don’t choose a template, Lightstep uses a default payload. These are not editable. For Stream-based alerts, you can add custom data to the default payload.

Create a Webhook notification destination

To create a webhook destination, you’ll need to know the URL where Lightstep can send POST requests with the alert payload. You can optionally add headers to be included in all POST requests (often needed to handle authentication or custom messaging).

To create a webhook:

  1. From the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

  2. Click the Notification destinations tab, click Create a destination and choose Webhook.

  3. Enter a name for the webhook and the HTTP or HTTPS URL where Lightstep can send POST requests with the alert payload. You can create any number of header key/value pairs to be included in all POST requests (often needed to handle authentication or custom messaging).

  4. Select a template for the payload, or choose Custom to build your own. Refer to the variables table for variable definitions. Template variables

    If you don’t select a template, or build your own, the webhook will use either the metric or span default payload.

  5. Click Create.

    The webhook destination is now ready to be selected for any alert.

    Edit a Webhook notification destination

You edit webhook notification destinations from the Notification destinations tab.

To edit a webhook notification destination:

  1. From the navigation bar, click Alerts and click the Notification destinations tab.

  2. Click Edit for the destination.

  3. You can change the name, URL, add or delete headers, and edit the template.

Variables

Variables in the templates pull values from Lightstep and add them to the payload. Use the links in the dialog to the various vendors’ API docs to understand where and when to use these variables in a template.

Enclose variables in double braces and precede the variable name with a period.

For example: {{ .Source }}

Variable Type Definition Example value
Source string Describes the type of alert "metric", "stream"
IncidentID string Grouping key for the alert "409f7e6f-645a-4f6d-93e0-fc5a5d967753"
ConditionID string Lightstep identifier of the alert object "Tr0FhDmV"
ConditionHref string Link to the alert "https://app.lightstep.com/demo/monitoring/condition/Tr0FhDmV"
Expression string String representation of the alert condition "p99 is above 100ms over the last 2.0m"
Title string Brief description of the alert "Alert Triggered"
Description string Longer description of the alert user-specified for metric alerts, empty for stream alerts
Project string Lightstep project name "demo"
Status string State of the alert (varies by Source "condition-violated"
ActualValue string String representation of numeric value that violated the alert condition "3.981549s"
StartedAt time When the alert started "2021-07-08T17:29:32.921207101Z"
LastEvaluated time When the alert was last evaluated "2021-07-08T17:29:32.921207101Z"
MetadataJSON string Object of string values that are attached to the alert (varies by Source) (stream alert): { "Query": "service IN (\"android\", \"apache\", \"iOS\")" }
MetadataJSONEscaped                                                                            string JSON-escaped string of MetadataJSON  

Metric alert default payload example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
{
  "timestamp": "2021-06-03T17:25:28.724661485Z",
  "status": "Critical Threshold Violated",
  "name": "*Critical Threshold Violated: Test Alert*",
  "incident_id": "409f7e6f-645a-4f6d-93e0-fc5a5d967753",
  "project": {
    "text": "demo",
    "href": "https://app.lightstep.com/demo?utm_source=webhook"
  },
  "metric-condition": {
    "text": "Test Alert",
    "href": "https://app.lightstep.com/demo/monitoring/condition/Tr0FhDmV?end_micros=1622741128724661&start_micros=1622740228724661&utm_source=webhook"
  },
  "details": {
    "Description": "test test",
    "Expression": "Critical Threshold Violated: alert 'Test Alert' is above 1000 over the last 2 minutes, computed on average.",
    "Started At": "Thu, 03 Jun 2021 17:25:28 UTC",
    "Queries":[
      {
        "query":{
          "metric":"test_metric.success",
          "filters":[
            {
              "label_key":"service",
              "label_value":"test_service",
              "operand":"eq"
            },
            {
              "label_key":"host",
              "label_value":"bad_host",
              "operand":"neq"
            }
          ],
          "timeseries_operator":"delta",
          "group_by":{
            "label_keys":[
              "host"
            ],
            "aggregation_method":"sum"
          },
          "distribution_operators":{
            "percentiles":[

            ]
          },
          "version":0,
          "query_aggregation_options":{
            "time_window_aggregation_method":"",
            "max_values":0,
            "rank_order":""
          }
        },
        "uql_query":"",
        "query_name":"a",
        "query_type":"single",
        "hidden":false,
        "display_type":"line",
        "style":{
          "line_type":"",
          "line_width":"",
          "palette":""
        },
        "version":0
      },
      {
        "query":{
          "metric":"test_metric.error",
          "filters":[
            {
              "label_key":"service",
              "label_value":"test_service",
              "operand":"eq"
            },
            {
              "label_key":"host",
              "label_value":"bad_host",
              "operand":"neq"
            }
          ],
          "timeseries_operator":"delta",
          "group_by":{
            "label_keys":[
              "host"
            ],
            "aggregation_method":"sum"
          },
          "distribution_operators":{
            "percentiles":[

            ]
          },
          "version":0,
          "query_aggregation_options":{
            "time_window_aggregation_method":"",
            "max_values":0,
            "rank_order":""
          }
        },
        "uql_query":"",
        "query_name":"b",
        "query_type":"single",
        "hidden":false,
        "display_type":"line",
        "style":{
          "line_type":"",
          "line_width":"",
          "palette":""
        },
        "version":0
      },
      {
        "query":{
          "metric":"",
          "filters":[

          ],
          "timeseries_operator":"",
          "group_by":{
            "label_keys":null,
            "aggregation_method":""
          },
          "distribution_operators":null,
          "version":0,
          "query_aggregation_options":{
            "time_window_aggregation_method":"",
            "max_values":0,
            "rank_order":""
          }
        },
        "uql_query":"",
        "query_name":"b/a",
        "query_type":"composite",
        "hidden":false,
        "display_type":"line",
        "style":{
          "line_type":"",
          "line_width":"",
          "palette":""
        },
        "version":0
      }
    ]
  }
}

Span alert default payload example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
{
  "grouping-id": "336f_54fd_2021-07-08T17:29:32Z",
  "timestamp": "2021-07-08T17:29:32.921207101Z",
  "title": "Alert Triggered",
  "project": {
    "text": "demo",
    "href": "https://app.lightstep.com/demo"
  },
  "saved-search": {
    "text": "service IN (\"android\", \"apache\", \"iOS\")",
    "href": "https://app.lightstep.com/demo/stream/ctfztmht?anchor=1625765373\u0026range=900"
  },
  "live-view": {
    "text": "Go to Explorer",
    "href": "https://app.lightstep.com/demo/explorer?query=service+IN+%28%22android%22%2C+%22apache%22%2C+%22iOS%22%29"
  },
  "payloads": [
    {
      "type": "condition-violated",
      "subtitle": {
        "text": "latency2",
        "href": "https://app.lightstep.com/demo/stream/ctfztmht?anchor=1625765373\u0026range=900\u0026selected_condition_id=VXmXbJsB"
      },
      "exemplars": [
        {
          "start": "2021-07-08T17:28:08.770045Z",
          "end": "2021-07-08T17:28:08.88467Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765288884670\u0026selected_span_guid=938797040ea8e787\u0026trace_handle=CIfPo_XA4OXDkwESBzExMTkxNDIoAjIMCKjrnIcGEMjol-8COg5zdHJlYW1zX3RyYWNlcw%3D%3D\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:26:50.694247Z",
          "end": "2021-07-08T17:26:50.83419Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765210834190\u0026selected_span_guid=03e3a83249ca873f\u0026trace_handle=CL-Oqs6khurxAxIHMTExOTE0MigCMgwI2uqchwYQ2LyFywI6DnN0cmVhbXNfdHJhY2Vz\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:28:10.547794Z",
          "end": "2021-07-08T17:28:10.75597Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765290755970\u0026selected_span_guid=a230186c3ec5a73b\u0026trace_handle=CLvOlvbDjYaYogESBzExMTkxNDIoAjIMCKrrnIcGENDYmoUCOg5zdHJlYW1zX3RyYWNlcw%3D%3D\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:27:06.62625Z",
          "end": "2021-07-08T17:27:06.859943Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765226859943\u0026selected_span_guid=03bb918b119c8ced\u0026trace_handle=CO2Z8oyxseTdAxIHMTExOTE0MigCMgwI6uqchwYQkKLPqgI6DnN0cmVhbXNfdHJhY2Vz\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:27:07.688467Z",
          "end": "2021-07-08T17:27:08.035603Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765228035603\u0026selected_span_guid=4b637ae9b66c3fb4\u0026trace_handle=CLT_sLOb3d6xSxIHMTExOTE0MigCMgwI6-qchwYQuNikyAI6DnN0cmVhbXNfdHJhY2Vz\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:27:05.576175Z",
          "end": "2021-07-08T17:27:06.718736Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765226718736\u0026selected_span_guid=9c78f1bc014b141b\u0026trace_handle=CJuorIrAt7y8nAESBzExMTkxNDIoAjIMCOnqnIcGEJj33pICOg5zdHJlYW1zX3RyYWNlcw%3D%3D\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:26:48.972024Z",
          "end": "2021-07-08T17:26:50.576486Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765210576486\u0026selected_span_guid=88b90093f1c2bf5a\u0026trace_handle=CNr-io6_ksDciAESBzExMTkxNDIoAjIMCNjqnIcGEMDRv88DOg5zdHJlYW1zX3RyYWNlcw%3D%3D\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:28:06.649611Z",
          "end": "2021-07-08T17:28:08.324684Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765288324684\u0026selected_span_guid=0cfbcaf67b02727c\u0026trace_handle=CPzkidjn3vL9DBIHMTExOTE0MigCMgwIpuuchwYQ-I3htQI6DnN0cmVhbXNfdHJhY2Vz\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:28:06.218973Z",
          "end": "2021-07-08T17:28:08.350201Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765288350201\u0026selected_span_guid=f88adc4e001f4034\u0026trace_handle=CLSA_YDgibfF-AESBzExMTkxNDIoAjILCKbrnIcGEMiGtWg6DnN0cmVhbXNfdHJhY2Vz\u0026utm_source=webhook"
        },
        {
          "start": "2021-07-08T17:28:06.223964Z",
          "end": "2021-07-08T17:28:08.977992Z",
          "is-error": false,
          "href": "https://app.lightstep.com/demo/trace?at_micros=1625765288977992\u0026selected_span_guid=a55d982ace8bd91e\u0026trace_handle=CJ6yr_SsheaupQESBzExMTkxNDIoAjILCKbrnIcGEODW5Wo6DnN0cmVhbXNfdHJhY2Vz\u0026utm_source=webhook"
        }
      ],
      "ls-info": {
        "Actual Value": {
          "ActualValue": "3.981549s"
        },
        "Expression": "p99 is above 100ms over the last 2.0m",
        "Query": "service IN (\"android\", \"apache\", \"iOS\")"
      },
      "custom-data": {}
    }
  ]
}