Create webhook notification destinations

You can use webhooks to send alert notifications to third-party apps. Cloud Observability offers a number of prebuilt payload templates for popular apps that you can use out of the box or edit to fit your workflow. Or you can build your own from scratch. The templates are JSON-based Go templates and you add variables to pass information in the payload from Cloud Observability to the third-party app.

If you don’t choose a template, Cloud Observability uses a default payload. These are not editable.

Create a Webhook notification destination

To create a webhook destination, you’ll need to know the URL where Cloud Observability can send POST requests with the alert payload. You can optionally add headers to be included in all POST requests (often needed to handle authentication or custom messaging).

To create a webhook:

1. From the navigation bar, click Alerts.
   The Alerts view shows a list of all current alerts.

2. Click the Notification destinations tab, click Create a destination and choose Webhook.

3. Enter a name for the webhook and the HTTP or HTTPS URL where Cloud Observability can send POST requests with the alert payload. You can create any number of header key/value pairs to be included in all POST requests (often needed to handle authentication or custom messaging).

4. Select a template for the payload, or choose Custom to build your own. Refer to the variables table for variable definitions.

   If you don’t select a template, or build your own, the webhook will use the unified default payload.

5. Click Create.

   The webhook destination is now ready to be selected for any alert.

Edit a Webhook notification destination

You edit webhook notification destinations from the Notification destinations tab.

To edit a webhook notification destination:

1. From the navigation bar, click Alerts and click the Notification destinations tab.

2. Click Edit for the destination.

3. You can change the name, URL, add or delete headers, and edit the template.

Variables

Variables in the templates pull values from Cloud Observability and add them to the payload. Use the links in the dialog to the various vendors’ API docs to understand where and when to use these variables in a template.

Enclose variables in double braces and precede the variable name with a period.

For example: {{ .Status }}

Unified alert default payload example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
  "timestamp": "2021-06-03T17:25:28.724661485Z",
  "title": "Critical Threshold Violated: Test Alert",
  "status": "Critical Threshold Violated",
  "name": "Test Alert",
  "incident_id": "409f7e6f-645a-4f6d-93e0-fc5a5d967753",
  "project": {
    "text": "demo",
    "href": "https://app.lightstep.com/demo?utm_source=webhook"
  },
  "metric-condition": {
    "text": "Test Alert",
    "href": "https://app.lightstep.com/demo/monitoring/condition/Tr0FhDmV?end_micros=1622741128724661&start_micros=1622740228724661&utm_source=webhook"
  },
  "details": {
    "Actual Value": "17.90",
    "Configuration": "Alert when any group value is above the following thresholds (Critical: 5)",
    "Description": "Test description",
    "Labels": "team:my-team",
    "Exemplars": [
      {
        "start": "2021-06-03T17:25:28.724661485Z",
        "end": "2021-06-03T17:25:28.724661485Z",
        "is-error": false,
        "href": "https://app.lightstep.com/demo/trace?at_micros=110&selected_span_guid=1423&trace_handle=woof&utm_source=webhook"
      }
    ],
    "Expression": "Critical Threshold Violated: alert 'Test Alert' is above 5 (value is 17.90)",
    "Attributes": "region:east, host:13",
    "Started At": "Thu, 03 Jun 2021 17:25:28 UTC"
  }
}

Inbound IP addresses

Because the webhooks send POST requests with JSON payloads over HTTP/HTTPS, you need to ensure your inbound allow list includes these IP addresses.

See also

Create alerts

Manage notification destinations

View and manage alerts

Updated Jul 5, 2023