Create webhook notification destinations

You can use webhooks to send alert notifications to third-party apps. Cloud Observability offers a number of prebuilt payload templates for popular apps that you can use out of the box or edit to fit your workflow. Or you can build your own from scratch. The templates are JSON-based Go templates and you add variables to pass information in the payload from Cloud Observability to the third-party app.

If you don’t choose a template, Cloud Observability uses a default payload. These are not editable.

Create a Webhook notification destination

To create a webhook destination, you’ll need to know the URL where Cloud Observability can send POST requests with the alert payload. You can optionally add headers to be included in all POST requests (often needed to handle authentication or custom messaging).

To create a webhook:

  1. From the navigation bar, click Alerts.
    The Alerts view shows a list of all current alerts.

  2. Click the Notification destinations tab, click Create a destination and choose Webhook.

  3. Enter a name for the webhook and the HTTP or HTTPS URL where Cloud Observability can send POST requests with the alert payload. You can create any number of header key/value pairs to be included in all POST requests (often needed to handle authentication or custom messaging).

  4. Select a template for the payload, or choose Custom to build your own. Refer to the variables table for variable definitions. Some available templates include ServiceNow Event Management, Microsoft Teams Webhook, and Slack Webhook.

    If you don’t select a template, or build your own, the webhook will use the unified default payload.

  5. Click Create.

    The webhook destination is now ready to be selected for any alert.

Edit a Webhook notification destination

You edit webhook notification destinations from the Notification destinations tab.

To edit a webhook notification destination:

  1. From the navigation bar, click Alerts and click the Notification destinations tab.

  2. Click Edit for the destination.

  3. You can change the name, URL, add or delete headers, and edit the template.

Variables

Variables in the templates pull values from Cloud Observability and add them to the payload. Use the links in the dialog to the various vendors’ API docs to understand where and when to use these variables in a template.

Enclose variables in double braces and precede the variable name with a period.

For example: {{ .Status }}

Variable Type Definition Example value
ConditionType string Describes the type of alert "metric"
IncidentID string Grouping key for the alert "409f7e6f-645a-4f6d-93e0-fc5a5d967753"
ConditionID string Cloud Observability identifier of the alert object "Tr0FhDmV"
ConditionHref string Link to the alert "https://app.lightstep.com/demo/monitoring/condition/Tr0FhDmV"
Title string Name of the alert, user-specified "Test Alert"
Description string Longer description of the alert, user-specified "Here is a description"
LabelsJSON string Labels assigned to the alert, user-specified "team:my-team"
Project string Cloud Observability project name "demo"
ProjectJSON string Object of string values with project name and URL { "text": "demo", "href": "https://app.lightstep.com/demo?utm_source=webhook" }
Status string State of the alert, varies by condition type unified alert: "Critical Threshold Violated"
Expression string String representation of the alert condition "Critical Threshold Violated: alert 'Test Alert' is above 5 (value is 17.90)"
Configuration string Configuration text for the alert "Alert when any value is above the following thresholds (Critical: 2.5, Warning: 1.5) or when no data is reporting"
Query string String representation of the alert condition "service IN (\"android\", \"apache\", \"iOS\")"
PrimaryService string Cloud Observability service associated with the alert, if one exists or can be determined "apache"
ViolatedAttributes string List of the attribute(s) that caused the alert to trigger, for multi-alerts only "region:east, host:13"
ActualValue string String representation of numeric value that violated the alert condition "3.98s"
StartedAt time When the alert started "2021-07-08T17:29:32.921207101Z"
LastEvaluated time When the alert was last evaluated "2021-07-08T17:29:32.921207101Z"
CustomData string Free-form string of additional data to send in the payload (if using JSON, be sure to validate it, as this is treated specifically as a string). This variable is only available using the APi.
MetadataJSON string Object of string values that are attached to the alert, varies by condition type
MetadataJSONEscaped string JSON-escaped string of MetadataJSON
ExemplarsJSON string Array of exemplar objects for when an alert fires [ { "start": "2021-06-03T17:25:28.724661485Z", "end": "2021-06-03T17:25:28.724661485Z", "is-error": false, "href": "https://app.lightstep.com/demo/trace?at_micros=110&selected_span_guid=1423&trace_handle=woof&utm_source=webhook" } ]

Unified alert default payload example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
  "timestamp": "2021-06-03T17:25:28.724661485Z",
  "title": "Critical Threshold Violated: Test Alert",
  "status": "Critical Threshold Violated",
  "name": "Test Alert",
  "incident_id": "409f7e6f-645a-4f6d-93e0-fc5a5d967753",
  "project": {
    "text": "demo",
    "href": "https://app.lightstep.com/demo?utm_source=webhook"
  },
  "metric-condition": {
    "text": "Test Alert",
    "href": "https://app.lightstep.com/demo/monitoring/condition/Tr0FhDmV?end_micros=1622741128724661&start_micros=1622740228724661&utm_source=webhook"
  },
  "details": {
    "Actual Value": "17.90",
    "Configuration": "Alert when any group value is above the following thresholds (Critical: 5)",
    "Description": "Test description",
    "Labels": "team:my-team",
    "Exemplars": [
      {
        "start": "2021-06-03T17:25:28.724661485Z",
        "end": "2021-06-03T17:25:28.724661485Z",
        "is-error": false,
        "href": "https://app.lightstep.com/demo/trace?at_micros=110&selected_span_guid=1423&trace_handle=woof&utm_source=webhook"
      }
    ],
    "Expression": "Critical Threshold Violated: alert 'Test Alert' is above 5 (value is 17.90)",
    "Attributes": "region:east, host:13",
    "Started At": "Thu, 03 Jun 2021 17:25:28 UTC"
  }
}

Inbound IP addresses

Because the webhooks send POST requests with JSON payloads over HTTP/HTTPS, you need to ensure your inbound allow list includes these IP addresses.

See also

Create alerts

Manage notification destinations

View and manage alerts

Updated Jul 5, 2023