About alerts

You can create alerts for your metric or span data by configuring thresholds against queries that when crossed, trigger the alert. You can also create notification destinations that determine where an alert is sent when that alert is triggered.

This topic is about creating alerts for span and metric data. If you want to create an alert for an existing Stream, see this topic.

Using Terraform? You can use the Lightstep Terraform provider to create and manage your alerts and destinations. You can also use it to export existing alerts into the Terraform format.

For example, let’s say you know your customer Packing Kings is onboarding new clients and you want to be warned if the average request rate gets above 11 requests per second and a critical alert sent when the rate gets above 13 per second over a period of two minutes. You might configure the alert like this: Alert configuration

When an alert threshold is crossed, a notification is sent to the configured destination. Alert notification

An alert is also sent once the issue is resolved (or the metric drops below the threshold).Resolution alert

Alerts are listed on the Alerts tab of the Alerts view. You can see the status of every alert and you can also delete alerts from here.Alerts tab

Metric and span alerts are listed as Unified. Alerts created for Streams are listed as Stream alerts.

Once created, you can edit alerts and snooze them.

Build the query

  1. From the navigation bar, click Alerts and click Create an alert.Create an alert

  2. Enter a name for the alert. This should be descriptive enough that it will be identifiable in the list of alerts on the Alerts tab.Create alert

  3. Enter a description for the alert. The description is useful for adding more information about the alert, such as teams responsible or links to playbooks. Add alert description

    Descriptions can be written in basic markdown.

  4. Build your query. The results of the query are what will help define the threshold for the alert.

    Unless using latest for a metric query aggregation, you must define an input window for the alert.
    Span queries can’t include group bys or regular expressions.
    When creating a query based on a latency, you can choose only one percentile to query on.

The chart on the page reflects the results of your query. In this example, the query results show request rates for the customer PackingKings grouped by method name, aggregated by the mean of all values over a final two minute rolling input window. Alert query and results

Now that you can see the results of your query, you can set the threshold that when crossed, triggers an alert.

Configure the alert

  1. In the Alert configuration section, set the threshold.

    • Single or separate notifications: If you’ve grouped your results, you can choose to send a single alert when any one of the group crosses the threshold during the evaluation window, or you can choose to send alerts each time one of the group crosses the threshold in that window.

      For example in this query, if set to single, you will be alerted once when one of the methods crosses the threshold during the two minute window. If another method also crosses within that two minutes, you won’t get another alert. If instead you set it to separate, you will be alerted each time one of the methods crosses the threshold within the rolling input window set in your query.Single or separate notifications

    • Above or below the threshold: Choose whether the alert should be sent when the metric goes either above or below the given threshold.
    • Threshold: You can set either a Critical or Warning level threshold, or both. A warning is less severe than Critical. In this example, a warning is set if the metric crosses 9 ops/s and a critical alert is set at 11 ops/s. When set, the chart redraws to show the two levels so you can immediately see if any metric is crossing a threshold.Critical and warning thresholds
    • Notify if no data is reporting for this query: Select this option if you want to be notified if Lightstep Observability is not collecting any data for the query.

If you want a team or person to be notified of an alert outside of Lightstep Observability, you can add a notification destination.

Assign a notification destination

Lightstep Observability can send notifications of an alert to many third-party tools.

Destinations must already exist before you can assign them

  1. Expand the Notification Destination section.

  2. Choose a destination type, then begin typing to find the destination you want.

  3. Enter a time period that a renotification should be sent if the threshold is still crossed.

Alert destinations

You can add as many destinations as you want.

Be sure to click Save to save your configuration.

Snooze an alert

You can snooze an alert when needed, for example if you know a team is working on a fix and don’t need to be further notified.

To snooze an alert:

  1. From the Alerts view, click the alert to open it in the editor.

  2. Click Snooze, choose the amount of time to snooze the alert for, and click Save. Snooze an alert The alert now displays in the Alert view as snoozed. When you hover over the snooze icon, a tooltip displays the time when the alert will reactivate. Snoozed alert

To un-snooze an alert:

You remove a snooze by returning to the editor using the Snooze button to choose Off.

Delete an alert

To delete an alert, from the Alerts view use the gear icon to choose Delete.

Add an alert’s chart to a notebook

You can add an alert’s chart to a notebook for when, during an investigation, you want to be able to run ad hoc queries, take notes, and save your analysis for use in postmortems or runbooks. Notebooks allow you to view metric and trace data from different places in Lightstep Observability together, in one place. Notebook

To add to a notebook, click Add to notebook and search to choose an existing notebook or create a new notebook.

Add alert chart to notebook

When you add to a notebook, a chart is created using the same query. The annotation is a link back to the original, so you can quickly return to the origin of your investigation.

Add alert chart to notebook

Learn more about notebooks.

Run Change Intelligence on an alert

Once you have the alert open in the editor, you can use Change Intelligence to determine what cased the change in performance. Change Intelligence links metric data with trace data to find components in your system whose performance changed at the same time as the change, allowing you to find the root cause without leaving Lightstep Observability.

You can’t use Change Intelligence with big number charts.

Change Intelligence

If you’ve made any edits to the alert, you need to save those changes before using Change Intelligence!

To run Change Intelligence, click Analyze deviation or click directly in the deviation and select What caused this change?. Start Change Intelligence

Follow these instructions to start your investigation.