You can create alerts for your metric or span data by configuring thresholds against queries that when crossed, trigger the alert. You can also create notification destinations that determine where an alert is sent when that alert is triggered.
This topic is about creating alerts for span and metric data. If you want to create an alert for an existing Stream, see this topic.
Using Terraform? You can use the Lightstep Terraform provider to create and manage your alerts and destinations. You can also use it to export existing alerts into the Terraform format.
For example, let’s say you know your customer Packing Kings is onboarding new clients and you want to be warned if the average request rate gets above 11 requests per second and a critical alert sent when the rate gets above 13 per second over a period of two minutes. You might configure the alert like this:
When an alert threshold is crossed, a notification is sent to the configured destination.
An alert is also sent once the issue is resolved (or the metric drops below the threshold).
Alerts are listed on the Alerts tab of the Alerts view. You can see the status of every alert and you can also delete alerts from here.
Metric and span alerts are listed as Unified. Alerts created for Streams are listed as Stream alerts.
Once created, you can edit alerts and snooze them.
Build the query
From the navigation bar, click Alerts and click Create an alert.
Enter a name for the alert. This should be descriptive enough that it will be identifiable in the list of alerts on the Alerts tab.
Enter a description for the alert. The description is useful for adding more information about the alert, such as teams responsible or links to playbooks.
Descriptions can be written in basic markdown.
Build your query. The results of the query are what will help define the threshold for the alert.
latestfor a metric query aggregation, you must define an input window for the alert.
Span queries can’t include group bys or regular expressions.
When creating a query based on a latency, you can choose only one percentile to query on.
The chart on the page reflects the results of your query. In this example, the query results show request rates for the customer PackingKings grouped by method name, aggregated by the mean of all values over a final two minute rolling input window.
Now that you can see the results of your query, you can set the threshold that when crossed, triggers an alert.
Configure the alert
In the Alert configuration section, set the threshold.
Single or separate notifications: If you’ve grouped your results, you can choose to send a single alert when any one of the group crosses the threshold during the evaluation window, or you can choose to send alerts each time one of the group crosses the threshold in that window.
For example in this query, if set to single, you will be alerted once when one of the methods crosses the threshold during the two minute window. If another method also crosses within that two minutes, you won’t get another alert. If instead you set it to separate, you will be alerted each time one of the methods crosses the threshold within the rolling input window set in your query.
- Above or below the threshold: Choose whether the alert should be sent when the metric goes either above or below the given threshold.
- Threshold: You can set either a Critical or Warning level threshold, or both. A warning is less severe than Critical. In this example, a warning is set if the metric crosses 9 ops/s and a critical alert is set at 11 ops/s. When set, the chart redraws to show the two levels so you can immediately see if any metric is crossing a threshold.
- Notify if no data is reporting for this query: Select this option if you want to be notified if Lightstep Observability is not collecting any data for the query.
If you want a team or person to be notified of an alert outside of Lightstep Observability, you can add a notification destination.
Assign a notification destination
Lightstep Observability can send notifications of an alert to many third-party tools.
Destinations must already exist before you can assign them
Expand the Notification Destination section.
Choose a destination type, then begin typing to find the destination you want.
Enter a time period that a renotification should be sent if the threshold is still crossed.
You can add as many destinations as you want.
Be sure to click Save to save your configuration.
Snooze an alert
You can snooze an alert when needed, for example if you know a team is working on a fix and don’t need to be further notified.
To snooze an alert:
From the Alerts view, click the alert to open it in the editor.
Click Snooze, choose the amount of time to snooze the alert for, and click Save. The alert now displays in the Alert view as snoozed. When you hover over the snooze icon, a tooltip displays the time when the alert will reactivate.
To un-snooze an alert:
You remove a snooze by returning to the editor using the Snooze button to choose Off.
Delete an alert
To delete an alert, from the Alerts view use the gear icon to choose Delete.
Add an alert’s chart to a notebook
You can add an alert’s chart to a notebook for when, during an investigation, you want to be able to run ad hoc queries, take notes, and save your analysis for use in postmortems or runbooks. Notebooks allow you to view metric and trace data from different places in Lightstep Observability together, in one place.
To add to a notebook, click Add to notebook and search to choose an existing notebook or create a new notebook.
When you add to a notebook, a chart is created using the same query. The annotation is a link back to the original, so you can quickly return to the origin of your investigation.
Learn more about notebooks.
Run Change Intelligence on an alert
Once you have the alert open in the editor, you can use Change Intelligence to determine what cased the change in performance. Change Intelligence links metric data with trace data to find components in your system whose performance changed at the same time as the change, allowing you to find the root cause without leaving Lightstep Observability.
You can’t use Change Intelligence with big number charts.
If you’ve made any edits to the alert, you need to save those changes before using Change Intelligence!
To run Change Intelligence, click Analyze deviation or click directly in the deviation and select What caused this change?.
Follow these instructions to start your investigation.