View, manage, and investigate Cloud Observability alerts.
You can get a quick overview of alerts, and if they are in compliance or not, from the Alerts page. Open the Alerts page by clicking Alerts from the navigation bar.
By default, the Alerts view lists all alerts by their status, with those in violation at the top. Alerts can have the following statuses:
Red/Critical: The alert is in violation of the Critical threshold.
You can filter alerts by name, status, date, or alert type.
Click on an alert to open it, view the associated data, edit it, and snooze it.
You can add to a notebook for when, during an investigation, you want to be able to run ad hoc queries, take notes, and save your analysis for use in postmortems or runbooks. Notebooks allow you to view metric and trace data from different places in Cloud Observability together, in one place.
To add to a notebook, click Add to notebook and search to choose an existing notebook or create a new notebook.
When you add to a notebook, a chart is created using the same query. The annotation is a link back to the original, so you can quickly return to the origin of your investigation.
Learn more about notebooks.
To delete an alert, from the Alerts view use the gear icon to choose Delete.
Once you have the alert open in the editor, use Cloud Observability’s correlation feature to investigate the deviation and find possible causes.
You can’t use the correlation feature on big number charts.
If you’ve made any edits to the alert, save those changes before using the correlation feature.
To run the correlation feature, click View correlations or click directly in the chart and select View correlations. Cloud Observability opens a side panel where you can begin your investigation.
Visit Investigate deviations for more information.
You can snooze an alert when needed, for example if you know a team is working on a fix and don’t need to be further notified.
To snooze an alert:
From the Alerts view, click the alert to open it in the editor.
Click Snooze, choose the amount of time to snooze the alert for, and click Save. The alert now displays in the Alert view as snoozed. When you hover over the snooze icon, a tooltip displays the time when the alert will reactivate.
To un-snooze an alert:
You remove a snooze by returning to the editor using the Snooze button to choose Off.
If you use Terraform to manage your alerts, a common use case is to create a “baseline” alert in the Lightstep UI and then export it to Terraform to create and manage new alerts. Creating an alert in the UI is easier and more intuitive than working in a Terraform configuration file. When you export the alert to a Terraform file, you can then clone it and edit the file to create new alerts, for example by changing the individual queries behind the charts.
To export an alert to Terraform, from the alert’s page, click the More ( ⋮ ) icon and
choose Export to Terraform. The
tf config file can then be downloaded to
any local directory.
Updated Jan 17, 2023