View and manage alerts

View, manage, and investigate Cloud Observability alerts.

View alerts and status

You can get a quick overview of alerts, and if they are in compliance or not, from the Alerts page. Open the Alerts page by clicking Alerts from the navigation bar.

By default, the Alerts view lists all alerts by their status, with those in violation at the top. Alerts can have the following statuses:

  • Red/Critical: The alert is in violation of the Critical threshold.

  • Green/OK: The alert is active and is currently in a non-triggered state. All inbound traffic is being analyzed against the criteria you have set over the defined evaluation window.
  • Light gray/Unknown: The alert has been created but it’s not yet in an active monitoring state. If an alert remains light gray, either:
    • Refresh the page to get the alert’s latest status.
    • Click the alert and click Alert configuration. If Cloud Observability is waiting to activate the alert, the page shows this message: Your alert will activate in X days when Cloud Observability has enough data to run the alert query.
  • Dark gray/No data: There is no data reporting for this query.
  • Grayed out: The alert is active, but is currently in a snoozed state. This means that ingested traffic is being analyzed against the criteria defined in the alert, but alerts will not be generated if a threshold is breached.

You can filter alerts by name, status, date, or alert type.

Click on an alert to open it, view the associated data, edit it, and snooze it.

Add alert charts to notebooks

You can add to a notebook for when, during an investigation, you want to be able to run ad hoc queries, take notes, and save your analysis for use in postmortems or runbooks. Notebooks allow you to view metric and trace data from different places in Cloud Observability together, in one place. Notebook

To add to a notebook, click Add to notebook and search to choose an existing notebook or create a new notebook.

When you add to a notebook, a chart is created using the same query. The annotation is a link back to the original, so you can quickly return to the origin of your investigation.

Learn more about notebooks.

Delete alerts

To delete an alert, from the Alerts view use the gear icon to choose Delete.

Investigate correlations

Once you have the alert open in the editor, use Cloud Observability’s correlation feature to investigate the deviation and find possible causes.

You can’t use the correlation feature on big number charts.

If you’ve made any edits to the alert, save those changes before using the correlation feature.

To run the correlation feature, click View correlations or click directly in the chart and select View correlations. Cloud Observability opens a side panel where you can begin your investigation.

Visit Investigate deviations for more information.

Snooze alerts

You can snooze an alert when needed, for example if you know a team is working on a fix and don’t need to be further notified.

To snooze an alert:

  1. From the Alerts view, click the alert to open it in the editor.

  2. Click Snooze, choose the amount of time to snooze the alert for, and click Save. Snooze an alert The alert now displays in the Alert view as snoozed. When you hover over the snooze icon, a tooltip displays the time when the alert will reactivate. Snoozed alert

To un-snooze an alert:

You remove a snooze by returning to the editor using the Snooze button to choose Off.

Export an alert to Terraform

If you use Terraform to manage your alerts, a common use case is to create a “baseline” alert in the Lightstep UI and then export it to Terraform to create and manage new alerts. Creating an alert in the UI is easier and more intuitive than working in a Terraform configuration file. When you export the alert to a Terraform file, you can then clone it and edit the file to create new alerts, for example by changing the individual queries behind the charts.

To export an alert to Terraform, from the alert’s page, click the More ( ⋮ ) icon and choose Export to Terraform. The tf config file can then be downloaded to any local directory. Export to terraform

See also

Create alerts

Manage notification destinations

Sample alerts

Updated Jan 17, 2023