Create alerts to monitor system performance, get notified when things change, and identify possible causes. With Lightstep, you can create alerts on metric or span data and send notifications to tools such as PagerDuty and Slack.

The content below shows how to create Lightstep alerts. By the end of this page, you’ll know how to create an alert from scratch and a percentage change alert.

This page uses Lightstep’s Unified Query Builder to create alerts. You can also create alerts in the Editor using Lightstep’s Unified Query Language (UQL). Visit UQL for alerts for more on configuring alerts with UQL.

Before you begin

The steps below are intended for Lightstep users with either the Admin or Member role. The steps also assume you’ve done the following:

Step 1: Name and describe your alert

Follow these steps to name and describe a new alert:

  1. In Lightstep, click Alerts > Create an alert.
  2. Click Enter an alert name to name your alert, for example, Supernova request rate alert.
  3. Add a description by clicking Describe important details about this alert. The description supports Markdown, and you can include links to teams and runbooks. Here’s an example:

    1
    2
    3
    4
    5
    6
    7
    
     Send alerts when request rates go above 13 per second over two minutes. Have questions?
     Ask in [#sre](https://lightstep.com/).
    
     Docs:
    
     - [Runbooks](https://lightstep.com/)
     - [Troubleshooting](https://lightstep.com/)
    

Step 2: Select an alert template

On the same page, select the template you want to use for your alert.

You can switch templates after configuring your query and alert. If your inputs aren’t compatible across templates, Lightstep will ask you to confirm the switch because you may lose information.

Start from scratch

Use this template to alert on one query or several queries combined with a formula. This option is the most flexible template. Choose this template if you’re starting out and exploring alerting options.

Here are some examples of Start from scratch alerts:

  • Send alerts if a customer’s average request rate goes above 13 requests per second over 2 minutes.
  • Send alerts if disk utilization goes above 75%.

Percentage change

Use this template to alert on changes in your data over time. With this option, you can set alert thresholds comparing current data to data from the last minutes, hours, days, or weeks. Choose this template if you have highly variable data, making it hard to identify fixed thresholds for system health.

Here are some examples of Percent change alerts:

  • Send alerts if API request rates have increased more than 5 times (500%) compared to last week.
  • Based on previous months of data, send alerts if disk utilization is projected to go above 80% within 30 days.

Step 3: Build your query

In the Query section, use the Unified Query Builder to create your query.

When building your query, keep in mind the following:

  • You must define an input window if you’re not using latest in the Aggregate input.
  • You can’t use Group by or regular expressions for span queries.
  • You can only choose one percentile to query on when creating a latency-based query.

You can also use existing queries on dashboards or existing queries on notebooks to create alert queries.

The sections below show examples of a Start from scratch query and a Percentage change query.

Start from scratch

The example below queries request rates for the customer PackingKings grouped by method name, aggregated by the mean of all values over a final two-minute rolling input window. Note that the Analysis chart visualizes the query results.

Visit Use the Unified Query Builder for more on configuring queries, including how to use formulas and compare data.

Start from scratch query example

Percentage change

The example below queries request rates aggregated every 30 minutes (a), compared to the same data from one hour ago (b). The join formula (abs((a-b)/b*100)) calculates the absolute percentage change between a and b.

The Analysis chart visualizes the query results, showing all three series (a, b, and abs((a-b)/b*100)).

Percentage change query example

Your query’s join formula affects how you’ll configure alert thresholds in step 4.

The join formula above (abs((a-b)/b*100)) uses the abs() arithmetic operator. abs() returns non-negative values, for example, the absolute values of -2% and 3% are 2% and 3%. With abs(), you’ll set alert thresholds for the overall change instead of the change’s direction. For example, if your alert threshold is above 150%, you’ll be notified if the percentage change is either -151% or 151%.

The same formula without abs() ((a-b)/b*100) lets you set alert thresholds based on the direction of the change. For example, if your alert threshold is above 150%, you’ll be notified if the percentage change is 151% but not -151%. Similarly, if your alert threshold is below -150%, you’ll be notified if the percentage change is -151% but not 151%.

Step 4: Configure your alert

Follow these steps to configure your alert:

  1. On the same page, go to the Alert configuration section.
  2. If your query includes Group by, click a single notification and select one of these options to configure notification behavior:
    • a single notification - Get only one notification if one or more group values cross the threshold during the input window. For example, if two group values cross the threshold in a 30-minute input window, you get one notification.
    • separate notifications - Get a notification every time a group value crosses the threshold during the input window. For example, if two group values cross the threshold in a 30-minute input window, you get two notifications.
  3. Click the above drop-down and select one of the following:
    • above - Send notifications when the metric goes above the threshold.
    • below - Send notifications when the metric goes below the threshold.
  4. To set alert thresholds, select the checkbox next to Critical threshold and/or Warning threshold and enter your values in the inputs. The Analysis chart plots the thresholds to help you configure them.
  5. If you want to be notified if Lightstep isn’t receiving any data for the query, select Notify if no data is reporting for this query.

For Percentage change alerts, you may see this note under Alert configuration: Your alert will activate in X days when Lightstep has enough data to run the alert query.

Because Percentage change alerts use historical data, Lightstep may need to wait and collect that data before evaluating your alert. Lightstep automatically activates the alert when ready. You can check your alert’s status in Lightstep’s Alerts tab.

Step 5: Configure your notification rules and save your alert

Follow these steps to set your notification rules and activate your alert:

  1. Under Notification rules, click Search for a destination type to select the third-party tool you use for notifications.
  2. Click Search for the destination name and enter the name of your notification destination. Lightstep shows your options in a drop-down.
  3. Make these optional changes to your notification rules setup:
    • By default, Lightstep sends notifications when data crosses the configured threshold. If you want to receive notifications until the issue is resolved, click Add notification frequency and configure the rate.
    • To send notifications to another destination, click Add another rule and repeat the steps above.
  4. Review your changes and click Save to activate your alert. Lightstep displays Alert saved and returns you to the Alerts tab.

Next steps

You’re all set! You now have an alert. Visit these links to learn more about Lightstep alerts: