Lightstep’s Unified Query Builder allows you to query both your metric and span data using one tool. You use the same builder across notebooks and dashboards to create charts that visualize your data. The builder allows you to quickly and easily ask questions of your data and see the results in one place.

Unified Query Builder

Query metric data

  1. Search for a metric to plot:
    Click into the search field and begin typing the metric name.

    You can select Metric from the All telemetry dropdown to narrow your search to only metric data.

    Search for metric

    Change Intelligence works best when it can focus on a single service and its dependencies. To narrow your search to metrics emitted from a specific service, click Filter to service and select the service name from the dropdown. Filter to a service

    When you select a metric, Lightstep expands the query builder and begins to chart the metric.

  2. Filter the data:
    Unless you’ve already filtered by a service, all data for the metric is displayed. You can filter the data using metric attributes found on the data.

    Enter an attribute key, select an operator or enter a regular expression, and enter one or more values. Multiple values are joined by OR.

    You can add more than one filter to the query where it makes sense (the query builder prunes the available list as you add attributes).

    Multiple filters use AND to join filters.

    Filter data with attributes

    By default, Lightstep Observability displays attributes that it’s seen in the last three days. But you can type in an attribute not in the dropdown and Lightstep will find it.

  3. Use the Represent as field to choose an operator for the data:

    • Latest: Graphs the latest value in a time series for a point in time.

    The latest operator can only be used with gauge-type metrics.

    • Count: Computes and graphs the total number of increments as whole numbers. Counts are most useful for infrequent events and are best visualized as stacked bar charts.

    • Rate: Graphs the number of operations per second. Rates are most useful for ongoing operations and are best visualized as line charts

      The query builder automatically configures the operator for distribution type metrics. If the distribution is a gauge, the operator is set to latest. If it is a counter (delta or cumulative), the operator is set to rate by default, but you can change it to count.

  4. Compute percentiles (distribution type metrics only):
    When the metric data is a distribution type (a set of values for each point in time), Lightstep Observability can compute the 50th, 90th, 99th, and 99.9th percentiles for you.

    For existing Lightstep customers interested in tracking distribution metrics, please opt-in here. For new customers to Lightstep, this feature is already enabled in your account.

    When using distributions in an alert, you must select only one percentile to alert on.

  5. Group the data:
    By default, Lightstep Observability aggregates the data from the metric into one line.

    Group all by default

    Instead, you can show lines for each available attribute value (group by). Select an attribute to display lines for each of the attribute’s values.

    In this example, by choosing to group by the host attribute, you can see the metrics for the individual hosts. Grouped by method

    Grouping isn’t available on big number charts.

  6. Aggregation method:
    Choose how you want the data aggregated into the chart.

  • count of non-null values: The number of values found that are not null. For example, given the values of [10, 15, null, 50] the count is 3.
  • count of non-zero values: The number of values found that are not zero (null is counted). For example, given the values of [10, 15, null, 0 50] the count is 4.
  • maximum value: The highest point in the data.
    For example, given the values of [10, 15, 50] the max is 50.
  • mean of all values: The average (sum of the data divided by the count) of the data.
    For example, given the values of [10, 15, 50] the mean is 25.
  • minimum value: The lowest point in the data.
    For example, given the values of [10, 15, 50] the min is 10.
  • sum of all values: The total of all points in the data.
    For example, given the values of [10, 15, 50] the sum is 75.

Distribution type metrics are automatically summed and then aggregated into percentiles.

When you hover over a point on the chart, you can see its value, along with the group-by value. Hover over a point

Clicking on a point allows you to start Change Intelligence to determine what cased the change in performance. Change Intelligence

Below the chart, a table displays the data for each line in the chart. Table for chart

Query span data

  1. In the first field, select operation or service, or start typing an attribute key name.

    You can select Spans with from the All telemetry dropdown to narrow your search to only span data.

    Search for operation

    By default, the query builder displays attribute keys that it’s seen in the last three days. But you can type in an attribute not in the dropdown and Lightstep will find it.

    Select an operator or enter a regular expression, and enter one or more values. Multiple values are joined by OR.

  2. Add an optional filter.

    You can further refine your query by adding adding a service, operation, or attribute key and value(s) to a filter where it makes sense (Lightstep prunes the available list as you add filters).

    Filter the query with attributes

    Multiple filters use AND to join filters.

  3. Optionally group the results.

    The builder aggregates the data from the span’s performance into one line. You can show lines for each available attribute value (group by). Select an attribute to display lines for each of the attribute’s values. In this example, by choosing to group by the client.platform attribute, you can see the percentiles for the individual platform values. Group by attribute

    You must be using Microsatellites to add a group-by.

  4. Use the Compute dropdown to have the chart display either latency percentiles (50th, 95th, 99th, and 99.9th), error rate, operation rate, or count (the number of spans). Compute

    By default, the 50th, 95th, 99th, and 99.9th percentiles are shown. You can use the toggles to select the ones to view. Toggle percentile views

The result of the query displays in a chart below the query builder.

By default, the chart shows lines for each series (group-by), and dots for sampled spans. Triangles represent spans that have errors. Charted query

Use the Show span samples toggle to turn these off. Turn sample spans off

With spans turned off, when you hover over a line, you can see it’s value and percentile. Span data

With span samples displayed, when you hover over a point, you can see the value at the point along with the group-by value. View span data

Clicking the point takes you to its full trace, where you can view the full request path that the span participated in. In this case, the error is coming from the GET operation on the store-server service. Full trace

Below the chart, a table displays the data for each line in the chart.

Table displays data

With sample spans displayed, the Value column shows the latest value for that series.

You can collapse both the query and the table.

Add multiple queries to a chart

You can more than one query to a chart. For example, you might want to show the request rate for iOS and Android on one chart.Two queries on one chart

For alerts, if you add more than one query, you must join them with a formula.

To add a query, click Plot another metric or Plot another span and build your query as you did the first one.

Add a query

Add a query

When you have multiple queries, you can edit the chart so only certain time series display. For example, in this chart, only the timeseries for metrics from the iOS service is displayed. Toggle time series

Once you save the chart, this display toggle is persisted to the chart in the dashboard.

What happens when you delete a query?

You can delete a query by clicking the X for that row. When you do, the remaining queries retain their order (for example if you deleted b, the remaining queries are a and c). If you then add another query, it uses the order that was deleted. If you continue to add queries, the order continues down the alphabet from the “highest” letter. b was deleted. Now a new query uses b. In the above example, three queries were originally plotted: a, b, and c. The user deleted b, so the next query plotted used b. When adding another query, the order continued to d.

expandable end

If you want to use a big number metric chart with more than one metric, you need to combine them using a formula (big number charts can only display a single value).

Add a formula to the query

For span queries, you must be using Microsatellites to add a formula.

You can perform arithmetic on a single time series or on multiple time series using Add a formula. For example you might enter a/(a+b) if you want to chart the percentage of the a metric to the sum of the a+b metrics.

Lightstep Observability supports +, -, /, and *.

You must use * for multiplication. Implicit multiplication (for example, ab) is not allowed.

When using a metric that is a distribution type in a formula, you must select only one percentile.
If you’re performing the arithmetic on multiple queries, they must all be grouped by the same attribute.

You can edit the chart so only the formula is shown. For example, in this chart, only the timeseries for the result of the formula is displayed.

The toggle display doesn’t affect when the alert is triggered. Alerts are triggered only on the result of the formula

Toggle time series Once you save the chart, this display toggle is persisted.

Troubleshoot query results

If your chart doesn’t look as expected, it may be because of one of the following:

  • The No data found message displays when Lightstep Observability can’t find a metric or span attribute key (service, operation, or attribute) by that name. Ensure you are using the right name in the query.

  • The No data found message also displays if you’re using the wrong time series operator for the metric kind.

    The latest operator can only be used with gauge metrics.

  • If no data displays and there’s no No data found message, then Lightstep Observability found the metric or span attribute key, but had no data to display

  • When adding a formula over multiple queries, they must all be grouped by the same attribute.

Data retention

Data for metric queries is retained for 13 months.

For notebooks, span data is retained for three days. For a chart on a dashboard, you can choose to retain the data for longer by creating it as a Stream. Data for alerts is always saved as a Stream.

Learn more about data retention.