Metrics dashboards are made up of individual charts that display your metric data. Each chart represents a query against that data.

Before you create charts, read how Lightstep displays metric data to understand the different metric kinds and how your configuration choices affect how your data is displayed.

Create a Metrics Chart

You create charts by querying your metric data. You can add multiple queries and also add formulas.

Add a Query

  1. From an open dashboard, click Add a Chart. Add a metrics chart

  2. In the Chart Builder window, click into the title to name your chart. You can rename it at anytime by clicking into the title. Enter chart name

  3. Build your query:

    • Search for a metric to plot: Click into the search field. Lightstep displays all metrics that it’s currently ingesting. Choose one, or to search, start typing the name of a metric. Lightstep starts auto-completing to match available metrics.

      Metric search

      Click the info icon to view and edit the metric’s details (like the description or units) in the details panel.

      When you select a metric, Lightstep expands the query builder and begins to chart the metric.

    • Choose an operator for the data:

      Lightstep automatically configures the operator for distribution type metrics. If the distribution is a gauge, the operator is set to latest. If it is a counter (delta or cumulative), the operator is set to rate by default, but you can change it to delta.

      • Latest: Graphs the latest value in a time series for a point in time.

      The latest operator can only be used with gauge-type metrics.

      • Delta: Computes and graphs the total number of increments as whole numbers. Deltas are most useful for infrequent events and are best visualized as stacked bar charts.

      • Rate: Graphs the number of operations per second. Rates are most useful for ongoing operations and are best visualized as line charts.

        For example, this chart uses Delta and shows there are a total of 364.474 requests at 11:45 am. Count used

        This chart has the same query but uses Rate, and shows there are 8.099 requests per second at 11:45 am. Rate used

    • Compute percentiles (distribution type metrics only): When the metric data is a distribution type (a set of values for each point in time), Lightstep can compute the 50th, 90th, 99th, and 99.9th percentiles for you.

      For existing Lightstep customers interested in tracking distribution metrics, please opt-in here. For new customers to Lightstep, this feature is already enabled in your account.

      You can control the percentiles displayed on the chart using the display toggles.

      Toggle percentile display

    • Filter the data: By default, all data for the metric is displayed. You can filter the data using metric tags found on the data. You can include or exclude data with a given tag and value. You can add more than one tag to a filter where it makes sense (Lightstep prunes the available list as you add tags).

    Multiple selections use AND to join filters.

    Change Intelligence works best when it can focus on a single service and its dependencies. If your query includes metrics coming from many services, use filters to choose one service to focus on.

    Filter data with tags

    By default, Lightstep displays tags that it’s seen in the last three days. But you can type in a tag not in the dropdown and Lightstep will find it.

    • Group the data: By default, Lightstep aggregates the data from the metric into one line.Group all by default

      Instead, you can show lines for each available tag value (group by). Select a tag to display lines for each of the tag’s values. In this example, by choosing to group by the method tag, you can see the metrics for the individual tag values. Grouped by method

      Grouping isn’t available on big number charts.

    • Aggregation method: Choose how you want the data aggregated into the chart.

      • Count (non-null): The number of values found that are not null. For example, given the values of [10, 15, null, 50] the count is 3.
        • Count (non-zero): The number of values found that are not zero (null is counted). For example, given the values of [10, 15, null, 0 50] the count is 4.
        • Mean: The average (sum of the data divided by the count) of the data.
          For example, given the values of [10, 15, 50] the mean is 25.
        • Min: The lowest point in the data.
          For example, given the values of [10, 15, 50] the min is 10.
        • Max: The highest point in the data.
          For example, given the values of [10, 15, 50] the max is 50.
        • Sum: The total of all points in the data.
          For example, given the values of [10, 15, 50] the sum is 75.

Distribution type metrics are automatically summed and then aggregated into percentiles.

  1. Click Save to save your changes. The chart now displays on the dashboard. New chart on dashboard

We cap the number of time series displayed to 1,000 to avoid unnecessary noise. If your query returned more than that, note that some were omitted. If you want to view all series, you may want to omit using “group by” to reduce the cardinality.

Add Multiple Queries to a Chart

You can display more than one metric on a chart. For example, you might want to show the request rate for iOS and Android on one chart.Two metrics on one chart

To add a metric query, click Plot another metric and build your query as you did the first one. Add a query

When you have multiple queries, you can edit the chart so only certain time series display. For example, in this chart, only the timeseries for metrics from iOS is displayed. Toggle time series

Once you save the chart, this display toggle is persisted to the chart in the dashboard.

What happens when you delete a query?

You can delete a metric by clicking the X for that row. When you do, the remaining metrics retain their order (for example if you deleted b, the remaining metrics are a and c). If you then add another metric, it uses the order that was deleted. If you continue to add metrics, the order continues down the alphabet from the “highest” letter. b was deleted. Now a new metric uses b. In the above example, three metrics were originally plotted: a, b, and c. The user deleted b, so the next metric plotted used b. When adding another metric, the order continued to d.

expandable end

If you want to use a big number chart with more than one metric, you need to combine them using a formula (big number charts can only display a single value).

Add a Formula to the Query

You can perform arithmetic on a single time series or on multiple time series using Add a formula. For example you might enter a/(a+b) if you want to chart the percentage of the a metric to the sum of the a+b metrics.

Lightstep supports +, -, /, and *.

When using a metric that is a distribution type in a formula, you must select only one percentile.
If you’re performing the arithmetic on multiple queries, they must all be grouped by the same tag.

You can edit the chart so only the formula is shown. For example, in this chart, only the timeseries for the result of the formula is displayed.

Toggle time series Once you save the chart, this display toggle is persisted to the chart in the dashboard.

Troubleshoot Query Results

If your chart doesn’t look as expected, it may be because of one of the following:

  • The No data found message displays when Lightstep can’t find a metric by that name. Ensure you are using the right name for the metric.

  • The No data found message also displays when you’re using the wrong time series operator for the metric kind.

    The latest operator can only be used with gauge metrics.

  • If no data displays and there’s no No data found message, then Lightstep found the metric, but had no data to display

  • If adding a formula over multiple queries, they must all be grouped by the same tag.

Change a Chart’s Display

You can change the time period that a chart reports for and also change the type of chart to use.

Change the Time Range

By default, the data shown is from the last hour. You can change that to a number of different time periods, up to one week in the past, using the dropdown. Use the < > controls to move backwards and forwards through time. Change the time range

Change the Chart Type

You can choose the type of chart used to display metrics.

Depending on the metric kind or type, only certain chart types may be available.

  • Line: Displays a line that connects the charted values.Line chart
  • Bar: Displays a bar for the value at a given time point. Bar charts are good for displaying discrete values, like the number of requests.Bar chart
  • Area: Displays a shaded area for the value. Area charts are helpful when you’re charting more than one metric.Area chart
  • Big number: Displays an aggregated value over time as a number. Big number charts are useful when you only need to know the current value of a metric and don’t expect to investigate any changes. When creating a big number chart, note the following:
    • To determine the number to display, select an aggregation method from the dropdown and a time period for that aggregation. For example, if you want to show the maximum value over the last four hours, you’d select maximum value and The Past 4 Hours.

      This aggregation is different than the aggregation used when building the query. For a query, the data is aggregated to determine the value for each data point in a time series. For a big number chart, those values are aggregated a second time to determine the final value to display.

    • Click under the number to add a subtitle.
    • Because a big number chart displays a single value, you can not group by an attribute. If you use more than one metric for your chart, you need to combine them using a formula. Big number chart

Delete a Chart

To delete a chart, use the More ( ⋮ ) icon and choose Delete. Delete a chart

Edit a Chart

You access and edit a chart from the Dashboard.

  1. From the navigation bar, click Dashboards and open the dashboard your chart is on.
    You can sort the dashboard list by type to bring all Metrics dashboards to the top or search for a dashboard. Dashboards sorted by Metrics

  2. Hover over the chart, click the More ( ⋮ ) icon and select Edit. Edit icon for a chart The chart opens in the builder where you can edit it.

Add a Chart to a Notebook

You can add a chart to a notebook for when, during an investigation, you want to be able to run ad hoc queries, take notes, and save your analysis for use in postmortems or runbooks. Notebooks

To add a chart to a notebook, click Add to notebook and search to choose an existing notebook or create a new notebook. Add chart to notebook

Receive an Alert if no Data is Reporting

You can create an alert that notifies you if no data is collected for the query over a period of time. Follow the instructions for configuring an alert and select Notify if no data is reporting for this query. Alert for no data

Investigate a Deviation in the Chart

Once you have the chart open in the editor, you can use Lightstep’s Change Intelligence to determine what cased the change in performance. Change Intelligence links metric data with trace data to find components in your system whose performance changed at the same time as the metric change, allowing you to find the root cause without leaving Lightstep.

You can’t use Change Intelligence with big number charts.

Change Intelligence

If you’ve made any edits to the chart, you need to save those changes before using Change Intelligence.