Step 3: Create a dashboard and charts

Now that you’ve sent telemetry data to Lightstep Observability, it’s time to build a unified dashboard and charts to monitor that data.

Run your service or app

The first step is to run your app so that it starts generating metrics and span data that you can see immediately in Lightstep Observability. Let it run for a few minutes before you start building the dashboard.

Create a dashboard

Dashboards can contain both metric and span charts that are built by querying of your telemetry data.

1. From the navigation bar, click Dashboards.

2. Click Create a dashboard.

3. Click in the title to rename your dashboard.

   You’re now ready to add a chart to the dashboard.

4. In your dashboard, click Add a Chart.

5. In the Chart Builder window, click into the title to name your chart.

6. Build the query for your chart. For this learning path, create at least one metric chart.

Learn how to use the Unified Query Builder

1. Search for a metric to plot:
   Click into the search field and begin typing the metric name.

   You can select Metric from the All telemetry dropdown to narrow your search to only metric data.

   

   Change Intelligence works best when it can focus on a single service and its dependencies. To narrow your search to metrics emitted from a specific service, click Filter to service and select the service name from the dropdown.

   When you select a metric, Lightstep expands the query builder and begins to chart the metric.

2. Filter the data:
   Unless you’ve already filtered by a service, all data for the metric is displayed. You can filter the data using metric attributes found on the data.

   Enter an attribute key, select an operator or enter a regular expression, and enter one or more values. Multiple values are joined by OR.

   You can add more than one filter to the query where it makes sense (the query builder prunes the available list as you add attributes).

   Multiple filters use AND to join filters.

   

   By default, Lightstep Observability displays attributes that it’s seen in the last three days. But you can type in an attribute not in the dropdown and Lightstep will find it.

3. Align the data by choosing an aggregation method. Alignment is the process by which time series data points are aggregated temporally to produce regular, periodic outputs. By aligning your data before plotting it, you can tell a clearer story and more easily see anomalies. You can also set an explicit rolling input window that determines the data points that will be used when computing the aggregation.

If you use the latest aggregation, no input window is needed.

More about alignment and time windows

Whenever you make a query, Lightstep determines the number of output points to display that will make the chart detailed without making it feel crowded or hard to read. This optimal distance between points is called the output period. Lightstep adjusts the output period depending on the the amount of time being displayed on the chart. If you set the time picker to one week, the output period is 2 hours (meaning that the query produces a series of points which are all two hours apart). If you change the time picker to one hour, the output period is 30 seconds.

For example, say you are querying the requests metric and the data is comming from two sources. Using the count aggregation, Lightstep combines the data from those sources to create individual time series points and then plots those on the chart, based on the output period. If you queried over the last 60 minutes, the output period (the space between each point on the graph) is 30 seconds.

The rolling input window is the duration of time that the data is pulled from before aggregating it. By default for charts on dashboards and notebooks, the input window is the same as the output period. That is, if the output period is 30 seconds, the data is aggregated over the last 30 seconds to produce the data point.

If your chart appears very noisy, or if you’re building a chart for an alert, it can be helpful to smooth the output data by using data from a wider rolling input window. For example, if you are querying on the requests metric and choose to aggregate using rate with a 10 minute input window, Lightstep computes the number of requests per second in 10 minute rolling time windows.

When querying for alerts, you must specifically set an rolling input window. Increasing the input window means increasing the time that it will take before a underlying change in the data will trigger an alert threshold. When it’s important to be notified immediatly, use smaller input windows.

Expandable end

Choose one of the following temporal aggregation methods:

• Count: Computes the total number of increments in the input window as whole numbers. Counts are most useful for infrequent events and are best visualized as stacked bar charts.

• Rate: Computes the number of operations per second in the input window. Rates are most useful for ongoing operations and are best visualized as line charts

Gauge metrics also allow the following aggregations:

• Latest: Computes the latest value in the input window. When using latest, the input window is the same as the output period.

• Mean: Computes the average value of the time series in the input window.

• Max: Computes the maximum value of the time series in the input window.

• Min: Computes the minimum value of the time series in the input window.

The query builder automatically configures the aggregation for distribution type metrics. If the distribution is a gauge, the aggregation is set to latest. If it is a counter (delta or cumulative), the aggregation is set to rate by default, but you can change it to count.

4. Set the rolling input window (Unless using latest aggregation. Alerts require an input window):
   By default for charts in notebooks and dashboards, the input window is the same as the output window (determined by the time period for the chart). By setting an explict input window, you can smooth out your data to avoid noise.

5. Compute percentiles (distribution type metrics only):
   When the metric data is a distribution type (a set of values for each point in time), Lightstep Observability can compute percentiles for you. Enter the value in the Percentile field (no % needed).

   For existing Lightstep customers interested in tracking distribution metrics, please opt-in here. For new customers to Lightstep, this feature is already enabled in your account.

   When using distributions in an alert, you must select only one percentile to alert on.

6. Group the data:
   By default, Lightstep Observability spatially aggregates the data from the metric into one line.

   

   Instead, you can show lines for each available attribute value (group by). Select an attribute to display lines for each of the attribute’s values.

   In this example, by choosing to group by the host attribute, you can see the metrics for the individual hosts.

   Grouping isn’t available on big number charts.

7. Choose how you want the data spatially aggregated into the chart.

• count of non-null values: The number of values found that are not null. For example, given the values of [10, 15, null, 50] the count is 3.
• count of non-zero values: The number of values found that are not zero (null is counted). For example, given the values of [10, 15, null, 0 50] the count is 4.
• maximum value: The highest point in the data.
  For example, given the values of [10, 15, 50] the max is 50.
• mean of all values: The average (sum of the data divided by the count) of the data.
  For example, given the values of [10, 15, 50] the mean is 25.
• minimum value: The lowest point in the data.
  For example, given the values of [10, 15, 50] the min is 10.

• sum of all values: The total of all points in the data.
  For example, given the values of [10, 15, 50] the sum is 75.

  Distribution type metrics are automatically summed and then aggregated into percentiles.

8. Click Save to save your chart.

When you hover over a point on the chart, you can see its value, along with the group-by value.

Clicking on a point allows you to start Change Intelligence to determine what cased the change in performance.

Below the chart, a table displays the data for each line in the chart.

Expandable end

7. Click Save to save your changes. The chart now displays on the dashboard.

You can add as many charts as you need for the dashboard.

Now that you have your data reporting and displayed in Lightstep Observability, it’s time to see how that data and data from trace instrumentation work together to find root causes of metric deviations deep in your stack.

What did we learn?

• Dashboards can contain any number of charts for both your metric and span data.
• Charts are built from queries to your data. You can aggregate, filter, and group that data so that the chart reports just the data you need.