When you are in a performance investigation, you often need to run ad hoc queries to see the performance of your system, both at the infrastructure level and app level, and then share your findings with your team. Instead of using multiple tabs or tools and creating “throw-away” dashboards, Lightstep provides notebooks that allow you to query and save both your infrastructure (metrics) and app performance (traces) data in one place.

You can add text blocks to note and explain your findings and share your notebook with anyone with a Lightstep account. The data for your trace-based charts is kept for three days and data for metric-based charts is kept for as long as your metric retention policy, so you once you mitigate an issue, you can go back to the data you worked with to fully investigate and remediate.

Notebooks view

You use the query builder to create and filter charts for your notebook for both metric and trace data. From metric charts, you can run Change Intelligence directly from a notebook to begin your investigation.Run Change Intelligence

From trace data charts, you can view example traces that have high latency, error, or operation rate. View exemplar traces

To view a trace, make sure the Show span samples toggle is on. Spans with errors are shown as a triangle. Spans without errors are shown as dots. Click on either one to view the associated trace.

When to Use Notebooks

Notebooks are helpful anytime during an investigation when you want to create ad-hoc queries, especially when you want to query both metric and trace data. They are also useful when you want a record of your investigation for postmortems or runbooks.

Here are some examples of how you might use notebooks:

  • Ad-hoc queries: You can build ad-hoc queries in a notebook and compare the charts and run Change Intelligence or view traces from there.
  • Collaboration: When your team needs to collaborate during an investigation, you can each add your own charts and text blocks to the notebook and work with the same data.

    When someone adds a chart or text block to a notebook, you will need to refresh your page to see any additions/changes. If you’ve shared your notebook and know that someone else may be editing it, best practice is to refresh the page after each change you make to ensure you don’t overwrite each other’s edits.

  • Postmortems and runbooks: When you use notebooks during an investigation, you can add your steps to resolution to the notebook once you’ve resolved the issue.

Users with Viewer permissions can create, view, and copy any notebook, but can only edit and delete their own.

Notebooks are saved indefinitely, but the data for span-based charts is available for three days and data for metric-based charts is available for the duration of your retention policy.

Work with Notebooks

You access notebooks from the Notebook icon in the navigation bar. Notebook icon in nav bar

Existing notebooks are listed in the sidebar in the order they were created (latest on top). You can see when the notebook was last edited and by who.

You can use the Search field to find a notebook by searching on name or user. Search notebooks

You can favorite a notebook by clicking the star. Your favorite notebooks are moved to the top of the list. Favorite a notebook

You can create notebooks from the notebook view. You can also add existing metric charts to a notebook directly from metric charts or alerts.

Create a Notebook

You can create a notebook from scratch from the Notebooks view and then query metric or trace data to add charts, or you can add metric charts to a new or existing notebook from metric dashboards and alerts.

Create a Notebook from Scratch

You create Notebooks from scratch from the Notebooks view.

To create a notebook from scratch:

  1. From the navigation bar, click Notebooks.

    Existing notebooks for the projects are shown on the left. Notebook icon in nav bar

    A new (unsaved) notebook shows the query builder. New notebook

    If an existing notebook is displayed, click Create notebook from the sidebar. Create notebook

  2. Click into the fields to add a title and an optional description for the notebook. Notebook title

    If you don’t add a title, Lightstep automatically names the notebook after the query, along with a timestamp. Auto title Click Save notebook. Save notebook

    You only need to explicitly save a notebook when you create it. All changes are auto-saved after that point.

You can now add a metric or span chart, text blocks, and share the notebook.

Create a Notebook from an Existing Metric Chart

You can create a notebook from existing metric query.

  1. From a chart on a dashboard click the the More ( ⋮ ) icon and choose Add to a notebook. Add chart from dashboard

    From an open chart or an alert, click Add to notebook. Add chart to notebook

  2. Click Create a notebook to start a new one. Create new notebook

    The query and chart display as the first entry in the notebook. An annotation shows the dashboard or alert it came from. The annotation is a link back to the original, so you can quickly return to the origin of your investigation. Notebook with chart

    The same chart title is used, but you can click into the field to edit it.

  3. Click into the fields to add a title and an optional description for the notebook. Notebook title

    If you don’t add a title, Lightstep automatically names the notebook after the metric used in the query, along with a timestamp. Auto title Click Save notebook. Save notebook

    You only need to explicitly save a notebook when you create it. All changes are auto-saved after that point.

You can now add another metric or span chart, text blocks, and share the notebook.

Add a Metric Chart

  1. If this is the first chart in the notebook, the query builder is open.

    If you’re adding a subsequent chart, to open the query builder click the + icon and choose Build another chart. Add chart

  2. Build your query.

    • Search for a metric to plot: Click into the search field and begin typing the metric name.

      You can select Metric from the All telemetry dropdown to narrow your search to only metric data.

      Search for metric

      Click the info icon to view and edit the metric’s details (like the description or units) in the details panel.

      When you select a metric, Lightstep expands the query builder and begins to chart the metric.

    • Filter the data: By default, all data for the metric is displayed. You can filter the data using metric tags found on the data.

      Enter a tag (attribute) key, select an operator (= or !=), and enter a value.

      You can add more than one tag to a filter where it makes sense (Lightstep prunes the available list as you add tags).

    Multiple filters use AND to join filters.

    Change Intelligence works best when it can focus on a single service and its dependencies. If your query includes metrics coming from many services, use filters to choose one service to focus on.

    Filter data with tags

    By default, Lightstep displays tags that it’s seen in the last three days. But you can type in a tag not in the dropdown and Lightstep will find it.

    • Use the Represent as field to choose an operator for the data:

      • Latest: Graphs the latest value in a time series for a point in time.

      The latest operator can only be used with gauge-type metrics.

      • Count: Computes and graphs the total number of increments as whole numbers. Counts are most useful for infrequent events and are best visualized as stacked bar charts.

      • Rate: Graphs the number of operations per second. Rates are most useful for ongoing operations and are best visualized as line charts

        Lightstep automatically configures the operator for distribution type metrics. If the distribution is a gauge, the operator is set to latest. If it is a counter (delta or cumulative), the operator is set to rate by default, but you can change it to count.

    • Compute percentiles (distribution type metrics only): When the metric data is a distribution type (a set of values for each point in time), Lightstep can compute the 50th, 90th, 99th, and 99.9th percentiles for you.

      For existing Lightstep customers interested in tracking distribution metrics, please opt-in here. For new customers to Lightstep, this feature is already enabled in your account.

      You can control the percentiles displayed on the chart using the display toggles.

      Toggle percentile display

    • Group the data: By default, Lightstep aggregates the data from the metric into one line.Group all by default

      Instead, you can show lines for each available tag value (group by). Select a tag to display lines for each of the tag’s values. In this example, by choosing to group by the method tag, you can see the metrics for the individual method values. Grouped by method

      Grouping isn’t available on big number charts.

    • Aggregation method: Choose how you want the data aggregated into the chart.

      • Count (non-null): The number of values found that are not null. For example, given the values of [10, 15, null, 50] the count is 3.
        • Count (non-zero): The number of values found that are not zero (null is counted). For example, given the values of [10, 15, null, 0 50] the count is 4.
        • Mean: The average (sum of the data divided by the count) of the data.
          For example, given the values of [10, 15, 50] the mean is 25.
        • Min: The lowest point in the data.
          For example, given the values of [10, 15, 50] the min is 10.
        • Max: The highest point in the data.
          For example, given the values of [10, 15, 50] the max is 50.
        • Sum: The total of all points in the data.
          For example, given the values of [10, 15, 50] the sum is 75.

Distribution type metrics are automatically summed and then aggregated into percentiles.

Add a Span Chart

  1. If this is the first chart in the notebook, the query builder is open.

    If you’re adding a subsequent chart, to open the query builder click the + icon and choose Build another chart. Add chart

  2. Build your query.

    • In the first field, select operation or service, or start typing an attribute key name, depending on the telemetry type you want to query on.

      You can select Spans with from the All telemetry dropdown to narrow your search to only span data.

      By default, Lightstep displays attribute keys that it’s seen in the last three days. But you can type in an attribute not in the dropdown and Lightstep will find it.

      Search for operation

    • Select an operator (= or !=), and enter a value for the operation, service, or attribute key.

  3. Refine the query.

    You can filter and group the query results. You can choose to compute latency percentiles, error rates, or operation rates.

    Filter: You can further refine your query by adding adding attribute keys and values to a filter where it makes sense (Lightstep prunes the available list as you add tags). Filter the query with attributes

    Multiple filters use AND to join filters.

    Group: Lightstep aggregates the data from the span’s performance into one line. You can show lines for each available attribute value (group by). Select an attribute to display lines for each of the attribute’s values. In this example, by choosing to group by the client.platform attribute, you can see the percentiles for the individual platform values. Group by attribute

    The result of the query displays in a chart below the query builder.

    Compute: Use the Compute dropdown to have the chart display either latency percentiles (50th, 95th, 99th, and 99.9th), error rate, or operation rate. Compute

    Below the chart, a table displays the data for each line in the chart. Table displays data

    You can collapse both the query and the table.

  4. Refine the chart’s display.

    You can view certain percentiles and turn/on off the markers that show exemplar spans and spans with errors.

    Percentiles: By default, the 50th, 95th, 99th, and 99.9th percentiles are shown. You can use the toggles to select the ones to view. Toggle percentile views

    Exemplars: The chart shows dots for exemplar traces from the span and triangles for any spans that have errors. Use the Show span samples toggle to turn these off. Turn sample spans off

    You can also view only spans with or without errors using the toggles. Toggle error spans

Work with Charts

  • You can change the query for any chart.

    Edit history for notebooks is not saved, so you can’t “undo” a change. If you think you might want to save a query but also want to try variations of the query, best practice is to duplicate the chart first (see bullet below).

  • The time range for charts imported from dashboards or alerts uses the time range from the original chart. Charts added directly to the notebook use the same time range as the chart directly above the new chart.

    You can change the time range using the time picker. When you change the range, you can apply it to just that chart or to all charts in the notebook (the default). Change time range

    When you apply a new time range, the time window is fixed and does not continuously update.
    When you use the < > controls to move backward and forward through time, the time range changes for only that chart.

  • Delete a chart clicking the More ( ⋮ ) icon Delete a chart
  • Duplicate a chart clicking the More ( ⋮ ) icon Duplicate a chart

    The chart title is automatically populated with Copy of [Chart title]

Add Text Blocks

You can add text blocks to enhance your investigation.

Users with Viewer permissions can edit only their own notebooks.

To add a text block, click the plus icon above or below an existing notebook entry. Add a chart or text block

Work with Text blocks

  • Text blocks are saved as plain text.
  • URLs resolve to clickable links.
  • Duplicate a text block by clicking the More ( ⋮ ) icon. Duplicate a note
  • Delete the contents of a text block by clicking the X icon. Delete note's contents
  • Delete the entire text block by clicking the More ( ⋮ ) icon Delete a note

Duplicate a Notebook

You can copy any existing notebook to start a new one.

To copy a notebook, click the More ( ⋮ ) icon and choose Duplicate notebook. Duplicate notebook

The notebook is saved as Copy of [Notebook title]. Click into the field to edit it.

Delete a Notebook

You can delete a notebook. When you do, all included charts and text blocks are also deleted.

Users with Viewer permissions can delete only their own notebooks.

To delete a notebook, click the More ( ⋮ ) icon and choose Delete. Delete notebook

Share a Notebook

You can share a notebook with any other Lightstep user in your organization. Users with Admin or Member roles can edit the notebook, including changing queries, time ranges, and text blocks. Users with Viewer roles will see a read-only version. They can’t edit the query, add notes or charts, or change the time range.

Read only notebook

To share a notebook, click the Share button and copy the link. Share a notebook

When someone adds a chart or text block to a notebook, you will need to refresh your page to see any additions/changes. If you’ve shared your notebook and know that someone else may be editing it, best practice is to refresh the page after each change you make to ensure you don’t overwrite each other’s edits.