The following table shows how long your metric and span data is stored for by default, and whether or not that time period is configurable. Continue reading the topic to understand the retention time periods and why you may want to change them.
|Data type||Default Retention||Configurable?|
|Metric data||13 months||No|
|Span data||3 days||No|
|Streams - SLI charts||2 years||No|
|Streams - exemplars||28 days||Yes|
|Notebook auto-saved queries||28 days||Yes|
|Notebook snapshots||28 days||Yes|
To understand how long data is retained in Lightstep, it’s important to first understand how data is obtained and stored. Lightstep ingests two major types of data, metrics from your metrics instrumentation (like OpenTelemetry or AWS) and span data from your OpenTelemetry instrumentation.
Collection and storage
Lightstep ingests metric data from a dedicated endpoint (
https://metricingest.lightstep.com) and stores each unique combination of a metric name, timestamp, and attribute name/value pairs as a time series in a time series database. That data, once ingested, can be queried to create charts in notebooks, dashboards, and alerts.
If your metric is a distribution type (that is, it automatically aggregates values by percentiles), then each percentile becomes a separate time series.
Your time series are retained for 13 months. You can create a chart and view the corresponding data up to 13 months in the past (or up to whenever it began reporting to Lightstep). If you change the chart to a time period that exceeds the retention period, the chart displays hashed lines to let you know data isn’t available.
Collection and storage
Your span data is handled differently. Instead of ingesting that data directly into a database, Lightstep uses Microsatellites to collect that data from your OpenTelemetry instrumentation. That data is then sent to the Lightstep Observability platform, which then processes and temporarily stores that data during trace assembly. The length of time between the newest and the oldest span currently held for analysis is the retention window, where it can be queried by charts in notebooks or dashboards, and viewed as performance indicators in the Service directory.
By default, the retention window for span data (including example traces) is three days from the time you make the query. Each query from a chart and each visit to the Service directory returns up to the past three days of data that match the query (you can control the amount of time shown on a chart using the time picker). If the time window for a chart is set to beyond three days from when you made the query, the chart displays hashed lines to show that data is no longer available.
Retain span queries as Streams for dashboards
While three days is the default retention, sometimes you need to see data beyond that period, especially when creating dashboards for monitoring purposes. When you know you will need more than three days of data, you can create the query as a Stream to retain the query, and Lightstep Observability will continuously collect and save that data from the Microsatellites. The span’s data and and any associated traces are retained for as long as your data retention policy. The service level indicators (SLIs) shown on a chart are stored for two years, regardless of your data retention policy.
The default data retention policy is 28 days. You can configure your data retention policy on the Project Settings page.
Lightstep charges for the amount of span data collected by the Microsatellites, so choosing to save data beyond their three day retention window has an impact to your monthly charges.
Auto-saved span data in notebooks
Data retention for charts in notebooks is handled a bit differently. As with dashboards, by default you can explore up to the past three days of span data. But because your notebooks become useful tools in rituals like post-mortems, you often want to have a record of the data you worked with as you investigated an issue. Lightstep automatically saves that data from when you created the chart, so you can continue to view the chart and the example traces from that time period.
If you try to change the query once it’s auto-saved (for example, by adding filters or changing the group-by), you’ll be overwriting the original query and may lose that data. You can duplicate the original query onto a new chart to get the most recent data or to make changes, without losing the original data.
The auto-saved chart is retained for as long as your data retention policy.
Additionally, sharing information becomes important during an investigation. You can share a read-only snapshot of a notebook at any point in time and that snapshot is also saved for the period of your data retention.