You have SLAs you have to meet and SLIs you have to monitor. You have found issues in your system that are being resolved, but you want to verify that the fixes work, or that the issue doesn't get worse before the fixes arrive. You want to make sure deployments don't result in regressions. These are all great reasons to monitor your system regularly. But to monitor these very specific things, you need a tool where you can create customized, specific ways to collect data proactively through time, and persist that data in a way that's easily accessible. That's where Streams come in.
When you create a Stream that's based on a query in Explorer or based on a specific operation, LightStep receives data from your Satellites and stores statistics and example traces to ensure you always have data from 0 to p99.9, including outliers. The Stream view displays statistical time series data and example traces and stores them for as long as your Data Retention policy allows.
Best Practice - Create streams for your ingress operations
Usually, ingress operations for a service are high level enough that they indicate performance problems within a single service, and granular enough that finding the root cause of a performance problem is straightforward.
For example, here's a Stream for the performance of the
ios-client service specifically for the customer BEEMO.
The first time series shows latency over a period of time. At the top, you can see the average duration of spans in the p50, p90, p99, and p99.9 buckets. The second series shows the operation rate over time, and the bottom series shows the error percentage rate over time.
When you hover over the latency series, you can view individual spans as dots in a scatterplot. This makes it very easy to see the actual distribution of latency and immediately notice outliers. You can click on any of the dots to open the full trace in Trace view.
Below the time series is a table that displays the historical spans used to create the stream. You can click a tag to run a query on that tag value in Explorer, and click the Trace to view the full trace in Trace View. A red line in the margin indicates an error was found on that trace.
The URL for a Stream is stable and shareable; perfect for alert messages, slack messages, and bot auto-responses. You can share a shortened URL for the Stream by clicking the Share button.
Once you're done creating Streams, consider bundling them into dashboards adding alerts to notify team members when thresholds in a Stream are crossed. You can also embed Streams into many third-party applications.
Create Streams proactively
LightStep starts collecting data for Streams once you create them, so when you create a Stream, it will initially be blank. Think about and create Streams before you actually need them!
You create a Stream based on queries made using Explorer. LightStep continuously monitors for span data based on that query and saves it as a Stream.
Want to programmatically create and manage Streams?
Use our public API to list, retrieve, update, and create Streams. You can also retrieve a specific time series for a Stream.
To create a stream:
- Open Explorer from the navigation bar and run any query.
- Once the query is run, click Create Stream.
Button grayed out?
If the button is grayed out, then a Stream already exists for that query.
- To view the Stream, click View Stream.
You won't see much data at first because the Satellites have just been told to start collecting it. Over time, the Stream becomes more populated.
- By default, the Stream takes the name of the query. You can change that to be more descriptive by clicking into the title and entering a name of your choice.
By default, LightStep displays all data for a Stream from the last 24 hours. You can change that time period. You can also filter the data by errors, percentiles, and duration.
Use the time period dropdown to choose a different period or enter a custom time period. Click Pick Time Range to enter a custom time period from a calendar. The time series and the Historical Spans table redraw to show spans just from that time period.
To view spans with errors, use the Filter dropdown and choose Errors. The series redraws to display only data for spans with errors and the Historical Spans table shows the spans with errors.
The scatterplot of spans does not display when filtering by errors.
By default, LightStep displays lines for the p50, p90, p95, and p99 latency percentiles. You can filter the data to display a specific percentile and above. Use the Filter dropdown and choose Percentile. Enter a value and LightStep redraws the graph and table to show just that percentile and above. You can change the value by clicking it to reopen the dialog.
For example, this data is filtered to show p99.5 and above.
By default, LightStep shows all spans, regardless of duration. Use the Filter dropdown, choose Duration, and enter a time period to see where that duration lies in the time series. LightStep redraws the graph to show where that duration lies, and the scatterplot only shows spans at or above that duration. The table shows spans lasting approximately that duration. You can change the value by clicking it to reopen the dialog.
You can view all Streams created for your Organization from the Streams view.
Click Streams from the navigation bar.
Streams are listed in alphabetical order. Use the search bar to find Streams quickly.
By default, Streams are given the query parameters as the name. You can change that to be more descriptive by clicking the Rename button.
Deleting a Stream also deletes all historical data persisted for that Stream and can't be undone, so be sure you want to delete it!
You can embed a Stream into any app that supports adding an
<iframe> object. By default, the Stream uses the light theme currently displayed in LightStep, but you can embed a dark-themed version instead.
Use our Grafana plugin to view LightStep Streams in Grafana. Once in Grafana, clicking on a LightStep panel takes you back to LightStep where you can view sample traces.
To embed a Stream:
Open the IFrame URL dialog to copy the code for the Stream.
You can find the code to embed the Stream in two places:
From the Stream, at the top, click Embed.
From a dashboard, click the gear icon for the Stream and choose Get embed URL.
- If you want to use a dark version of the Stream, click Dark Theme in the dialog.
- Copy the code and paste into the third-party app as needed.
For example in Datadog, you can drag an IFrame widget onto a ScreenBoard and paste in the code from LightStep.