You use the Trace view to see a trace from beginning to end of a request. The Trace view shows you a flame graph of the full trace and below that, each span is shown in a hierarchy, allowing you to see the parent-child relationship of all the spans in the trace (spans with errors are shown in red).
Clicking a span shows details in the right panels, based on the span’s metadata. Traces can be bookmarked, shared, and reviewed by your team.
The data is retained for the length of your retention window.
When viewing a trace from a notebook or dashboard, Lightstep displays all spans from the current retention window and builds the trace as the data comes in (that’s why you’ll often see Trace Assembling with a spinning icon as the data loads). Your view into the trace is compiled from real-time data, giving you an accurate view into a request. Traces built from historical data (such as Snapshots or links) load more quickly.
When traces contain over 4,000 spans the graph is partially loaded. Click on the More button load the child spans. Only the first 4,000 spans for the selected parent are loaded, which may mean there are unseen spans (the largest generation of parent to child spans is 4,000).
If you’re viewing a trace from outside a notebook or dashboard, or from data older than the retention window, the total number of spans displayed is capped at 4,000.
Access Trace view
You access a Trace view by clicking on a span from several places within Lightstep Observability:
- Span charts in notebooks, dashboards and alerts: With the Show span samples toggle on, click a dot or error triangle to view the trace.
- Change Intelligence: Click View sample traces below the SLI tables.
- Trace Analysis tab in Explorer: Click a span in the table to open. The span is selected in the trace.
- Service diagram: With the Show span samples toggle on, click on a span in the Span Samples table below the diagram.
- Developer Mode: Click View Trace to see traces sent by the Developer Satellite.
- Alerts: Alerts include a link to a trace that is an exemplar of the current issue.
View Trace graph
The Trace view is a flame graph that displays the overall trace. Spans participating in the trace are displayed in different colors to give a sense of the overall service interactions. You can click on a colored bar to select the corresponding span and see its details.
The graph shows where each span participates in the trace by presenting a hierarchical representation of spans in parent/child relationships, in the order that they participated in the request. The number to the left of the span tells you how many children that span has. You can collapse all or part of the hierarchy, focus on just one span and its hierarchy, and filter to see only specific spans.
Click on a span to view details provided by span metadata from your instrumentation (attributes and events), any Workflow Links you’ve created, and other details, such as Span and Trace IDs, latency contribution, and language of the reporting service.
Spans that are linked are shown in the Details panel.
See the critical path
Lightstep Observability automatically runs a heuristic to determine and display the critical path of a given trace, using a black line. During a request, a parent span might be waiting for a child span (or even a child of the child) to complete before it can finish. The critical path is the time when an operation in a span is actively doing something.
Consider this trace of a request. Each horizontal bar represents the full duration of an operation’s span. You can see that the spans towards the top take much longer, mostly because they’re waiting for the spans further down the stack to complete. Lightstep shows you the critical path through the trace, allowing you to see how long that operation was actually active during the request and to help identify bottlenecks in the overall transactions.
When errors are found on a span, Lightstep displays the span in red. The Details panel shows that there’s an error, and log of events should help determine the nature of the error.
See missing spans
Spans highlighted with a red dashed box are missing spans; their parent span wasn’t found in the data from the Microsatellites.
Missing spans can happen when either the Microsatellite dropped the span (for example, your Microsatellite pool is not auto-scaling to keep up with traffic) or the tracer dropped the span (for example, your application crashed or never called
span.flush() ). When you see missing spans, check the Reporting Status page to find the culprit.
Modify display of spans
You can change the graph’s display to zero in on just what’s important.
Collapse the hierarchy
You can collapse the hierarchy to the level of the selected span. Select a span and click the Collapse/Expand button at the top of the graph. Parent spans with collapsed children display the number of hidden child spans.
To expand the hierarchy to reshow all spans, re-click the Collapse/Expand button.
Focus on a single span’s hierarchy
You can show just the spans in a sub-hierarchy to get a closer view. Select the span to focus on and click Focus on selected span. The graph filters to show only that hierarchy.
Click Reset Focus to show the full graph.
Filter the graph
You can show only specific operations or services. Enter a name into the search bar and Lightstep Observability filters the table.
View details of a span
The panel to the right of the graph shows details for the selected span.
Service and operation
The top of the panel shows the service and operation name. Click the name link to run a query in a notebook for that operation or service.
Use the Share button to share the trace with someone. You can share the URL for the trace and the recipient is taken to that exact trace in Lightstep. When you integrate Lightstep Observability with Slack, you can share previews of the Trace View page in any channel of your workspace. Other Slack members can see information about the selected span, its parent span, critical path information, and errors. Lightstep users can click View Trace to jump right to that page.
Attributes and events
The Attributes & Events panel shows the attributes and their values on the selected span, and any logs of event data as reported by your instrumentation.
Click the attribute value to run a query in a notebook for that value.
The Workflow Links panel shows any links you’ve created that apply to the selected span. Workflow Links allow you to create dynamic links to information outside of this page, allowing access to all the info you need when you need it. For example, say you want to view your playbook when the span includes a certain error code. Lightstep can construct these customized links automatically, using attributes and other metadata from a span in the trace.
The Details panel shows details captured from your instrumentation, such as span timestamps, latency, IDs, and language. For certain details, like the client library, you can click a link to visit that version’s GitHub page.