Now that we’ve narrowed down our hypothesis, it’s time to look more closely at a trace for verification.
We want to make sure we look at a trace from the meowsy customer, so let’s add that to the query.
Click back on the Trace Analysis table and in the Search bar, click the + icon, select tag: customer=”meowsy”, and click Run.
Now that we’ve narrowed down the spans in the Trace Analysis table to match our hypothesis, we can be assured that any span we look at from these results will return a representative trace showing the issue. No need to open a multitude of traces to find one with the issue.
Click the Duration heading twice to sort the table from longest to shortest duration.
Click on any of the spans in the Trace Analysis table. Lightstep builds the full distributed trace that contains that particular span.
The Trace View page shows you a flame graph at the top, and below, a breakdown of that graph by operation (each service in a different color), in the order of the request. Notice that when it gets to the
update-inventoryoperation, there’s a yellow bar. This designates the critical path of latency through this trace. It’s very easy to see which spans meaningfully block the request.
Detailed information about this span is shown in the panel on the right. Scroll down to the Additional Details area to see that this operation is taking 4.62s and is contributing to 96.9% of the latency (your numbers may vary, based on the span you chose).
And in looking at the Logs area, the issue seems to be with the network connection.
So now we know the issue is that the inventory service is having trouble connecting to the inventory database.
Yeah! We found the root cause! Now we need to share what we’ve found and then set up a monitor and alert to keep our eye on it.
What Did We Learn?
- The Trace view lets us see every span involved in a trace.
- It’s easy to verify the critical path of latency.
- Looking at the details, we can find the root cause.
Now we know who to notify about the issue. Let’s share our research with them.