A standard method of identifying the root cause of a performance regression is to manually comb through traces and search for common system attributes associated with that regression. With Correlations, Lightstep helps you find attributes correlated with latency and errors automatically.

Lightstep Correlations search over all the span data in all traces that the current query participates in and finds patterns in that data for latency and error contribution, looking at service/operation pairs and tags. This analysis allows you to identify services, operations, and tags that are meaningfully associated with observed regressions. By surfacing statistical relationships between these system attributes, you can quickly generate and validate hypotheses during a root cause investigation.

View Correlations

Correlations are shown to the right of the Trace Analysis table and reflect the current data returned by the query. If you select a range in the histogram, correlations are redetermined based on that range. There are separate tabs for latency and error correlations.

The correlations are sorted in descending order based on the absolute value of the correlation score.

There are three types of correlations:

  • Service/operation pair\ (latency only): Shows the percentage of latency contribution to the critical path.
  • Tag in the trace: The tag appears on a span in the trace (but not currently in the results because it did not match the query).
  • Tag in the result: The tag appears on a span that is in the result set.

Correlation scores fall between -1 and +1 and are determined as follows:

  • +1: The attribute is only found in the selected latency range. So the closer to +1, the higher the chance that the attribute is causing latency in the currently selected range in the histogram.
  • -1: The attribute is never found in the selected latency range. So the closer to -1, the less chance that attribute has anything to do with latency.
  • 0: The attribute is equally likely to be found inside and outside of the selected latency range and thus has no statistical relationship to the latency.

Increase Correlation Accuracy

By default, the Trace Analysis table shows only spans that match the query. To get greater accuracy with correlations, you need a more holistic view to find a downstream issue. Click Show all Spans in Traces to include all spans that contribute to the trace.

Find Latency Using Correlations

Correlations allow you to quickly find attributes that are contributing to latency.

View Correlations in the Histogram to Determine Impact

Hover over a correlation to see the latency for that attribute overlayed in the histogram. For example, in this histogram, the operation wait-for-client-queue on the krakend-api-gateway service shows a high correlation for spans in the p95 and above range. It does not seem to have much of a correlation with lower latency.

Selecting the correlation allows you to easily refine the results in the Trace Analysis table to quickly reach hypothetical causes for latency.

Once you see attributes in traces that may correspond with latency or errors, you’ll likely want to filter the span data to further refine your hypothesis. For example, in this query, there are three attributes with high correlation scores for latency. Before you alert teams responsible for all three, you may want to see how they correlate with each other.

Filter the Trace Analysis Table

Click on a correlation and choose Filter by to show only results that match that criteria.

Once you filter the results, you can sort by duration to see the longest spans of correlated latency. Clicking a span takes you to the Trace view, where you can see the actual latency contribution.

The Latency Contribution is the contribution to the critical path latency. In this case, 6.01 seconds of the 6.4 second span is considered on the critical path, or 93.9% of the total critical path.

Group the Trace Analysis Table

To get a comparison of how different values for a service, operation, or tag might contribute to latency, you can group results. In this example, you notice that the tag customer_id with the value BEEMO shows high correlation, meaning that many spans with that tag correlate to latency.

Click Group by to group by the attribute in the Correlations panel.

Click on a group to see span data just for that group. Lightstep shows you the average latency, the error percentage, and the number of spans in that group.

Find the Cause of Errors Using Correlations

When you see that there are error correlations for your query, you can easily find out where the errors are coming from by filtering and/or grouping the spans in the Trace Analysis table to further refine your query.

For example, in this query on the android service, there are a number of tags with high error correlation scores. The http.status_code tag with a value of 400 shows the highest correlation.

Filter the Trace Analysis Table

You can filter the Trace Analysis table to see only the spans that are highly correlated with errors.

Click on a correlation and choose Filter by to show only results that match that criteria.

For example, because the http.status_code tag with a value of 400 showed the highest correlation, you might filter by that. Now you see only spans that have that tag value. To further narrow down the results, you can group them by another tag, operation, or service.

In the example, looking again at the Correlations panel, you see that the tag canary with a value of true is also highly correlated with errors. To see if all the filtered spans are tagged with canary: true, at the top of the table click Group by and choose canary.

The results only show the value of true, which means all the spans are tagged with canary: true. Had there been spans in the filtered table with the value false, you would see that group as well.

Click on the group to see spans that are tagged both with canary: true and http.status_code: 400 (remember, you filtered the table previously for that tag). The table shows you the tag group you are looking at (in this case canary: true), the average latency for these spans, the error percentage, and the number of spans in the group.

Now you can click on a span to view a full trace and find the error. In this example, it seems to come from the get-profile-for-user operation on the profile service. Sure enough, the logs on the trace show that an invalid profile ID is causing the error.

Group the Trace Analysis Table

You can also start your investigation by comparing the different values of a tag correlated with errors by clicking Group by for a tag in the Correlations panel.

In this example, maybe you immediately notice the canary tag with the value true is correlated with errors and decided to group by the value of the canary tag to see how many spans are coming from that environment.

In this case, you can see that 137 spans are from the Canary environment and they all have errors.

Click on a group to see span data just for that group. Lightstep shows you the average latency, the error percentage, and the number of spans in that group. .

Now you can click on a span to view a full trace and find the error. As before, it seems to come from the get-profile-for-user operation on the profile service due to an invalid profile ID.

Troubleshoot Correlations

Lightstep needs significant data to create correlations. You may see errors when there is not enough data to determine correlations.

No Correlations

If no correlations are returned, then there are no strong correlations found in the selected region.

This indicates that there are no system attributes that correlate strongly inside of the selected latency region (in comparison with the region outside the selection). To see strong enough correlations, try adjusting your query or histogram selection.

Less Than 50 Traces

Lightstep needs at least 50 traces to compile meaningful correlations. If your query results set or if the range you selected in the histogram has less than 50, you get a warning.

Adjust your query or increase the selected area in the histogram to include more traces.

Based on Low Data

The Correlations panel displays a warning when the amount of data from the query was not enough to determine true correlations. You have over 50 traces, so correlations can be compiled, but they may not provide a good signal.

Expand your selection to allow Lightstep to analyze a larger data set.