Troubleshoot correlations

Cloud Observability’s correlation feature relates changes found in traces with deviations in your data. But there may be times when you need help getting that correlation to work as expected. Here are some issues you may run into when running the correlation feature and how to solve them.

Troubleshoot metric data

The following sections show issues that may affect running the correlation feature on metric data.

Service name not found on metric

For Cloud Observability to be able to correlate changes in service performance with metrics deviations, it needs to know the service that emitted the metric. By default, the correlation feature looks for the attribute service.name in your metrics. If you use different or additional attributes, you need to register those with Cloud Observability.

Key operations not found

Key Operations are operations from your tracing data whose performance is considered strategic to the health of your system. By default, Cloud Observability considers all ingress operations on a service to be your Key Operations. If Cloud Observability can’t find Key Operations, it may be because:

The time windows configured for your metric baseline and/or deviation are too small and no spans were reported from an ingress operation. Try changing the size of those windows.
Your tracing instrumentation is not complete on your ingress operations. Check your Instrumentation Quality Score for your Service boundary spans.

No significant correlations found

The correlation feature uses span attributes in your traces to find correlations in service performance and metric health. If no significant correlations can be found, it may be because:

The time windows configured for your metric baseline and/or deviation are too small and don’t contain enough trace data to make the correlations. Try changing the size of those windows.
Your tracing instrumentation doesn’t contain enough attributes to make correlations. Try adding attributes that reflect important areas of your system and business model. You can also check your Instrumentation Quality Score for custom, environment, and hostname attributes for general guidelines and tips.
The correlation feature couldn’t find any performance changes in your system at the time of the metric deviation.

Trace data not found

The correlation feature looks at the service that emitted the metric and finds traces that the service participated in. If no traces are found, it may be because:

Your metric query is further in the past than the trace data is kept for, so the correlation feature can’t find corresponding trace data. Adjust the time period to be more recent.
The service that emitted the metric and its dependencies aren’t instrumented. Check your Instrumentation Quality Score for more information.

Troubleshoot metric and span data

The following sections show issues that may affect running the correlation feature on metric or span data.

The correlation feature looks for performance issues in the trace data that occurred at the same time as the metric deviation. If no significant performance changes are found, it may be because:

The service that emitted the metric and its dependencies aren’t instrumented. Check your Instrumentation Quality Score for more information.
The service that emitted the metric and its dependencies didn’t experience any performance changes in the same time period as your metric deviation. Try looking at other Key Operations for attributes that were found in traces with performance issues.

Not enough traces found

The correlation feature makes sure there are enough traces for each correlated attribute to support its statistical analysis. If there aren’t enough traces, it may be because:

The time windows configured for your metric baseline and deviation are too small and don’t contain enough trace data to make the correlations. Try changing the size of those windows.