Lightstep Observability’s Change Intelligence correlates changes found in traces with deviations in your data. But there may be times when you need help getting that correlation to work as expected. Here are some issues you may run into when running Change Intelligence and how to solve them.
Troubleshoot metric data
Following are issues that may affect running Change Intelligence on metric data.
Service name not found on metric
For Change Intelligence to be able to correlate changes in service performance with metrics deviations, it needs to know the service that emitted the metric. By default, Change Intelligence looks for the attribute
service.name in your metrics. If you use a different (or additional) attributes, you need to register those with Lightstep Observability.
Key operations not found
Key Operations are operations from your tracing data whose performance is considered strategic to the health of your system. By default, Lightstep Observability considers all ingress operations on a service to be your Key Operations. If Change Intelligence can’t find Key Operations, it may be because:
The time windows configured for your metric baseline and/or deviation are too small and no spans were reported from an ingress operation. Try changing the size of those windows.
No significant correlations found
Change Intelligence uses span attributes in your traces to find correlations in service performance and metric health. If no significant correlations can be found, it may be because:
The time windows configured for your metric baseline and/or deviation are too small and don’t contain enough trace data to make the correlations. Try changing the size of those windows.
Your tracing instrumentation doesn’t contain enough attributes to make correlations. Try adding attributes that reflect important areas of your system and business model. You can also check your Instrumentation Quality Score for custom, environment, and hostname attributes for best practices and tips.
Change Intelligence couldn’t find any performance changes in your system at the time of the metric deviation.
Trace data not found
Change Intelligence looks at the service that emitted the metric and finds traces that the service participated in. If no traces are found, it may be because:
Your metric query is further in the past than the trace data is kept for, so Change Intelligence can’t find corresponding trace data. Adjust the time period to be more recent.
Troubleshoot metric and span data
Following are issues that may affect running Change Intelligence on metric or span data.
No significant changes found in related traces
Change Intelligence looks for performance issues in the trace data that ocurred at the same time as the metric deviation. If no significant performance changes are found, it may be because:
The service that emitted the metric and its dependencies didn’t experience any performance changes in the same time period as your metric deviation. Try looking at other Key Operations in Change Intelligence for attributes that were found in traces with performance issues.
Not enough traces found
Change Intelligence makes sure there are enough traces for each correlated attribute to support its statistical analysis. If there aren’t enough traces, it may be because:
- The time windows configured for your metric baseline and deviation are too small and don’t contain enough trace data to make the correlations. Try changing the size of those windows.