When your instrumentation includes attributes (key/value pairs that carry metadata descriptions about your spans), Lightstep can use them to help quickly pinpoint root causes of issues. You can see when spans with certain attribute values have errors or latency, while others with different values don’t.
In OpenTracing, attributes are known as tags and Lightstep supports those too.
More about Attributes
A span can have zero or more key/value attributes (called tags in OpenTracing). Attributes allow you to create metadata about the span. For example, you might create attributes that hold a customer ID, or information about the environment that the request is operating in, or an app’s release. Attributes do not reflect any time-based events (log events handle events). The OpenTelemetry spec defines several standard attributes.
You can also implement your own attributes and events. The following are attributes and log events that work very well with Lightstep in addition to the OpenTelemetry semantic conventions. Any attribute that you add to your span data will enable more segmentation, making it easier to find, filter, and group your span data in Lightstep. Lightstep doesn’t have cardinality limitations, so the more attributes you use, the greater your insights will be.
Learn more about attributes.
Use the OpenTelemetry Semantic Attributes
Start with the OpenTelemetry semantic span attributes. These are standardized attributes that provide much of the functionality you’ll need.
Use Attributes to Identify Issues Related to Your Business Values
Attributes help identify issues that are affecting business flows and assets that are important to you. For example, if your business’s number one concern is customers, then create an attribute to carry the customer’s name so you can quickly see when issues hit your highest priority customers.
Similarly, attributes that describe the device type or OS can help determine how widespread an issue is. Is it only a problem for customers on a particular combination of OS and device? Is that a large part of your customer base? Being able to make that determination quickly can help prioritize mitigation.
Catch System Changes
Knowing immediately when a deploy affects your system, or when an a/b segment of your environment is affected means you can react quickly by starting a rollback or moving customers to a different segment.
When you use the
service.version attribute, LightStep automatically creates markers showing when deployments occurred. Here’s an example of increased latency, likely caused by the deployment immediately before it.
In this graphic, you can see that the attribute
service.version with a value of
10.8.585 has many errors during the regression time range.
When you filter on the
service.version:10.8.585 attribute to see only spans with that attribute value, and then in the Trace Analysis table, group by the
service attribute, you can see which services were deployed with that version.
In this example, only spans on the
store-server service have the version value of
10.8.585, which tells you that a deploy of that service is causing the errors. And you can look at the event log messages to see what those errors are.
Because of the
service attributes, you were able to quickly pinpoint the source of errors!
Knowing where an issue is occurring, and just as important, ruling out where it’s not, can save a tremendous amount of time. In this example, you can see that the attribute
db.type=cassandra is on almost every span at the high-end of latency.
When you group the results by the
db.type attribute, you can see that the Cassandra database is experiencing much higher latency than others.
What Did We Learn?
Attributes are valuable when looking for root causes. Lightstep can correlate attributes with error and latency rate, and you can group and filter by attributes to narrow down the data and find issues fast.
service.versionattribute allows Lightstep to show you when deploys occur, letting you associate a deploy with a regression at a glance.
You should create attributes related to valued business functionality so you can easily monitor flows and assets important to you.