When your instrumentation includes attributes (key/value pairs that carry metadata descriptions about your spans), Lightstep can use them to help quickly pinpoint root causes of issues. You can see when spans with certain attribute values have errors or latency, while others with different values don’t.

In OpenTracing, attributes are known as tags and Lightstep supports those too.

More about Attributes

A span can have zero or more key/value attributes (called tags in OpenTracing). Attributes allow you to create metadata about the span. For example, you might create attributes that hold a customer ID, or information about the environment that the request is operating in, or an app’s release. Attributes do not reflect any time-based events (log events handle events). The OpenTelemetry spec defines several standard attributes.

You can also implement your own attributes and events. The following are attributes and log events that work very well with Lightstep in addition to the OpenTelemetry semantic conventions. Any attribute that you add to your span data will enable more segmentation, making it easier to find, filter, and group your span data in Lightstep. Lightstep doesn’t have cardinality limitations, so the more attributes you use, the greater your insights will be.

Learn more about attributes.

Expandable end

Use the OpenTelemetry Semantic Attributes

Start with the OpenTelemetry semantic span attributes. These are standardized attributes that provide much of the functionality you’ll need.

Attributes help identify issues that are affecting business flows and assets that are important to you. For example, if your business’s number one concern is customers, then create an attribute to carry the customer’s name so you can quickly see when issues hit your highest priority customers.

Similarly, attributes that describe the device type or OS can help determine how widespread an issue is. Is it only a problem for customers on a particular combination of OS and device? Is that a large part of your customer base? Being able to make that determination quickly can help prioritize mitigation.

Catch System Changes

Knowing immediately when a deploy affects your system, or when an a/b segment of your environment is affected means you can react quickly by starting a rollback or moving customers to a different segment.

When you use the service.version attribute, LightStep automatically creates markers showing when deployments occurred. Here’s an example of increased latency, likely caused by the deployment immediately before it.

In this graphic, you can see that the attribute service.version with a value of 10.8.585 has many errors during the regression time range.Attributes correlated with errors

When you filter on the service.version:10.8.585 attribute to see only spans with that attribute value, and then in the Trace Analysis table, group by the service attribute, you can see which services were deployed with that version.

In this example, only spans on the store-server service have the version value of 10.8.585, which tells you that a deploy of that service is causing the errors. And you can look at the event log messages to see what those errors are. Event logs, attributes, and filters pinpoint the cause

Because of the service.version and service attributes, you were able to quickly pinpoint the source of errors!

Label Infrastructure

Knowing where an issue is occurring, and just as important, ruling out where it’s not, can save a tremendous amount of time. In this example, you can see that the attribute db.type=cassandra is on almost every span at the high-end of latency.

When you group the results by the db.type attribute, you can see that the Cassandra database is experiencing much higher latency than others.

What Did We Learn?

  • Attributes are valuable when looking for root causes. Lightstep can correlate attributes with error and latency rate, and you can group and filter by attributes to narrow down the data and find issues fast.

  • Using the service.version attribute allows Lightstep to show you when deploys occur, letting you associate a deploy with a regression at a glance.

  • You should create attributes related to valued business functionality so you can easily monitor flows and assets important to you.