Firefighting with traditional solutions is hard. Scrolling through multitudes of dashboards only shows you there is a problem, not where, and not what caused it. Sifting through logs takes time, and it’s hard to find that needle in a haystack. Metrics and logs can only tell you so much - they let you know there’s an issue, but they can’t always tell you where and when. That’s where distributed tracing comes in.
Distributed tracing provides a view of the life of a request as it travels across multiple hosts and services communicating over various protocols. Here’s an example of a request from a client through a load balancer into several backend systems. With distributed tracing implemented, you have a window into performance at every step in the request.
Distributed tracing relies on instrumentation of the system you’re trying to observe. You can use specifications such as OpenTelemetry to provide a consistent interface across a variety of languages to write this instrumentation code. Some systems may require custom instrumentation at the service level, while others may only need instrumentation of the framework. Often, you’ll need to use a combination of these approaches.
Before you start that instrumentation, read on to learn about the different components that make up a distributed trace, and how the data from that instrumentation makes into Lightstep where you can view and work with it.
Spans and Traces
In distributed tracing, a trace is a view into a request as it moves through a distributed system. Multiple spans represent different parts of the workflow and are pieced together to create a trace. A span is a named, timed operation that represents a piece of the workflow.
In Lightstep, you view traces as a “tree” of spans that reflects the time that each span started and completed. It also shows you the relationship between spans. Here’s a simplified view of a trace, as it relates to the request above.
A trace starts with a root span where the request starts. This root span can have one or more child spans, and each one of those child spans can have child spans.
Child spans don’t always finish before their parent when the two are asynchronous. For example, an RPC call might time out, and so the parent span finishes before the “hanging” child span.
As you can see in the above illustration, there can be two types of child spans. A
ChildOf span is one where the parent depends on that child span’s result (like the relationship of the load balancer and the auth span). Spans doing concurrent (perhaps distributed) work may all individually be the
ChildOf a single parent span that merges the results for all children.
The second is the
FollowsFrom relationship, where the parent span is not dependent on the child (like the auth span and the billing span). These often represent “fire-and-forget” operations, for example, an opportunistic write to cache or a message that doesn’t care about its consumer.
In order for the trace tree to be built with these relationships intact, each span needs to propagate its context to its child. The context (or trace context) contains several pieces of information that can be passed between functions inside a process or between processes over an RPC. The context tells the child span who its parent is (parent SpanID) and what trace it belongs to (trace ID). The child span creates its own ID and then propagates both that ID (as the parent span ID) and the trace ID in the context to its child span. There can be other components in the context, but the parent span ID and trace ID are what allow a trace tree to be built.
OpenTelemetry uses headers to propagate context from span to span. Tracer objects are configured with Propagator objects that support transferring the context across process boundaries. Tracers provide a default tracer for your spans, and/or a Tracer provider capable of granting access to the tracer for your component. As spans are created and completed, the Tracer dispatches them to the OpenTelemetry SDK’s Exporter, which is responsible for sending your spans to a backend system for analysis. How tracers are created and registered depends on the language. Read the OpenTelemetry spec for more info .
A span may also have zero or more key/value attributes (known as tags in OpenTtracing). Attributes allow you to create metadata about the span. For example, you might create attributes that hold a customer ID, or information about the environment that the request is operating in, or an app’s release. Attributes don’t reflect any time-based event (events in OpenTelemtry and logs in OpenTracing handle events). The OpenTelemetry and OpenTracing specs define several standard attributes. You can also implement your own attributes.
Span events (logs in OpenTracing) contain time-stamped information. A span can have zero or more events. Each is a time-stamped event name, optionally accompanied by a structured data payload of arbitrary size.
You can add events to any span where the additional context would add value and the information included would be unique to an individual trace.
Sending Span Data to Lightstep with Tracers
Once you’ve done your instrumentation, you instantiate tracers that know how to create the spans and their associated attributes, events, and context. That instrumentation collects 100% of that data and send it to the Lightstep Satellites, who piece together the spans into traces. The Satellites then send any data that serves as examples of application errors, high latency, or other interesting events in real time to the Lightstep Hypothesis Engine. You use the Lightstep web application to view the actual traces, along with all the associated metadata from attributes and events. Read How Lightstep Works for more info.
Of course, that’s not all there is to distributed tracing or Lightstep! Here are more resources that can help you get started.
- OpenTelemetry and Lightstep: Read about using OpenTelemetry with Lightstep.
- OpenTelemetry: Learn more about OpenTelemetry
- OpenTracing: Learn more about how OpenTracing works to create spans and their data.
- Quick Starts: In available languages, auto-install OpenTracing to your framework. Configure your tracer and see requests travel through the framework. For other languages, use the API to create a single span by instantiating a tracer and adding instrumentation to a small piece of code and then view that span in Lightstep.
- Use tracing libraries other than Lightstep’s: If you’ve already instrumented your system, but were using another tracing library, no worries! Lightstep can ingest that instrumentation with almost no additional code needed.
- Understand Lightstep’s architecture: Read how Lightstep uses tracers, Satellites, and the Hypothesis engine to deliver visualized data to the Lightstep UI.
- Learn how to use Lightstep: Learn, once you instrument your system, how easy it is to analyze and resolve issues and get an understanding of the value Lightstep brings to your organization.