Create links to important information

When investigating an incident or debugging a performance problem, rarely do you use just one tool or need information from just one source. You may need to jump between different tools, synthesize information from multiple sources, and notify other people quickly. Cloud Observability lets you add flexible Workflow Links on the Trace View page that link to other resources, allowing access to all the info you need when you need it.

For example, say you want to jump directly to logs from the time when an issue occurred or to quickly view your playbook’s instructions when the span includes a certain error code. Cloud Observability can construct these customized links automatically, using attributes and other metadata from a span in the trace.

You can create Workflow Links to anything, but the real power comes from using parameters directly from the span to substitute attribute values, timestamps, and other metadata.

About Workflow Links

Every Workflow Link consists of a name, a URL, and a set of rules to determine when to display the link.

Name

Every link must have a name - this will be used as the link anchor text. Names can include variables, allowing the link name to change dynamically based on the span being viewed. For example, say you want to create a link to contact the owner of a service. You might configure the link name as Email {$service} owner, and at runtime, {$service} will be replaced by the service name from the current span. You can find out more about using variables here.

Names don’t have to be unique. For example, in the email link above, different services likely have different owners. You can create multiple links, each named “Email Service Owner,” which point to different email addresses.

URL

Every link must point to a valid, resolvable URL. This can be a static URL, or it can include variables that will be replaced at runtime.

For example, you might create a link to the homepage of your Git repo so someone debugging an issue can jump right in. You’d simply create a static URL like https://github.com/acme/acme_code.

But say instead, you want to quickly find the code that generated the span. You can use {$operation} in the URL to perform a search for that operation: https://github.com/acme/acme_code/search?q={$operation}.

Rules

Finally, you need to decide which spans will show the link. You define this as a set of rules written as a JSON object with name/value pairs.

Even if you don’t want to include any variables in the link (for example, a static email address), you still need to create a rule.

Like names and URLs, rules also use variables. In the GitHub example above, you could define a rule using the $operation variable. If you want the link to display for all operations, you use:

1
2
3
{
  "$operation": null
}

For rule definitions, a value of null means “any attribute value,” or “always display this link if the attribute is present on the span.”

If you want the link to display for one specific operation only, you would define that operation as the single value in an array. The values in the array are interpreted as an “allow” list.

1
2
3
{
  "$operation": ["HTTP POST"]
}

In this example, the link only displays for spans with the HTTP POST operation.

If you want to show the link for several different attribute values, you define those values as an array:

1
2
3
{
  "$operation": ["HTTP POST", "api-explore/get"]
}

In this example, the link displays for any span with the HTTP POST or the api-explore/get operations.

Negative rules are also supported. To display a link only when a certain attribute is NOT present on the span, define the rule as follows.

1
2
3
{
  "!error": null,
}

In this example, the link displays only if the span does not contain an attribute called error.

To display a link only when a span does not have an attribute with any of a set of values, define the rule(s) as follows. When multiple rules are listed, they will be evaluated with a logical AND operator.

1
2
3
4
{
  "!region": ["us-west1", "us-west2"],
  "!$service": ["my-unhealthy-service"]
}

In this example, the link displays only when the span does not contain an attribute named region with the value us-west1 or us-west2, AND the service name is not my-unhealthy-service.

If you want a link to always display, no matter what, set the rule to {"$service": null}, as every span will have a service.

Putting it All Together

Here’s an example of a Workflow Link definition that, when displayed, emails Bob, who is the owner of the ios-client and react-client services.

Make sure to create rules that include all the conditions when the link should display. For example, in the above Workflow Link, if Bob becomes the owner of another service, the link won’t display until you edit the rule to include it.

You don’t need to have a rule for every variable that you use in the URL, but do ensure that there will always be a value available. For example, say you have this definition that displays a link to logs for a specific customer:

These rules say to apply the link on any span that has an attribute called environment with either the value dev or the value staging. However, the URL itself also includes a reference to an attribute called customer_id.

If the span doesn’t contain an attribute called customer_id, the URL will end up like this: https://my.logs.com/non-prod?service=some_service&env=some_environment&id=undefined

The link will be displayed, but the value of customer_id will appear as “undefined”. For this reason, we recommend including a rule entry for every variable used in the Name or URL fields.

Need some ideas?

Cloud Observability provides a number of templates in the product to get you started. These templates are taken from the Workflow Links Cookbook. You can find more examples there.

Lifecycle of a Workflow Link

Cloud Observability re-renders the current set of Workflow Link definitions each time you view a trace. They are not persisted with the trace itself. When links are added or modified, the current set of link definitions will appear on all applicable spans at the time a trace is viewed, including historical traces. If you delete a link, it will no longer be shown on any trace, including traces that you have looked at in the past.

You can also use our Public API to create Workflow Links.

See also

Create and manage Workflow Links

Workflow Links cookbook

Use variables in Workflow Links

Updated Dec 1, 2019