This Quick Start shows you how to use OpenTelemetry in your Go app to:

  • Configure a tracer
  • Generate trace data
  • Propagate context in the application’s context
  • Propagate context over HTTP
  • Export the trace data to the console and to Lightstep
  • Use Lightstep to view the trace data

This Quick Start uses a simple Go app and then a client and server component to show how to propagate context over HTTP.

The full code for the examples in this guide can be found here.

Requirements

Go version 1.13 or newer

Setup

To use OpenTelemetry, you need to install the API and SDK packages. The version of the SDK and API used in this guide is v0.3.0, the most current version as of writing.

1
2
go mod init github.com/lightstep/examples
go get -u go.opentelemetry.io/otel

Collect Trace Data

You need to configure a TraceProvider to collect tracing information. A tracer is an object that tracks the currently active span and allows you to create (or activate) new spans. As spans are created and completed, the tracer dispatches them to an exporter that can send the spans to a backend system for analysis.

In this first example, the TraceProvider is configured using stdout.NewExporter, which prints tracing information to the console.

  1. Import OpenTelemetry and create a tracer configured to send data to the console, saving it as first_tracer.go.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    
    //first_tracer.go
    package main
    
    import (
      "context"
      "log"
    
      "go.opentelemetry.io/otel/api/core"
      "go.opentelemetry.io/otel/api/global"
      apitrace "go.opentelemetry.io/otel/api/trace"
      "go.opentelemetry.io/otel/exporters/trace/stdout"
      sdktrace "go.opentelemetry.io/otel/sdk/trace"
    )
    
    func initTracer() {
      exporter, err := stdout.NewExporter(stdout.Options{PrettyPrint: true})
      if err != nil {
         log.Fatal(err)
       }
       tp, err := sdktrace.NewProvider(sdktrace.WithConfig(sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
       sdktrace.WithSyncer(exporter))
       if err != nil {
         log.Fatal(err)
       }
       global.SetTraceProvider(tp)
    }
    func main() {
      initTracer()
      tracer := global.TraceProvider().Tracer("ex.com/basic")
    }
    
  2. Now create a Span object. A span is the building block of a trace and is a named, timed operation that represents a piece of the workflow in the distributed system. Multiple spans are pieced together to create a trace.

    The only required parameters are the span’s name and a reference to a context. But often, more information about the span is needed so that you can effectively debug and monitor in your backend system. Attributes allow you to add name/value pairs to describe the span. Events represent an event that occurred at a specific time within a span’s workload.

    Along with the name, add attributes for the platform and version and an event.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    
     func main() {
       initTracer()
       tracer := global.TraceProvider().Tracer("ex.com/basic")
    
       ctx, span := tracer.Start(context.Background(), "foo")
       span.SetAttributes(core.KeyValue{Key: "platform", Value: core.String("osx")})
       span.SetAttributes(core.KeyValue{Key: "version", Value: core.String("1.2.3")})
       span.AddEvent(ctx, "event in foo", core.KeyValue{Key: "name", Value: core.String("foo1")})
     }
    
  3. Add a child span to the existing span. To create the relationship between the new span and the previous one, Start is called with the ctx containing the parent span. Note that in this example, you also set attributes by passing a slice of KeyValue as an attributes argument to Start.
    1
    2
    3
    4
    5
    6
    
     attributes := []core.KeyValue{
       core.KeyValue{Key: "platform", Value: core.String("osx")},
       core.KeyValue{Key: "version", Value: core.String("1.2.3")},
     }
    
     ctx, child := tracer.Start(ctx, "baz", apitrace.WithAttributes(attributes...))
    
  4. Operations that are represented by spans are not considered complete, until the span is ended. You need to call End on the parent and child span objects in order to signal to the exporter that the tracing information is ready to be sent to its destination. In this example, the exporter will print the information to the console.
    1
    2
    
     child.End()
     span.End()
    
  5. Run the component (first_tracer.go) and view the data in your console. $ go run first_tracer.go

    The output on the console should show us something like this:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    
    {
         "SpanContext": {
                 "TraceID": "6f8d2544515621fdc119608d8f9b7d20",
                 "SpanID": "1f762a8402174dd1",
                 "TraceFlags": 1
         },
         "ParentSpanID": "2f0e08a71cc435ea",
         "SpanKind": 1,
         "Name": "baz",
         "StartTime": "2020-03-24T15:23:17.813856-07:00",
         "EndTime": "2020-03-24T15:23:17.81385922-07:00",
         "Attributes": [
                 {
                         "Key": "platform",
                         "Value": {
                                 "Type": "STRING",
                                 "Value": "osx"
                         }
                 },
                 {
                         "Key": "version",
                         "Value": {
                                 "Type": "STRING",
                                 "Value": "1.2.3"
                         }
                 }
         ],
         "MessageEvents": null,
         "Links": null,
         "StatusCode": 0,
         "StatusMessage": "",
         "HasRemoteParent": false,
         "DroppedAttributeCount": 0,
         "DroppedMessageEventCount": 0,
         "DroppedLinkCount": 0,
         "ChildSpanCount": 0,
         "Resource": null
    }
    {
         "SpanContext": {
                 "TraceID": "6f8d2544515621fdc119608d8f9b7d20",
                 "SpanID": "2f0e08a71cc435ea",
                 "TraceFlags": 1
         },
         "ParentSpanID": "0000000000000000",
         "SpanKind": 1,
         "Name": "foo",
         "StartTime": "2020-03-24T15:23:17.813847-07:00",
         "EndTime": "2020-03-24T15:23:17.814250944-07:00",
         "Attributes": [
                 {
                         "Key": "platform",
                         "Value": {
                                 "Type": "STRING",
                                 "Value": "osx"
                         }
                 },
                 {
                         "Key": "version",
                         "Value": {
                                 "Type": "STRING",
                                 "Value": "1.2.3"
                         }
                 }
         ],
         "MessageEvents": [
                 {
                         "Name": "event in foo",
                         "Attributes": [
                                 {
                                         "Key": "name",
                                         "Value": {
                                                 "Type": "STRING",
                                                 "Value": "foo1"
                                         }
                                 }
                         ],
                         "Time": "2020-03-24T15:23:17.813852-07:00"
                 }
         ],
         "Links": null,
         "StatusCode": 0,
         "StatusMessage": "",
         "HasRemoteParent": false,
         "DroppedAttributeCount": 0,
         "DroppedMessageEventCount": 0,
         "DroppedLinkCount": 0,
         "ChildSpanCount": 1,
         "Resource": null
    }
    

Propagating the Context

Creating relationships between spans is what allows the propagation of the context of a trace across different parts of a process or across processes. Another way to create a hierarchical relationship between spans is using WithSpan, which supports recovering from panics automatically, so that spans still exported in these situations. The following example sets up the tracer and creates three spans, with foo being the parent of bar and bar being the parent of baz. They are all part of a single trace.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
_ = tracer.WithSpan(ctx, "foo",
        func(ctx context.Context) error {
            tracer.WithSpan(ctx, "bar",
                func(ctx context.Context) error {
                    tracer.WithSpan(ctx, "baz",
                        func(ctx context.Context) error {
                            return nil
                        },
                    )
                    return nil
                },
            )
            return nil
        },
    )

HTTP propagation

You can instrument a request from a client to a server which allows you to propagate the context across process boundaries. This ensures that you can correlate events that happen in separate processes into a single trace.

  1. Instrument the server (server.go). Select the type of context carrier—in this example you use W3C.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    
     // server.go
     package main
    
     import (
          "io"
           "log"
           "net/http"
    
           "go.opentelemetry.io/otel/api/correlation"
           "go.opentelemetry.io/otel/api/global"
           "go.opentelemetry.io/otel/api/trace"
           "go.opentelemetry.io/otel/exporters/trace/stdout"
           "go.opentelemetry.io/otel/plugin/httptrace"
           sdktrace "go.opentelemetry.io/otel/sdk/trace"
     )
    
     func initTracer() {
           // Create stdout exporter to be able to retrieve
           // the collected spans.
           exporter, err := stdout.NewExporter(stdout.Options{PrettyPrint: true})
           if err != nil {
                 log.Fatal(err)
           }
    
           // For the demonstration, use sdktrace.AlwaysSample sampler to sample all traces.
           // In a production application, use sdktrace.ProbabilitySampler with a desired probability.
           tp, err := sdktrace.NewProvider(sdktrace.WithConfig(sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
                 sdktrace.WithSyncer(exporter))
           if err != nil {
                 log.Fatal(err)
           }
            global.SetTraceProvider(tp)
     }
    
     func helloHandler(w http.ResponseWriter, req *http.Request) {
          tracer := global.TraceProvider().Tracer("example/server")
    
           // Extracts the conventional HTTP span attributes,
           // distributed context tags, and a span context for
           // tracing this request.
           attrs, entries, spanCtx := httptrace.Extract(req.Context(), req)
           ctx := req.Context()
           if spanCtx.IsValid() {
                 ctx = trace.ContextWithRemoteSpanContext(ctx, spanCtx)
           }
    
           // Apply the correlation context tags to the request
           // context.
           req = req.WithContext(correlation.ContextWithMap(ctx, correlation.NewMap(correlation.MapUpdate{
                 MultiKV: entries,
           })))
    
           // Start the server-side span, passing the remote
           // child span context explicitly.
           _, span := tracer.Start(
                 req.Context(),
                 "hello",
                 trace.WithAttributes(attrs...),
           )
           defer span.End()
    
           io.WriteString(w, "Hello, world!\n")
     }
    
     func main() {
          initTracer()
    
          http.HandleFunc("/hello", helloHandler)
          err := http.ListenAndServe(":7777", nil)
          if err != nil {
                panic(err)
          }
      }
    
  2. Have the client code (client.go) call httptrace.W3C, which configures the http.Request object to use W3C trace context headers. Also add a call to httptrace.Inject, which both injects the headers and applies conventional HTTP attributes to the span.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
  // client.go
  package main

  import (
	       "context"
	       "io/ioutil"
	       "log"

	       "net/http"
	       "time"

	       "google.golang.org/grpc/codes"

	       "go.opentelemetry.io/otel/api/correlation"
	       "go.opentelemetry.io/otel/api/global"
	       "go.opentelemetry.io/otel/api/key"
	       "go.opentelemetry.io/otel/api/trace"
	       "go.opentelemetry.io/otel/exporters/trace/stdout"
	       "go.opentelemetry.io/otel/plugin/httptrace"
	       sdktrace "go.opentelemetry.io/otel/sdk/trace"
  )

  func initTracer() {
	   exporter, err := stdout.NewExporter(stdout.Options{PrettyPrint: true})
	   if err != nil {
		     log.Fatal(err)
	   }

	   tp, err := sdktrace.NewProvider(sdktrace.WithConfig(sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
		      sdktrace.WithSyncer(exporter))
	   if err != nil {
		      log.Fatal(err)
	   }
	   global.SetTraceProvider(tp)
   }

   func main() {
	    initTracer()

	     client := http.DefaultClient
	     ctx := correlation.NewContext(context.Background(),
		      key.String("username", "donuts"),
	     )

	     var body []byte

	     tracer := global.TraceProvider().Tracer("example/client")
	     err := tracer.WithSpan(ctx, "client hello",
		       func(ctx context.Context) error {
			          req, _ := http.NewRequest("GET", "http://localhost:7777/hello", nil)

			          ctx, req = httptrace.W3C(ctx, req)
			          httptrace.Inject(ctx, req)
			          res, err := client.Do(req)
			          if err != nil {
				              panic(err)
			          }
			          body, err = ioutil.ReadAll(res.Body)
			          _ = res.Body.Close()
			          trace.SpanFromContext(ctx).SetStatus(codes.OK, "OK")
			          return err
		       })

	     if err != nil {
		       panic(err)
	     }
	     time.Sleep(time.Second * 5)
  }

  1. Launch the server in a terminal.
    go run ./server.go

  2. Open a second terminal to run the client.
    go run client.go

The output from running those scripts looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
$ go run server.go
{
        "SpanContext": {
                "TraceID": "778d20556018af6bed2cb307e61a1408",
                "SpanID": "1651d883c585104e",
                "TraceFlags": 1
        },
        "ParentSpanID": "31f227bd11360e12",
        "SpanKind": 1,
        "Name": "hello",
        "StartTime": "2020-03-24T15:32:01.940785-07:00",
        "EndTime": "2020-03-24T15:32:01.940801044-07:00",
        "Attributes": [
                {
                        "Key": "http.url",
                        "Value": {
                                "Type": "STRING",
                                "Value": "/hello"
                        }
                }
        ],
        "MessageEvents": null,
        "Links": null,
        "StatusCode": 0,
        "StatusMessage": "",
        "HasRemoteParent": true,
        "DroppedAttributeCount": 0,
        "DroppedMessageEventCount": 0,
        "DroppedLinkCount": 0,
        "ChildSpanCount": 0,
        "Resource": null
}

$ go run client.go
{
        "SpanContext": {
                "TraceID": "778d20556018af6bed2cb307e61a1408",
                "SpanID": "222c35d347f10790",
                "TraceFlags": 1
        },
        "ParentSpanID": "31f227bd11360e12",
        "SpanKind": 3,
        "Name": "http.dns",
        "StartTime": "2020-03-24T15:32:01.903995-07:00",
        "EndTime": "2020-03-24T15:32:01.939465601-07:00",
        "Attributes": [
                {
                        "Key": "http.host",
                        "Value": {
                                "Type": "STRING",
                                "Value": "localhost"
                        }
                }
        ],
        "MessageEvents": null,
        "Links": null,
        "StatusCode": 0,
        "StatusMessage": "",
        "HasRemoteParent": false,
        "DroppedAttributeCount": 0,
        "DroppedMessageEventCount": 0,
        "DroppedLinkCount": 0,
        "ChildSpanCount": 0,
        "Resource": null
}
{
        "SpanContext": {
                "TraceID": "778d20556018af6bed2cb307e61a1408",
                "SpanID": "e5d13d02162e1fcf",
                "TraceFlags": 1
        },
        "ParentSpanID": "31f227bd11360e12",
        "SpanKind": 3,
        "Name": "http.connect",
        "StartTime": "2020-03-24T15:32:01.939952-07:00",
        "EndTime": "2020-03-24T15:32:01.940249351-07:00",
        "Attributes": [
                {
                        "Key": "http.remote",
                        "Value": {
                                "Type": "STRING",
                                "Value": "[::1]:7777"
                        }
                }
        ],
        "MessageEvents": null,
        "Links": null,
        "StatusCode": 0,
        "StatusMessage": "",
        "HasRemoteParent": false,
        "DroppedAttributeCount": 0,
        "DroppedMessageEventCount": 0,
        "DroppedLinkCount": 0,
        "ChildSpanCount": 0,
        "Resource": null
}
{
        "SpanContext": {
                "TraceID": "778d20556018af6bed2cb307e61a1408",
                "SpanID": "4ea4705eb9870639",
                "TraceFlags": 1
        },
        "ParentSpanID": "31f227bd11360e12",
        "SpanKind": 3,
        "Name": "http.getconn",
        "StartTime": "2020-03-24T15:32:01.903903-07:00",
        "EndTime": "2020-03-24T15:32:01.940384053-07:00",
        "Attributes": [
                {
                        "Key": "http.host",
                        "Value": {
                                "Type": "STRING",
                                "Value": "localhost:7777"
                        }
                },
                {
                        "Key": "http.remote",
                        "Value": {
                                "Type": "STRING",
                                "Value": "[::1]:7777"
                        }
                },
                {
                        "Key": "http.local",
                        "Value": {
                                "Type": "STRING",
                                "Value": "[::1]:62512"
                        }
                }
        ],
        "MessageEvents": null,
        "Links": null,
        "StatusCode": 0,
        "StatusMessage": "",
        "HasRemoteParent": false,
        "DroppedAttributeCount": 0,
        "DroppedMessageEventCount": 0,
        "DroppedLinkCount": 0,
        "ChildSpanCount": 0,
        "Resource": null
}
{
        "SpanContext": {
                "TraceID": "778d20556018af6bed2cb307e61a1408",
                "SpanID": "778bddf383d90840",
                "TraceFlags": 1
        },
        "ParentSpanID": "31f227bd11360e12",
        "SpanKind": 3,
        "Name": "http.send",
        "StartTime": "2020-03-24T15:32:01.940517-07:00",
        "EndTime": "2020-03-24T15:32:01.940520523-07:00",
        "Attributes": null,
        "MessageEvents": null,
        "Links": null,
        "StatusCode": 0,
        "StatusMessage": "",
        "HasRemoteParent": false,
        "DroppedAttributeCount": 0,
        "DroppedMessageEventCount": 0,
        "DroppedLinkCount": 0,
        "ChildSpanCount": 0,
        "Resource": null
}
{
        "SpanContext": {
                "TraceID": "778d20556018af6bed2cb307e61a1408",
                "SpanID": "7f5e127cb1b213fa",
                "TraceFlags": 1
        },
        "ParentSpanID": "31f227bd11360e12",
        "SpanKind": 3,
        "Name": "http.receive",
        "StartTime": "2020-03-24T15:32:01.941275-07:00",
        "EndTime": "2020-03-24T15:32:01.941325854-07:00",
        "Attributes": null,
        "MessageEvents": null,
        "Links": null,
        "StatusCode": 0,
        "StatusMessage": "",
        "HasRemoteParent": false,
        "DroppedAttributeCount": 0,
        "DroppedMessageEventCount": 0,
        "DroppedLinkCount": 0,
        "ChildSpanCount": 0,
        "Resource": null
}
{
        "SpanContext": {
                "TraceID": "778d20556018af6bed2cb307e61a1408",
                "SpanID": "31f227bd11360e12",
                "TraceFlags": 1
        },
        "ParentSpanID": "0000000000000000",
        "SpanKind": 1,
        "Name": "client hello",
        "StartTime": "2020-03-24T15:32:01.903687-07:00",
        "EndTime": "2020-03-24T15:32:01.941379893-07:00",
        "Attributes": null,
        "MessageEvents": null,
        "Links": null,
        "StatusCode": 0,
        "StatusMessage": "OK",
        "HasRemoteParent": false,
        "DroppedAttributeCount": 0,
        "DroppedMessageEventCount": 0,
        "DroppedLinkCount": 0,
        "ChildSpanCount": 6,
        "Resource": null
}

Looking at the output, you can see that the same "TraceID": "778d20556018af6bed2cb307e61a1408" is propagated across the spans from the client and the server.

Export and Explore Trace Data in Lightstep

To make use of the trace data being generated, exporters send the data to various supported collectors. The stdout.Exporter is really useful when debugging while instrumenting. But now, you’ll use the experiemental lightstep.Exportor to send tracing data to Lightstep’s public Satellites.

If you don’t already have a Lightstep account, you can create a free account here.

  1. Install the exporter library. go get github.com/lightstep/opentelemetry-exporter-go

  2. Update the exporter configuration in client.go and server.go to use the lightstep.Exporter. You’ll need to provide a name for your service and a

1
2
3
4
5
6
7
8
9
10
// NOTE: replace the import of:
//     "go.opentelemetry.io/otel/exporter/trace/stdout"
// with:
       "github.com/lightstep/opentelemetry-exporter-go/lightstep"

func initTracer() {
    exporter, err := lightstep.NewExporter([]lightstep.Option{
        lightstep.WithServiceName("go-guide-server"),
        lightstep.WithAccessToken("<YOUR ACCESS TOKEN HERE"),
    }...)
  1. Restart server.go and run client.go again. Tracing information is now available in your Lightstep project!

  2. Click Explorer in the left navigation bar to see the trace data.

  3. Click on any span in the Trace Analysis table to view the full trace.