Send traces to Lightstep with Go

This Quick Start guide shows you how to configure your Go applications to send OpenTelemetry traces to Cloud Observability.

This guide does not provide documentation on application instrumentation. For Go-specific information on instrumentation, please see the Go OpenTelemetry getting started guide.

The sections below contain code snippets only. For full code listings, please see go/opentelemetry and go/launcher in the Cloud Observability OpenTelemetry examples repository.

Sending OpenTelemetry data directly to Cloud Observability without a Collector for most developer setups will suffice. For non-development setups, however, it is highly recommended that you send OpenTelemetry data to Cloud Observability by way of the OpenTelemetry Collector. This can be done with or without a Launcher, as we’ll see below.

Pre-Requisites

Before you get started with sending OpenTelemetry data to Cloud Observability, you will need the following:

  • A Cloud Observability account.
  • A Cloud Observability access token for the Cloud Observability project you would like to use.
  • The OpenTelemetry Getting Started guide for Go.

Install OpenTelemetry Packages

In your application code, you will need to install dependencies and import OpenTelemetry packages before you can send data to Cloud Observability.

Start by installing the Golang OpenTelemetry packages. Do this by opening up a terminal window and pasting the following snippet:

Start tabs

Direct (gRPC)

1
2
3
4
5
6
7
8

go get go.opentelemetry.io/otel \
	go.opentelemetry.io/otel/exporters/otlp/otlptrace \
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
	go.opentelemetry.io/otel/propagation \
	go.opentelemetry.io/otel/sdk/resource \
	go.opentelemetry.io/otel/sdk/trace \
	go.opentelemetry.io/otel/semconv/v1.10.0 \
	go.opentelemetry.io/otel/trace

Collector (gRPC)

1
2
3
4
5
6
7
8

go get go.opentelemetry.io/otel \
	go.opentelemetry.io/otel/exporters/otlp/otlptrace \
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
	go.opentelemetry.io/otel/propagation \
	go.opentelemetry.io/otel/sdk/resource \
	go.opentelemetry.io/otel/sdk/trace \
	go.opentelemetry.io/otel/semconv/v1.10.0 \
	go.opentelemetry.io/otel/trace

Launcher

1
2
3
4

go get github.com/lightstep/otel-launcher-go/launcher \ 
	go.opentelemetry.io/otel \ 
	go.opentelemetry.io/otel/semconv/v1.10.0 \ 
	go.opentelemetry.io/otel/trace

End tabs

For non-Launchers only: If you wish to use HTTP instead of gRPC, replace otlptracegrpc with otlptracehttp.

Code Setup

Before you can start sending OpenTelemetry data to Lightstep, you will need to:

  • Configure an Exporter (tells OpenTelemetry how to send data to Lightstep)
  • Configure a TracerProvider (provides an entrypoint to the OpenTelemetry API, allowing you to create Spans)

Import OpenTelemetry Packages

Now that you’ve installed the OpenTelemetry packages, you will need to import them in your application code.

Open up your application code, and add the following imports to your .go file:

Start tabs

Direct (gRPC)

1
2
3
4
5
6
7
8
9
10

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.10.0"
	"go.opentelemetry.io/otel/trace"
)

Collector (gRPC)

1
2
3
4
5
6
7
8
9
10

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.10.0"
	"go.opentelemetry.io/otel/trace"
)

Launcher

1
2
3
4
5
6

import (
	"github.com/lightstep/otel-launcher-go/launcher"
	"go.opentelemetry.io/otel"
	semconv "go.opentelemetry.io/otel/semconv/v1.10.0"
	"go.opentelemetry.io/otel/trace"
)

End tabs

For non-Launchers only: If you wish to use HTTP instead of gRPC, replace otlptracegrpc with otlptracehttp.

Configure

Start by setting up the connection to your endpoint, as illustrated in the code snippets below.

Start tabs

Direct (gRPC)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

var (
	tracer         trace.Tracer
	serviceName    = "test-go-server-grpc"
	serviceVersion = "0.1.0"
	endpoint       = "ingest.lightstep.com:443"
	lsToken        = "<LS_ACCESS_TOKEN>"
	lsEnvironment  = "dev"
)

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {

	var headers = map[string]string{
		"lightstep-access-token": lsToken,
	}

	client := otlptracegrpc.NewClient(
		otlptracegrpc.WithHeaders(headers),
		otlptracegrpc.WithEndpoint(endpoint),
	)
	return otlptrace.New(ctx, client)
}

func newTraceProvider(exp *otlptrace.Exporter) *sdktrace.TracerProvider {

	resource, rErr :=
		resource.Merge(
			resource.Default(),
			resource.NewWithAttributes(
				semconv.SchemaURL,
				semconv.ServiceNameKey.String(serviceName),
				semconv.ServiceVersionKey.String(serviceVersion),
				attribute.String("environment", lsEnvironment),
			),
		)

	if rErr != nil {
		panic(rErr)
	}

	return sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp),
		sdktrace.WithResource(resource),
	)
}

Collector (gRPC)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

var (
	tracer         trace.Tracer
	serviceName    = "test-go-server-collector"
	serviceVersion = "0.1.0"
	collectorAddr  = "localhost:4317"
	lsEnvironment  = "dev"
)

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {
	exporter, err :=
		otlptracehttp.New(ctx,
			// WithInsecure lets us use http instead of https (for local dev only).
			otlptracegrpc.WithInsecure(),
			otlptracegrpc.WithEndpoint(collectorAddr),
		)

	return exporter, err
}

func newTraceProvider(exp *otlptrace.Exporter) *sdktrace.TracerProvider {

	resource, rErr :=
		resource.Merge(
			resource.Default(),
			resource.NewWithAttributes(
				semconv.SchemaURL,
				semconv.ServiceNameKey.String(serviceName),
				semconv.ServiceVersionKey.String(serviceVersion),
				attribute.String("environment", lsEnvironment),
			),
		)

	if rErr != nil {
		panic(rErr)
	}

	return sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp),
		sdktrace.WithResource(resource),
	)
}

Launcher

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

var (
	tracer         trace.Tracer
	serviceName    = "test-go-server-launcher"
	serviceVersion = "0.1.0"
	endpoint       = "ingest.lightstep.com:443"
	lsToken        = "<LS_ACCESS_TOKEN>"
)

func newLauncher() launcher.Launcher {

	otelLauncher := launcher.ConfigureOpentelemetry(
		launcher.WithServiceName(serviceName),
		launcher.WithServiceVersion(serviceVersion),
		launcher.WithAccessToken(lsToken),
		launcher.WithSpanExporterEndpoint(endpoint),
    launcher.WithMetricExporterEndpoint(endpoint),    
		launcher.WithPropagators([]string{"tracecontext", "baggage"}),
		launcher.WithResourceAttributes(map[string]string{
			string(semconv.ContainerNameKey): "my-container-name",
		}),
	)

	return otelLauncher
}

End code tabs

Notes: Direct (OTLP from application code)

  • The endpoint is set to ingest.lightstep.com:443, which points to the public Microsatellite pool. If you are using an on-premise satellite pool, then check out these docs.
  • You must provide a value for <LS_ACCESS_TOKEN> with your own Lightstep access token.

  • If you wish to use HTTP instead of gRPC, your client connection will look like this:

    1
2
3
4
5

      client := otlptracehttp.NewClient(
      otlptracehttp.WithHeaders(headers),
      otlptracehttp.WithEndpoint(endpoint),
      otlptracehttp.WithURLPath("traces/otlp/v0.9"),
  )

    Note that otlptracegrpc has been replaced with otlptracehttp, and that we added a new configuration attribute, otlptracehttp.WithURLPath. This URL path is required for HTTP connections, as per these docs.

Expandable end

Notes: OpenTelemetry Collector

  • The endpoint is your Collector’s URL.
  • In the example below, the Collector endpoint is set to localhost:4317, which means that the OpenTelemetry Collector is running locally, using Docker, listening on gRPC port 4317.
  • To use HTTP instead of gRPC, change the port to 4318, and change all occurrences of otlptracegrpc to otlptracehttp.

Expandable end

Notes: Launcher

  • The endpoint is set to ingest.lightstep.com:443, which points to the public Microsatellite pool. If you are using an on-premise satellite pool, then check out these docs.
  • You must provide a value for <LS_ACCESS_TOKEN> with your own Lightstep access token.

  • If you want to send your Traces to an OpenTelemetry Collector instance instead of sending them directly to Lightstep, simply change the endpoint to 0.0.0.0:4317 (assuming you are running a Collector instance locally), and modify the configuration as follows:

    1
2
3
4
5
6
7
8
9
10
11
12

    otelLauncher := launcher.ConfigureOpentelemetry(
  launcher.WithServiceName(serviceName),
  launcher.WithServiceVersion(serviceVersion),
  launcher.WithSpanExporterInsecure(true),
  launcher.WithSpanExporterEndpoint(endpoint),
  launcher.WithMetricExporterInsecure(true),
  launcher.WithMetricExporterEndpoint(endpoint),
  launcher.WithPropagators([]string{"tracecontext", "baggage"}),
  launcher.WithResourceAttributes(map[string]string{
    string(semconv.ContainerNameKey): "my-container-name",
  }),
)

    Note how we have added two new configuration options: WithSpanExporterInsecure and WithMetricExporterInsecure. We have also removed WithAccessToken, since the Lightstep access token is set as part of the Collector config file.

Expandable end

Initialize

You are now ready to initialize your setup functions in your main function.

Start tabs

Direct (gRPC & HTTP)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

func main() {
	ctx := context.Background()

	exp, err := newExporter(ctx)
	if err != nil {
		log.Fatalf("failed to initialize exporter: %v", err)
	}

	tp := newTraceProvider(exp)
	defer func() { _ = tp.Shutdown(ctx) }()

	otel.SetTracerProvider(tp)

	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.TraceContext{},
			propagation.Baggage{},
		),
	)

	tracer = tp.Tracer(serviceName, trace.WithInstrumentationVersion(serviceVersion))

  // More code here
  ...
}

Collector (gRPC & HTTP)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

func main() {
	ctx := context.Background()

	exp, err := newExporter(ctx)
	if err != nil {
		log.Fatalf("failed to initialize exporter: %v", err)
	}

	tp := newTraceProvider(exp)
	defer func() { _ = tp.Shutdown(ctx) }()

	otel.SetTracerProvider(tp)

	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.TraceContext{},
			propagation.Baggage{},
		),
	)

	tracer = tp.Tracer(serviceName, trace.WithInstrumentationVersion(serviceVersion))

  // More code here
  ...
}

Launcher

1
2
3
4
5
6
7
8
9

func main() {
	otelLauncher := newLauncher()
	defer otelLauncher.Shutdown()

	tracer = otel.Tracer(serviceName)

// More code here
  ...
}

End code tabs

Troubleshooting

gRPC Connectivity

To debug Go gRPC connectivity issues, set the following gRPC debug environment variables:

1
2
3

export GRPC_GO_LOG_VERBOSITY_LEVEL=99
export GRPC_GO_LOG_SEVERITY_LEVEL=info
go run <your_app>.go

Expandable end

Check for multiple versions of OpenTelemetry

If multiple versions of OpenTelemetry are installed, traces are not created or propagated correctly. Check your dependencies to ensure that only a single version of OpenTelemetry is installed.

Expandable end

Check that security is configured correctly

Cloud Observability’s public Microsatellites only accept spans via a secure endpoint. If you see security related errors (e.g., Netty TLS errors with Java), you may not have the necessary root certificate installed on the machine where the tracer is running. To add a root certificate, see the documentation about encrypted connections.

Expandable end

Check that Metrics are enabled in the OpenTelemetry Collector

Not seeing metrics come through the OpenTelemetry Collector? Make sure that you have defined a Metrics pipeline in your Collector’s YAML config file.

Cloud Observability access token

See also

Updated Jul 21, 2022