Send traces to Lightstep with Python

This Quick Start guide shows you how to configure your Python applications to send OpenTelemetry traces to Cloud Observability.

This guide does not provide documentation on application instrumentation. For Python-specific information on instrumentation, please see the Python OpenTelemetry getting started guide.

The sections below contain code snippets only. For full code listings, please see python/opentelemetry and python/launcher in the Cloud Observability OpenTelemetry examples repository.

Sending OpenTelemetry data directly to Cloud Observability without a Collector for most developer setups will suffice. For non-development setups, however, it is highly recommended that you send OpenTelemetry data to Cloud Observability by way of the OpenTelemetry Collector. This can be done with or without a Launcher, as we’ll see below.

Pre-Requisites

Before you get started with sending OpenTelemetry data to Cloud Observability, you will need the following:

Install OpenTelemetry Packages

In your application code, you will need to install dependencies and import OpenTelemetry packages before you can send data to Cloud Observability.

Start by installing the Python OpenTelemetry packages. Do this by opening up a terminal window and pasting the following snippet:

Start tabs

Direct (gRPC)

1
2
3
4

pip install opentelemetry-distro \
	opentelemetry-exporter-otlp

opentelemetry-bootstrap -a install

Collector

1
2
3
4

pip install opentelemetry-distro \
	opentelemetry-exporter-otlp

opentelemetry-bootstrap -a install

Launcher

1
2
3
4

pip install opentelemetry-launcher \
	protobuf==3.20.1

opentelemetry-bootstrap -a install

End tabs

The opentelemetry-bootstrap -a install command will read through the packages installed in your active site-packages folder, and will install the applicable auto-instrumentation libraries. For example, if you already installed the flask package, running opentelemetry-bootstrap -a install will install opentelemetry-instrumentation-flask for you. If you leave out -a install, it will simply list out the recommended auto-instrumentation packages to be installed.

For more information, check out the official OpenTelemetry docs here.

The Python package installs require gcc and gcc-c++, which you may need to install if you’re running a slim version of Linux. More info here.

opentelemetry-distro (Non-Launchers): The opentelemetry-distro package installs the API, SDK, and the opentelemetry-bootstrap and opentelemetry-instrument tools.

Launchers only: It is necessary to force a specific version of protobuf because of Launcher compatibility issues with newer versions. This has already been fixed in opentelemetry-python.

Code Setup

Although auto-instrumentation is a great way to get started quickly with instrumenting your Python application, it is often not sufficient. In order to gain greater insights into your application code, you should also add manual instrumentation to your business logic. More on auto-instrumentation vs manual instrumentation here.

In order to manually instrument your Python code, you need to acquire a Tracer. The Tracer is responsible for creating Spans and interacting with the Context.

Import OpenTelemetry Packages

Before you can acquire a Tracer, you must first import the appropriate package. Open up your application code, and add the following import to your .py file:

1

from opentelemetry import trace

Acquire a Tracer

Now you are ready to acquire your tracer. This should be declared as a global variable in your .py file.

1

tracer = trace.get_tracer_provider().get_tracer(__name__)

Start Auto-Instrumentation

The opentelemetry-instrument tool is used to configure OpenTelemetry and auto-instrument your code. It is used as a wrapper around your Python program.

The tool will automatically detect packages used by your Python program and when possible, apply automatic tracing instrumentation on them. This means that your program will get automatic distributed tracing where applicable for free without having to make any code changes at all.

A full list of auto-instrumented Python libraries can be found here.

Start tabs

Direct (gRPC)

1
2
3
4
5
6
7
8

export OTEL_EXPORTER_OTLP_TRACES_HEADERS="lightstep-access-token=<LS_ACCESS_TOKEN>"

opentelemetry-instrument \
    --traces_exporter console,otlp_proto_grpc \
    --metrics_exporter console,otlp_proto_grpc \
    --service_name <service_name> \
    --exporter_otlp_endpoint "ingest.lightstep.com:443" \
    python <your_app>.py

Collector (gRPC)

1
2
3
4
5

opentelemetry-instrument \
   --traces_exporter console,otlp \
   --metrics_exporter console,otlp \
   --service_name test-py-auto-collector-server \
   python <your_app>.py

Launcher

1
2
3
4
5

export LS_ACCESS_TOKEN="<LS_ACCESS_TOKEN>"

opentelemetry-instrument \
    --service_name <service_name> \
    python <your_app>.py

End tabs

To view traces in your Cloud Observability project, click explorer in the left navigation bar, and then click on any span in the Trace Analysis table.

Notes: Direct (OTLP from application code)

  • Replace <LS_ACCESS_TOKEN> with your own Lightstep access token.
  • traces_exporter and metrics_exporter specify which trace exporter and which metrics to use, respectively. In this case, traces and metrics are being exported to console (stdout) and to otlp_proto_grpc. The otlp_proto_grpc option tells opentelemetry-instrument to send it to an endpoint that accepts OTLP via gRPC. The full list of available options for traces_exporter can be found here.
  • service_name sets the name of the service. This is the value that will show up in the Lightstep service explorer. Be sure to replace <service_name> with your own service name.
  • exporter_otlp_endpoint tells opentelemetry-instrument to send the traces to gRPC endpoint ingest.lightstep.com:443 (i.e. Lightstep).

To use HTTP instead of gRPC, make sure that opentelemetry-exporter-otlp-proto-http is installed. This should be already be installed as part of installing opentelemetry-exporter-otlp.

Next, opentelemetry-instrument must be altered as follows:

1
2
3
4
5
6

opentelemetry-instrument \
    --traces_exporter console,otlp_proto_http \
    --metrics_exporter console \
    --service_name <service_name> \
    --exporter_otlp_traces_endpoint "https://ingest.lightstep.com:443/traces/otlp/v0.9" \
    python <your_app>.py

Noteworthy items:

  • The traces_exporter uses otlp_proto_http instead of otlp_proto_grpc.
  • The exporter_otlp_traces_endpoint endpoint is https://ingest.lightstep.com/traces/otlp/v0.9 (see docs, instead of ingest.lightstep.com:443).
  • There is currently no metrics support for otlp_proto_http and there is no exporter_otlp_metrics_endpoint option, which is why metrics are being sent to console only.

Expandable end

Notes: OpenTelemetry Collector

Notice that the endpoint isn’t specified. That’s because it assumes that you are using the default Collector gRPC endpoint, 0.0.0.0:4317. The above command is the equivalent of saying:

1
2
3
4
5
6
7

opentelemetry-instrument \
  --traces_exporter console,otlp \
  --metrics_exporter console,otlp\
  --service_name test-py-auto-collector-server \
  --exporter_otlp_endpoint "0.0.0.0:4317" \
  --exporter_otlp_insecure true \
  python server.py

If you specify the endpoint, you must also specify --exporter_otlp_insecure true if a certificate isn’t configured with your Collector.

Noteworthy items:

  • otlp, used in configuring traces_exporter and metrics_exporter, is equivalent to using otlp_proto_grpc
  • To use a different Collector endpoint, simply replace it with your own. If you don’t have a Certificate configured with your Collector, remember to add --exporter_otlp_insecure true
  • You don’t need to set OTEL_EXPORTER_OTLP_TRACES_HEADERS, because that’s already configured in the Collector’s config.yml file.
  • service_name sets the name of the service. This is the value that will show up in the Lightstep service explorer. Be sure to replace <service_name> with your own service name.

If you wish to use HTTP instead of gRPC, the command would then look like this:

1
2
3
4
5

opentelemetry-instrument \
  --traces_exporter console,otlp_proto_http \
  --metrics_exporter console,otlp_proto_http \
  --service_name test-py-auto-collector-server \
  python server.py

Which is the same as saying:

1
2
3
4
5
6
7

opentelemetry-instrument \
  --traces_exporter console,otlp_proto_http \
  --metrics_exporter console,otlp_proto_http \
  --service_name test-py-auto-collector-server \
  --exporter_otlp_endpoint "http://0.0.0.0:4318" \
  --exporter_otlp_insecure true \
  python server.py

Again, if you wish to use your own Collector endpoint, simply replace the value in exporter_otlp_endpoint, making sure that you prefix it with http:// or https://. Remember to add --exporter_otlp_insecure true if you don’t have a Certificate configured with your Collector.

Expandable end

Notes: Launcher

  • Instead of setting the Lightstep access token via OTEL_EXPORTER_OTLP_TRACES_HEADERS="lightstep-access-token=<LS_ACCESS_TOKEN>", the Launcher allows us to simplify it to LS_ACCESS_TOKEN="<LS_ACCESS_TOKEN>".
  • There is no need to specify exporter_otlp_traces_endpoint, because that is already implicitly set to ingest.lightstep.com:443.
  • service_name sets the name of the service. This is the value that will show up in the Lightstep service explorer. Be sure to replace <service_name> with your own service name.

To send data to Lightstep via an OpenTelmetry Collector using gRPC, opentelemetry-instrument would be configured as follows:

1
2
3
4
5

opentelemetry-instrument \
    --service_name <service_name>  \
    --exporter_otlp_traces_endpoint "0.0.0.0:4317" \
    --exporter_otlp_traces_insecure true \
    python <your_app>.py

Noteworthy items:

  • Do not set LS_ACCESS_TOKEN, since that’s already configured in the Collector’s config.yaml file.
  • If you attempt to override exporter_otlp_endpoint to send traces to a Collector, the traces will be sent directly to ingest.lightstep.com:443 instead of via the Collector. Instead, you need to override exporter_otlp_traces_endpoint,
  • exporter_otlp_traces_endpoint sends traces to a Collector running on 0.0.0.0:4317 (gRPC). If you wish to use a different Collector endpoint, simply include exporter_otlp_traces_endpoint, using your own Collector’s endpoint.
  • exporter_otlp_traces_insecure is set to true. This is required if you are using a Collector and if a certificate isn’t configured in the Collector.
  • There is currently no HTTP support for Python Launchers.

Expandable end

Troubleshooting

Python package installs

The Python package installs require gcc and gcc-c++, which you may need to install if you’re running a slim version of Linux (e.g., CentOS):

Start tabs

CentOS

1
2

yum -y install python3-devel
yum -y install gcc-c++

Debian/Ubuntu

1
2

apt install -y python3-dev
apt install -y build-essential

Alpine

1
2

apk add python3-dev
apk add build-base

End tabs

Expandable end

gRPC Connectivity

To debug Python gRPC connectivity issues, set the following gRPC debug environment variables:

1
2
3

export GRPC_VERBOSITY=debug
export GRPC_TRACE=http,call_error,connectivity_state
opentelemetry-instrument python <your_app>.py

Expandable end

Check for multiple versions of OpenTelemetry

If multiple versions of OpenTelemetry are installed, traces are not created or propagated correctly. Check your dependencies to ensure that only a single version of OpenTelemetry is installed.

Expandable end

Check that security is configured correctly

Cloud Observability’s public Microsatellites only accept spans via a secure endpoint. If you see security related errors (e.g., Netty TLS errors with Java), you may not have the necessary root certificate installed on the machine where the tracer is running. To add a root certificate, see the documentation about encrypted connections.

Expandable end

Check that Metrics are enabled in the OpenTelemetry Collector

Not seeing metrics come through the OpenTelemetry Collector? Make sure that you have defined a Metrics pipeline in your Collector’s YAML config file.

Cloud Observability access token

See also

Updated Sep 13, 2022