The Prometheus sidecar transmits data collected through Prometheus servers. The sidecar reads the Prometheus server’s write-ahead log, then re-exports timeseries using the OpenTelemetry protocol to Lightstep.

To export metric data read from the Prometheus write-ahead log, the sidecar uses Prometheus APIs to obtain metadata and apply the correct metric instrument type, labels, and resources to each timeseries. The sidecar maintains an in-memory cache of metadata needed to correctly and efficiently translate Prometheus log entries into complete OpenTelemetry protocol export requests.

Requirements

Lightstep supports Prometheus server versions 2.10.0 (released May 2019) and above.

The sidecar runs in a container with read-write access to the Prometheus data directory.

Prerequisites

You must have a Lightstep access token for the project to report metrics to.

Installation

Download the Prometheus sidecar.

Installation varies by system. As an example, to deploy the sidecar using Helm alongside a Kubernetes deployment of Prometheus using the Prometheus Community Helm Charts:

1
2
3
4
5
6
7
8
9
10
11
12
server:
  sidecarContainers:
  - name: otel-sidecar
    image: lightstep/opentelemetry-prometheus-sidecar
    imagePullPolicy: Always
    args:
    - --prometheus.wal=/data/wal
    - --destination.endpoint=https://ingest.lightstep.com:443
    - --destination.header=lightstep-access-token=<token>
    volumeMounts:
    - name: storage-volume
      mountPath: /data

Where <token>is the Lightstep project access token.

The sidecar sends OpenTelemetry Protocol v0.5 using gRPC to the Lightstep endpoint ingest.lightstep.com:443, requiring only a Lightstep access token set in the lightstep-access-token header.

Labels output by Prometheus service discovery are applied as OpenTelemetry resources, while application-provided labels are preserved on individual timeseries in the OpenTelemetry data model. Additional labels can be added from the sidecar command-line.

You can use the sidecar in a High Availability (HA) environment by attaching one to each replica and then configuring one of the sidecars to be the “leader” using the Kubernetes Coordination API.
Before enabling the --leader-election.enabled flag in the sidecar, ensure that there is a service account for Prometheus in Kubernetes and then bind the role permissions to that service account.

Recording Rules

Prometheus Recording Rules, which are timeseries computed in the Prometheus server, are supported provided these output timeseries define the conventional target labels job and instance.

You can find more information in the repo’s README