The Prometheus sidecar transmits data collected through Prometheus servers. The sidecar reads the Prometheus server’s
write-ahead log, then re-exports timeseries using the OpenTelemetry protocol to Lightstep.
To export metric data read from the Prometheus
write-ahead log, the sidecar uses Prometheus APIs to obtain metadata and apply the correct metric instrument type, labels, and resources to each timeseries. The sidecar maintains an in-memory cache of metadata needed to correctly and efficiently translate Prometheus log entries into complete OpenTelemetry protocol export requests.
Lightstep supports Prometheus server versions 2.10.0 (released May 2019) and above.
The sidecar runs in a container with read-write access to the Prometheus data directory.
You must have a Lightstep access token for the project to report metrics to.
Download the Prometheus sidecar.
1 2 3 4 5 6 7 8 9 10 11 12 server: sidecarContainers: - name: otel-sidecar image: lightstep/opentelemetry-prometheus-sidecar imagePullPolicy: Always args: - --prometheus.wal=/data/wal - --destination.endpoint=https://ingest.lightstep.com:443 - --destination.header=lightstep-access-token=<token> volumeMounts: - name: storage-volume mountPath: /data
<token>is the Lightstep project access token.
The sidecar sends OpenTelemetry Protocol v0.5 using gRPC to the Lightstep endpoint
ingest.lightstep.com:443, requiring only a Lightstep access token set in the
Labels output by Prometheus service discovery are applied as OpenTelemetry resources, while application-provided labels are preserved on individual timeseries in the OpenTelemetry data model. Additional labels can be added from the sidecar command-line.
You can use the sidecar in a High Availability (HA) environment by attaching one to each replica and then configuring one of the sidecars to be the “leader” using the Kubernetes Coordination API.
Before enabling the
--leader-election.enabled flag in the sidecar, ensure that there is a service account for Prometheus in Kubernetes and then bind the role permissions to that service account.
Prometheus Recording Rules, which are timeseries computed in the Prometheus server, are supported provided these output timeseries define the conventional target labels job and instance.
You can find more information in the repo’s README.