Lightstep accepts metrics in the OpenTelemetry Protocol (OTLP). This protocol was designed as a format to support both StatsD and Prometheus style metrics exposition for reporting to backend systems. OpenTelemetry distinguishes between these differing formats by introducing a concept of ‘aggregation temporality’; This can be a delta (statsd-like), or cumulative (prometheus-like).
Delta temporality means that the aggregator reports the change in a value since the last time of report, and cumulative temporality means that the aggregator reports the cumulative change in a value since a fixed starting time.
OTLP supports three distinct point types: Sums, Gauges, and Histograms. All of these data points can be expressed as integers or floating point measurements. Sums can use either temporality, and can be monotonic (always increasing) or not; Histograms can also use either temporality. Lightstep fully supports this protocol.
Accepted Service Endpoints and Authentication
The OpenTelemetry Metrics Protocol documents the expected receiver method for exporting metrics, which is fully supported by Lightstep. gRPC and HTTP transport is available, with the appropriate application/x-protobuf or application/json content type. We suggest gRPC or HTTP with application/x-protobuf for metrics exposition to Lightstep at this time.
All authentication occurs using the same token for metrics as it does for tracing, regardless of protocol used. The Lightstep-Access-Token header should be set to your project access token.
For the appropriate endpoint to send metric traffic to, please consult the following table –
| Transport | Content-Type | Endpoint |
|---|---|---|
| gRPC | protobuf | ingest.lightstep.com:443 |
| HTTP | application/x-protobuf | https://ingest.lightstep.com/metrics/otlp/v0.5 |
| HTTP | application/json | https://ingest.lightstep.com/metrics/otlp/v0.5 |
OpenTelemetry Protocol Details
Every export contains a series of Metric time series data, each containing one or more data points, wrapped by the InstrumentationLibrary and Resource objects. InstrumentationLibrary contains a description of which library generated the metric points, and Resource describes the process which produced the metric points. Lightstep does not record nor use the InstrumentationLibrary field at this time. Resource fields are translated into labels where appropriate (such as service.name, service.version, etc.)
Please note that OTLP contains repeated fields on multiple levels of the protocol to support aggregation early in the data ingestion path. We suggest grouping metric points by Resource and Metric, then making use of the repeated fields inside those objects to send multiple points in a compressed manner.
Request Size and Compression
Requests are limited to 4mb of compressed data. For gRPC, use snappy compression. For HTTP, use gzip. For writes larger than that, chunk into batches of around 64KB (uncompressed) for best performance.
Translating Custom Metrics Pipelines
At this time, we suggest creating a custom OTLP exporter in lieu of using the OpenTelemetry SDK to send custom metrics to Lightstep – this is because the metrics SDK continues to be in active development, and is less stable compared to the Metrics API Specification.
Please consult the following external resources for more information on the metrics protocol, service, and specification:
Frequently Asked Questions
What are the allowed characters for metric names and labels?
Metric names and label keys should meet the following requirements: [a-Z][0-9][\.-/]+. Label keys should be less than 100 characters, and values should be less than 1000 characters.
What happens to values with different labels but the same name?
Each unique combination of name and labels corresponds to a single time series.
Future Considerations
OpenTelemetry continues to be developed, and Lightstep will continue to support OpenTelemetry.
Protocol Support
As new versions of the OpenTelemetry Protocol are released by the project, we will support them for data ingest.
Histograms
As of OpenTelemetry Metrics v0.5, we accept Histogram data points with explicit boundaries. Currently, our metrics system implements Histograms with log-linear boundaries. Upon ingestion, Lightstep will interpolate OpenTelemetry Histograms into log-linear histograms.
The OpenTelemetry metrics group is currently evaluating decisions on long-term histogram support and default histogram aggregations. See this discussion for more information.
We will continue to support OpenTelemetry as histogram encoding support evolves, and we expect to store OpenTelemetry histogram data without interpolation directly in the future.