Lightstep Observability accepts metrics in the OpenTelemetry Protocol (OTLP). This protocol was designed as a format to support both StatsD and Prometheus style metrics exposition for reporting to backend systems. OpenTelemetry distinguishes between these differing formats by introducing a concept of ‘aggregation temporality’; This can be a delta (statsd-like), or cumulative (prometheus-like).
Delta temporality means that the aggregator reports the change in a value since the last time of report, and cumulative temporality means that the aggregator reports the cumulative change in a value since a fixed starting time.
OTLP supports three distinct point types: Sums, Gauges, and Histograms. All of these data points can be expressed as integers or floating point measurements. Sums can use either temporality, and can be monotonic (always increasing) or not; Histograms can also use either temporality. Lightstep Observability fully supports this protocol.
Accepted service endpoints and authentication
The OpenTelemetry Metrics Protocol documents the expected receiver method for exporting metrics, which is fully supported by Lightstep Observability. gRPC and HTTP transport is available, with the appropriate application/x-protobuf or application/json content type. We suggest gRPC or HTTP with application/x-protobuf for metrics exposition to Lightstep at this time.
All authentication occurs using the same token for metrics as it does for tracing, regardless of protocol used. The Lightstep-Access-Token header should be set to your project access token.
For the appropriate endpoint to send metric traffic to, please consult the following tables:
gRPC protobuf
| Endpoint | Supported versions | Notes |
|---|---|---|
ingest.lightstep.com:443 |
All | This endpoint is backwards compatible to OTLP v0.5 |
HTTP application/x-protobuf
| Endpoint | Supported versions | Notes |
|---|---|---|
ingest.lightstep.com/metrics/otlp/v0.5OR ingest.lightstep.com/metrics/otlp/v0.6 |
v0.5-v0.7 | We recommend upgrading your OTLP version if you're on v0.5 |
ingest.lightstep.com/metrics/otlp/v0.9 |
v0.9 |
HTTP application/x-json
| Endpoint | Supported versions | Notes |
|---|---|---|
ingest.lightstep.com/metrics/otlp/v0.5 |
v0.5-v0.7 | We recommend upgrading your OTLP version if you're on v0.5 |
ingest.lightstep.com/metrics/otlp/v0.6 |
v0.5-v0.7 | |
ingest.lightstep.com/metrics/otlp/v0.9 |
v0.9 |
OpenTelemetry protocol details
Every export contains a series of Metric time series data, each containing one or more data points, wrapped by the InstrumentationLibrary and Resource objects. InstrumentationLibrary contains a description of which library generated the metric points, and Resource describes the process which produced the metric points. Lightstep Observability does not record nor use the InstrumentationLibrary field at this time. Resource fields are translated into attributes where appropriate (such as service.name, service.version, etc.)
Please note that OTLP contains repeated fields on multiple levels of the protocol to support aggregation early in the data ingestion path. We suggest grouping metric points by Resource and Metric, then making use of the repeated fields inside those objects to send multiple points in a compressed manner.
Request size and compression
Requests are limited to 4mb of compressed data. For gRPC, use snappy compression. For HTTP, use gzip. For writes larger than that, chunk into batches of around 64KB (uncompressed) for best performance.
Translating custom metrics pipelines
At this time, we suggest creating a custom OTLP exporter in lieu of using the OpenTelemetry SDK to send custom metrics to Lightstep Observability – this is because the metrics SDK continues to be in active development, and is less stable compared to the Metrics API Specification.
Please consult the following external resources for more information on the metrics protocol, service, and specification:
Frequently asked questions
What are the allowed characters for metric names and attributes?
Metric names and attribute keys should meet the following requirements: [a-Z][0-9][\.-/]+. Attribute keys should be less than 100 characters, and values should be less than 1000 characters.
What happens to values with different attribute but the same name?
Each unique combination of name and attributes corresponds to a single time series.
Future considerations
OpenTelemetry continues to be developed, and Lightstep Observability will continue to support OpenTelemetry.
Protocol support
As new versions of the OpenTelemetry Protocol are released by the project, we will support them for data ingest. We will also deprecate releases over time as they are no longer used by OpenTelemetry SDKs and tools.
Release History
OTLP v0.1 - v0.4 These releases were never supported by Lightstep Observability.
OTLP v0.5 This release is currently supported through the appropriate ingest path.
OTLP v0.6 This release had no changes to the metrics protocol.
OTLP v0.7 This release is currently supported through the appropriate ingest path.
OTLP v0.8 This release is not recommended and is not currently supported.
OTLP v0.9 This release is expected to be supported in Q321.
OTLP v0.10 This version of OTLP is currently unreleased. We anticipate support for it in Lightstep Observability by Q421.
Histograms
As of OpenTelemetry Metrics v0.5, we accept Histogram data points with explicit boundaries. Currently, our metrics system implements Histograms with log-linear boundaries. Upon ingestion, Lightstep Observability will interpolate OpenTelemetry Histograms into log-linear histograms.
The OpenTelemetry metrics group is currently evaluating decisions on long-term histogram support and default histogram aggregations. See this discussion for more information.
We will continue to support OpenTelemetry as histogram encoding support evolves, and we expect to store OpenTelemetry histogram data without interpolation directly in the future.
Validation errors
Lightstep Observability Metrics applies several forms of validation to incoming metric points, in addition to any rules set by the OpenTelemetry project concerning valid OTLP points. Common validation errors include the following:
- Change of instrument type (e.g., Counter to Gauge)
- Metric points greater than 24 hours old
- Empty metric name
When Lightstep Observability receives metric data from an OTLP source with invalid points, it returns success to the calling process as to ensure that the message is not retried.
We return several pieces of information to assist you in diagnosing metrics ingestion problems. The OpenTelemetry project has not formalized a structured error reporting format, nor semantic conventions for relaying errors at this time. Lightstep Observability has elected to emit this data in HTTP Response trailers.
As response trailers are limited in size, and since validation errors are likely to repeat, we return counts describing the problem and a single example error per request, including the metric name and an explanation of the error. These trailers, described below, are set when an export request is successfully processed with one or more validation errors present:
otlp-metrics-dropped:. This response trailer is set with the number of whole metric series dropped due to validation errors. These errors include an empty metric name or an unrecognized point kind.- `otlp-points-dropped:’. This response trailer is set with the number of points dropped due to validation errors. These errors include data over 24 hours old, or a change in instrument type.
otlp-invalid-*:. This response trailer consists of a short, hyphenated string that explains the reason the point or series was considered invalid, as well as the name of the metric that caused the error.
An example of parsing these trailers can be found in our Prometheus sidecar.