Lightstep accepts Prometheus Counters, Gauges, and Histograms through the OpenTelemetry Collector or the Prometheus Sidecar. However, there may be differences in how these metric instruments and measurements are displayed in Lightstep as opposed to Prometheus. This document collects these differences for reference.
Prometheus Cumulative Counters
Lightstep accepts a Counter, but for technical and philosophical reasons, we do not provide the underlying cumulative count to our graphs and instead plot them as deltas. The resolution of these deltas varies based on the age of the data - for example, 45 second buckets for data recorded in the past hour, 2 minute buckets for data recorded between one and four hours ago, and so forth.
For an example of how this might manifest, consider the
promhttp_metric_handler_requests_total counter that Prometheus exposes for itself (at the
/metrics endpoint). Assuming a regular scape cadence of 1 scrape per 10 seconds, graphing this metric in Prometheus would result in a plot of values increasing steadily, up and to the right. In Lightstep, the same metric represented as a count would result in a straight line over time assuming a consistent rate of scrape requests.
Default Grouping for Rate Calculations
When taking the
rate of a counter in Prometheus, the default plot will implicitly group the plot by labels. In Lightstep, you must explicitly specify the label to group by.