Cloud Observability recommends running the OpenTelemetry Collector with the Prometheus receiver to ingest infrastructure metrics. This ensures the highest data quality and completeness, and also allows the Collector to leverage the Prometheus ecosystem of exporters to scrape targets.
There are three approaches to deploying the Collector in Kubernetes:
When first deploying the OpenTelemetry Collector, you can start with a single replica deployment within a Kubernetes cluster, or for additional scalability, deploy Collectors as a DaemonSet to scrape appliction metrics. Both modes can be combined. For example, you can use the DaemonSet to scrape application metrics along with a single deploy to scrape infrastructure metrics and static targets.
For clusters that require increased performance without sacrificing increased resourcing, the StatefulSet collector can be used to shard a Prometheus configuration. The StatefulSet can be scaled both horiztonally and vertically with a specified amount of replicas.
The StatefulSet collector is under active development, some features may not yet be available.
Once the Collector has been deployed for a subset of services, it’s much easier to estimate metrics traffic for remaining services, and then plan and scale other the deployments appropriately. In any deployment mode, the Collector is configured to scrape Prometheus metrics using scrape targets. Compare and contrast the deployment types below:
Use Case: Trying out the OpenTelemetry Collector for the first time for a subset of services. Enables scraping application metrics, infrastructure metrics, and static targets in a Kubernetes cluster.
Use case: Scraping application metrics on each node. Best to use when workload on each node is constant or has some fixed limit, or the resources needed by each node is similar. Pods in a DaemonSet have the same configuration, so if all the nodes have similar resource needs, resources will be used more efficiently.
Use case: Horizontally scalable, sharded Prometheus scraping. Allows for the provided Prometheus scrape configuration to be sharded by a set of Opentelemetry Collectors. Best to use for most topologies as it allows for more fault tolerance.
Updated Aug 30, 2022