If you use the Prometheus sidecar to ingest metrics, you can now configure the side car specifically for a High Availability (HA) environment. You can attach a sidecar to each replica and then configure one of the sidecars to be the “leader” using the Kubernetes Coordination API.
Before enabling the
--leader-election.enabled flag in the sidecar, ensure that there is a service account for Prometheus in Kubernetes and then bind the role permissions to that service account.
Read about the sidecar and configuration options, including HA, here.