Scale an OpenTelemetry Collector's pipeline in Kubernetes

Scaling your collector deployment

This topic provides instructions for scaling the OpenTelemetry Collector deployment for tracing, metrics, or logs in Kubernetes.

Prerequisites

Scaling possibilities

There are two ways to scale your deployment: vertically and horizontally. In Kubernetes, vertical scaling entails increasing the memory and CPU requests/limits of your Kubernetes pod. Horizontal scaling means adding more pods (collector instances) to handle your traffic.

We recommend you use the pre-built dashboard for the OpenTelemetyr Collector to monitor and tune resource usage of collectors.

Vertically scale the Collector

Define a new values.yaml for your collectors deployed using the Kubernetes Quickstart. Adjust the following blocks, as needed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Vertically scale the tracesCollector
tracesCollector:
  resources:
    limits:
      cpu: CPU_LIMIT_HERE
      memory: MEMORY_LIMIT_HERE
    requests:
      cpu: CPU_LIMIT_HERE
      memory: MEMORY_LIMIT_HERE
# Vertically scale the singleton collector for cluster-level metrics data
clusterCollector:
  resources:
    limits:
      cpu: CPU_LIMIT_HERE
      memory: MEMORY_LIMIT_HERE
    requests:
      cpu: CPU_LIMIT_HERE
      memory: MEMORY_LIMIT_HERE
# Vertically scale the log collector
logsCollector:
  resources:
    limits:
      cpu: CPU_LIMIT_HERE
      memory: MEMORY_LIMIT_HERE
    requests:
      cpu: CPU_LIMIT_HERE
      memory: MEMORY_LIMIT_HERE

Ensure that your memory and CPU values are well formatted for Kubernetes

Horizontally scale the Collector

In the values.yaml for your collector deployment, adjust the replica count:

1
2
3
4
5

tracesCollector:
  hpa:
    minReplicas: 1
    maxReplicas: 3
    targetMemoryUtilization: 70

Apply the changes

To upgrade the otel-cloud-stack installed in the Kubernetes Quickstart, run the following:

1
2
3
4
5
6

helm upgrade otel-cloud-stack lightstep/otel-cloud-stack \
  - n default --set tracesCollector.enabled=true \
  --set logsCollector.enabled=true
  # --set otlpDestinationOverride="ingest.eu.lightstep.com:443" \ # EU data center
  # --set opAMPBridge.endpoint="wss://opamp.eu.lightstep.com/v1/opamp" \ # EU data center
  -f values.yaml -n opentelemetry --install

Observe the changes

Your new Collectors should appear in your OpenTelemetry Collector dashboard.

See also

Install the OpenTelemetry Collector on Kubernetes

Updated Oct 21, 2022