You can use the OpenTelemetry Collector to ingest Istio metrics using Istio proxies within each pod. These metrics can be efficiently collected and forwarded to the OpenTelemetry Collector, which acts as a central collection and processing point and then sends those metrics to Cloud Observability.
To set up Istio Metrics Ingestion using OpenTelemetry Collector, you:
- Create a configuration file for the OpenTelemetry Collector, specifying the sources, processors, and exporters to be used.
- Create a configuration file for the Istio Operator for your Kubernetes environment.
- Create access permissions for the Collector.
- Configure Istio to send metrics to the OpenTelemetry Collector by modifying Istio’s configuration files.
- Create your Cloud Observability access token as a Kubernetes secret.
- Verify that metrics from Istio are successfully ingested by the OpenTelemetry Collector and exported to Cloud Observability.
- Istio configured as a network mesh on a Kubernetes cluster
- A running OpenTelemetry Collector v0.77 or later, configured to export metric data to Cloud Observability
- A good understanding of Kubernetes
You use a Kubernetes ConfigMap file to configure the Collector to scrape Prometheus metrics and a deployment file to deploy it to Kubernetes.
-
Create an otel-collector-configmap.yaml
file by copying the following code.
otel-collector-configmap.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
| apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-conf
data:
otel-collector-config.yaml: |
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 5s
static_configs:
- targets: ['0.0.0.0:8888']
- job_name: "istio"
scrape_interval: 5s
metrics_path: "/stats/prometheus"
kubernetes_sd_configs:
- role: "pod"
relabel_configs:
// add labels
processors:
batch:
exporters:
logging:
loglevel: debug
otlp:
endpoint: ingest.lightstep.com:443 #US data center
#endpoint: ingest.eu.lightstep.com:443 #EU data center
headers:
"lightstep-access-token": "{LS_ACCESS_TOKEN}"
service:
telemetry:
logs:
level: "debug"
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [logging,otlp]
|
-
Create an otel-collector-deployment.yaml
file by copying the following code.
otel-collector-deployment.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
| apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
labels:
app: otel-collector
spec:
replicas: 1
selector:
matchLabels:
app: otel-collector
template:
metadata:
labels:
app: otel-collector
spec:
serviceAccountName: otel-collector
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:latest
args:
- "--config=/conf/otel-collector-config.yaml"
ports:
- containerPort: 55681
env:
- name: LS_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: lightstep-access-token
key: {LS_ACCESS_TOKEN}
volumeMounts:
- name: otel-collector-config-vol
mountPath: /conf
volumes:
- configMap:
name: otel-collector-conf
name: otel-collector-config-vol
|
Create an istio-operator.yaml
file by copying the following code.
istio-operator.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
profile: default
components:
prometheus:
enabled: true
values:
global:
proxy:
autoInject: "enabled"
|
Create an otel-collector-rbac.yaml
file by copying the following code.
otel-collector-rbac.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
| apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: otel-collector
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/metrics
- nodes/proxy
- nodes/stats
- pods
- services
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: otel-collector
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: otel-collector
subjects:
- kind: ServiceAccount
name: otel-collector
namespace: default
|
Create a lightstep-secret.yaml
file to hold your access token by copying the following code and replacing ${LS_ACCESS_TOKEN}
with your access token.
lightstep-secret.yaml
1
2
3
4
5
6
7
| apiVersion: v1
kind: Secret
metadata:
name: lightstep-access-token
type: Opaque
data:
access_token: ${LS_ACCESS_TOKEN}
|
-
Apply the deployment file.
1
| kubectl apply -f otel-collector-deployment.yaml
|
Verify that the OpenTelemetry Collector is running:
1
| kubectl get pods -l app=otel-collector
|
-
Apply the ConfigMap to your Kubernetes cluster
1
| kubectl apply -f otel-collector-configmap.yaml
|
Verify that the ConfigMap has been created
1
| kubectl get configmap otel-collector-conf
|
-
Apply the Secret to your Kubernetes cluster
1
| kubectl apply -f lightstep-secret.yaml
|
Once you have Cloud Observability ingesting the Istio metrics, you can begin using them to build dashboards in Cloud Observability.
See also
Create and manage dashboards
Updated Jun 21, 2023