Istio metrics

You can use the OpenTelemetry Collector to ingest Istio metrics using Istio proxies within each pod. These metrics can be efficiently collected and forwarded to the OpenTelemetry Collector, which acts as a central collection and processing point and then sends those metrics to Cloud Observability.

To set up Istio Metrics Ingestion using OpenTelemetry Collector, you:

  1. Create a configuration file for the OpenTelemetry Collector, specifying the sources, processors, and exporters to be used.
  2. Create a configuration file for the Istio Operator for your Kubernetes environment.
  3. Create access permissions for the Collector.
  4. Configure Istio to send metrics to the OpenTelemetry Collector by modifying Istio’s configuration files.
  5. Create your Cloud Observability access token as a Kubernetes secret.
  6. Verify that metrics from Istio are successfully ingested by the OpenTelemetry Collector and exported to Cloud Observability.

Prerequisites

  • Istio configured as a network mesh on a Kubernetes cluster
  • A running OpenTelemetry Collector v0.77 or later, configured to export metric data to Cloud Observability
  • A good understanding of Kubernetes

Configure the Collector

You use a Kubernetes ConfigMap file to configure the Collector to scrape Prometheus metrics and a deployment file to deploy it to Kubernetes.

  1. Create an otel-collector-configmap.yaml file by copying the following code.

    otel-collector-configmap.yaml

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

     apiVersion: v1
 kind: ConfigMap
 metadata:
   name: otel-collector-conf
 data:
   otel-collector-config.yaml: |
     receivers:
       prometheus:
         config:
           scrape_configs:
             - job_name: 'otel-collector'
               scrape_interval: 5s
               static_configs:
                 - targets: ['0.0.0.0:8888']
             - job_name: "istio"
               scrape_interval: 5s
               metrics_path: "/stats/prometheus"
               kubernetes_sd_configs:
                 - role: "pod"
               relabel_configs:
               // add labels
  
    processors:
       batch:

     exporters:
         logging:
         loglevel: debug
       otlp:
         endpoint: ingest.lightstep.com:443 #US data center
         #endpoint: ingest.eu.lightstep.com:443 #EU data center
         headers:
                 "lightstep-access-token": "{LS_ACCESS_TOKEN}"

     service:
       telemetry:
         logs:
           level: "debug"
       pipelines:
         metrics:
           receivers: [prometheus]
           processors: [batch]
           exporters: [logging,otlp]

  2. Create an otel-collector-deployment.yaml file by copying the following code.

    otel-collector-deployment.yaml

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

     apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: otel-collector
   labels:
     app: otel-collector
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: otel-collector
   template:
     metadata:
       labels:
         app: otel-collector
     spec:
       serviceAccountName: otel-collector
       containers:
       - name: otel-collector
         image: otel/opentelemetry-collector-contrib:latest
         args:
         - "--config=/conf/otel-collector-config.yaml"
         ports:
         - containerPort: 55681
         env:
         - name: LS_ACCESS_TOKEN
           valueFrom:
             secretKeyRef:
               name: lightstep-access-token
               key: {LS_ACCESS_TOKEN}

         volumeMounts:
         - name: otel-collector-config-vol
           mountPath: /conf
       volumes:
       - configMap:
           name: otel-collector-conf
         name: otel-collector-config-vol

Configure Istio Operator

Create an istio-operator.yaml file by copying the following code.

istio-operator.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: default
  components:
    prometheus:
      enabled: true
  values:
    global:
      proxy:
        autoInject: "enabled"

Configure RBAC (Role-based access control)

Create an otel-collector-rbac.yaml file by copying the following code.

otel-collector-rbac.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/metrics
      - nodes/proxy
      - nodes/stats
      - pods
      - services
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-collector
subjects:
  - kind: ServiceAccount
    name: otel-collector
    namespace: default

Configure LS secret

Create a lightstep-secret.yaml file to hold your access token by copying the following code and replacing ${LS_ACCESS_TOKEN} with your access token.

lightstep-secret.yaml

1
2
3
4
5
6
7

apiVersion: v1
kind: Secret
metadata:
  name: lightstep-access-token
type: Opaque
data:
  access_token: ${LS_ACCESS_TOKEN}

Apply the configurations to your Kubernetes cluster

  1. Apply the deployment file.

    1

     kubectl apply -f otel-collector-deployment.yaml

    Verify that the OpenTelemetry Collector is running:

    1

     kubectl get pods -l app=otel-collector

  2. Apply the ConfigMap to your Kubernetes cluster

    1

     kubectl apply -f otel-collector-configmap.yaml

    Verify that the ConfigMap has been created

    1

     kubectl get configmap otel-collector-conf

  3. Apply the Secret to your Kubernetes cluster

    1

     kubectl apply -f lightstep-secret.yaml

View metrics in Cloud Observability

Once you have Cloud Observability ingesting the Istio metrics, you can begin using them to build dashboards in Cloud Observability.

See also

Create and manage dashboards

Updated Jun 21, 2023