Istio emits metrics through OpenTelemetry-compatible endpoints using each pod’s Envoy (or Istio proxy), which can be forwarded to the OpenTelemetry Collector by defining attributes in Istio’s mesh-config.yaml template. From there the metrics are processed and exported to Lightstep Observability. The process of setting up Istio observability using OpenTelemetry is broken up into four parts:

  1. Configure a Helm template for the OpenTelemetry Collector to accept metrics from Istio
  2. Configure a Helm template for Istio to communicate with the Collector
  3. Verify Istio deployments
  4. Verify the Collector is receiving metrics

Prerequisites

  • Istio configured as a network mesh on a Kubernetes cluster
  • A running OpenTelemetry Collector v0.56 or later
  • A good understanding of Kubernetes and Helm
  • You’ve configured the Collector to export metric data to Lightstep Observability.

Configure Helm template or the Collector

To configure the Collector, you can use either a Helm template, or configure the Collector directly.

Collector configuration (using a Kubernetes ConfigMap)

The Collector configuration must have gRPC and HTTP ports exposed through the OTLP receiver Collector subcomponent. Istio sends data to the Collector through the gRPC endpoint and HTTP may be necessary for preliminary health checks.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: v1
kind: ConfigMap
metadata:
  name: OpenTelemetry-collector-conf
  namespace: istio-system
  labels:
    app: OpenTelemetry-collector
data:
  OpenTelemetry-collector-config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    service:
      pipelines:
        logs:
          receivers: [otlp]
	   metrics:
		receivers: [otlp]

Configure the Helm template for Istio components

You can configure the Helm template for Istio using one of the following components (or templates):

Configure the Helm template using mesh-config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
  namespace: istio-system
data:
  mesh: |-
    accessLogFile: /dev/stdout
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      proxyMetadata: {}
    enablePrometheusMerge: true
    extensionProviders:
    - name: OpenTelemetry
      envoyOpenTelemetryAls:
        Service: OpenTelemetry-collector.istio-system.svc.cluster.local
        port: 4317
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

Configure the Helm template for namespace injection

The namespaces you want to observe must have Envoy/Istio proxy injection enabled.

1
2
3
4
5
6

kind: Namespace
apiVersion: v1
metadata:
  name: ${YOUR_APPLICATION}
  labels:
    istio-injection: enabled

Configure the Helm template to use the API

Istio has a native telemetry API made available through an Istio CRD (custom resource definition). This allows you to send application-specific telemetry to the Collector.

1
2
3
4
5
6
7
8
9
10
11

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: ${YOUR_APPLICATION}
spec:
  selector:
    matchLabels:
      app: ${SELECTOR}
  accessLogging:
    - providers:
      - name: OpenTelemetry

${SELECTOR} must be included as a Label within your pod, replicationset, or deployment template.

Verify Istio components

Validate Istio is working properly using the command line.

  1. Verify the pods you wish to observe are running. 
    1

    istioctl proxy-status ${POD_TO_OBSERVE) -n=${PODS_NAMESPACE}
  2. Verify that side-car (Envoy) injection is working correctly in the namespace you want to observe. 
    1

    istioctl analyze –namespace ${YOUR_APPLICATION}
  3. View the generated side-car (Envoy) statistics. These are generated on one pod before being sent to the Collector. 
    1

    istioctl experimental envoy-stats pod/${POD_TO_OBSERVE} -n=${POD_NAMESPACE}
  4. Expose web traffic for the application. 
    1

    kubectl logs -l app=${YOUR_APPLICATION} -c istio-proxy
  5. Observe the Envoy container (Istio proxy) logs in the application’s pod.

Verify the Collector

Verify that each Envoy (Istio proxy) container’s web traffic is forwarding to the Collector.

1

kubectl logs -l app=OpenTelemetry-collector -n istio-system

View metrics in Lightstep

Once you have Lightstep ingesting the Istio metrics, you can begin using them to build dashboards in Lightstep Observability.