Istio emits metrics through OpenTelemetry-compatible endpoints using each pod’s Envoy (or Istio proxy), which can be forwarded to the OpenTelemetry Collector by defining attributes in Istio’s mesh-config.yaml
template. From there the metrics are processed and exported to Lightstep Observability. The process of setting up Istio observability using OpenTelemetry is broken up into four parts:
- Configure a Helm template for the OpenTelemetry Collector to accept metrics from Istio
- Configure a Helm template for Istio to communicate with the Collector
- Verify Istio deployments
- Verify the Collector is receiving metrics
Prerequisites
- Istio configured as a network mesh on a Kubernetes cluster
- A running OpenTelemetry Collector v0.56 or later
- A good understanding of Kubernetes and Helm
- You’ve configured the Collector to export metric data to Lightstep Observability.
Configure Helm template or the Collector
To configure the Collector, you can use either a Helm template, or configure the Collector directly.
Collector configuration (using a Kubernetes ConfigMap)
The Collector configuration must have gRPC and HTTP ports exposed through the OTLP receiver Collector subcomponent. Istio sends data to the Collector through the gRPC endpoint and HTTP may be necessary for preliminary health checks.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: v1
kind: ConfigMap
metadata:
name: OpenTelemetry-collector-conf
namespace: istio-system
labels:
app: OpenTelemetry-collector
data:
OpenTelemetry-collector-config: |
receivers:
otlp:
protocols:
grpc:
http:
service:
pipelines:
logs:
receivers: [otlp]
metrics:
receivers: [otlp]
Configure the Helm template for Istio components
You can configure the Helm template for Istio using one of the following components (or templates):
- Istio’s MeshConfig
- Namespace injection
- Istio’s telemetry API to configure application-specific telemetry
Configure the Helm template using mesh-config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: v1
kind: ConfigMap
metadata:
name: istio
namespace: istio-system
data:
mesh: |-
accessLogFile: /dev/stdout
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
proxyMetadata: {}
enablePrometheusMerge: true
extensionProviders:
- name: OpenTelemetry
envoyOpenTelemetryAls:
Service: OpenTelemetry-collector.istio-system.svc.cluster.local
port: 4317
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
Configure the Helm template for namespace injection
The namespaces you want to observe must have Envoy/Istio proxy injection enabled.
1
2
3
4
5
6
kind: Namespace
apiVersion: v1
metadata:
name: ${YOUR_APPLICATION}
labels:
istio-injection: enabled
Configure the Helm template to use the API
Istio has a native telemetry API made available through an Istio CRD (custom resource definition). This allows you to send application-specific telemetry to the Collector.
1
2
3
4
5
6
7
8
9
10
11
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: ${YOUR_APPLICATION}
spec:
selector:
matchLabels:
app: ${SELECTOR}
accessLogging:
- providers:
- name: OpenTelemetry
${SELECTOR}
must be included as a Label within your pod, replicationset, or deployment template.
Verify Istio components
Validate Istio is working properly using the command line.
- Verify the pods you wish to observe are running.
1
istioctl proxy-status ${POD_TO_OBSERVE) -n=${PODS_NAMESPACE}
- Verify that side-car (Envoy) injection is working correctly in the namespace you want to observe.
1
istioctl analyze –namespace ${YOUR_APPLICATION}
- View the generated side-car (Envoy) statistics. These are generated on one pod before being sent to the Collector.
1
istioctl experimental envoy-stats pod/${POD_TO_OBSERVE} -n=${POD_NAMESPACE}
- Expose web traffic for the application.
1
kubectl logs -l app=${YOUR_APPLICATION} -c istio-proxy
- Observe the Envoy container (Istio proxy) logs in the application’s pod.
Verify the Collector
Verify that each Envoy (Istio proxy) container’s web traffic is forwarding to the Collector.
1
kubectl logs -l app=OpenTelemetry-collector -n istio-system
View metrics in Lightstep
Once you have Lightstep ingesting the Istio metrics, you can begin using them to build dashboards in Lightstep Observability.