Lightstep Observability works with many service meshes, and Lightstep Observability + Istio is an easy way to get quick visibility into service performance and availability from the perspective of the service mesh.

Istio Proxy, based on Envoy, uses OpenTracing (OT) to start new traces and join existing traces, based on HTTP request headers. After configuring Istio, it sends all span data it generates to Microsatellites. You start seeing traces and spans in Lightstep Observability immediately - no additional code-level instrumentation is needed. By default, all HTTP requests are captured (to see end-to-end traces, your code needs to forward OT headers even if it does not join the traces).

If you only want to collect tracing spans directly from Istio (and not add specific instrumentation directly to your code), then you don’t need to configure any tracers, as long as your services forward the HTTP headers generated by traces.

These steps use Kubernetes and Helm. You’ll learn how to install Istio and configure it to work with Lightstep Observability in that environment.

After that, you can install Istio’s Bookinfo sample application and see example spans immediately in Lightstep Observability.

Prerequisites

You’ll need the following to configure Istio:

  • kubectl configured with the appropriate access for your cluster. Refer to Istio’s Platform Setup documentation if necessary
  • Helm (v3+).
  • The access token for your Lightstep Observability project.
  • Satellite pool address in the format <host>:<port>. You find this in your configuration file. For reporting to Public Microsatellites, use ingest.lightstep.com:443
  • (Public Microsatellites only) Download the cacert.pem file at the end of this page to a local directory.

Install Microsatellites On-Premise (optional)

There are many ways of setting up a private satellite pool. This step uses helm to set up a single satellite in your cluster to receive data from Istio. If you skip this step, you can use the Lightstep Observability Public Microsatellites. However, for a production deployment we recommend that you use Private Satellite pools. You will need a Satellite Key for this step.

1
2
3
4
5
6
7
8
9

# Add the Lightstep Observability helm repo
helm repo add lightstepsatellite https://lightstep.github.io/lightstep-satellite-helm-chart/
# Install the Satellite in your default cluster
helm install satellite lightstepsatellite/lightstep \
   --version 1.1.4 \
   --set lightstep.satelliteKey=$SATELLITE_KEY

# Get the satellite service name, this will be used later in Istio config
export SATELLITE_ENDPOINT=$(kubectl get services -n default -l "app.kubernetes.io/name=lightstep" -o jsonpath="{.items[0].metadata.name}").default.svc:8184

The default deployment sets up Microsatellites without TLS. If you want to use TLS, confirm that you configured your Microsatellites to use TLS certs and that they expose a secure gRPC port.

Download Istio

Istio 1.8 has been tested with these Kubernetes releases: 1.16, 1.17, 1.18, 1.19. Read More

1
2
3
4

export ISTIO_VERSION=1.8.1
wget https://github.com/istio/istio/releases/download/$ISTIO_VERSION/istio-$ISTIO_VERSION-osx.tar.gz
tar zxvf istio-$ISTIO_VERSION-osx.tar.gz
cd istio-$ISTIO_VERSION

Install and configure Istio

All commands should be executed from the istio-$ISTIO_VERSION directory.

1
2
3
4
5
6
7
8

# 1. Create a namespace for Istio.
kubectl create namespace istio-system

# 2. Install Base CRDs
helm install -n istio-system istio-base manifests/charts/base

# 3. Verify Istio CRDs were committed to Kubernetes api-server
kubectl get crds | grep 'istio.io\|certmanager.k8s.io' | wc -l

Start tabs

Public Microsatellites (TLS)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# Public Microsatellites only: Add the CA cert used for signing the satellite's identity as a secret. This is the `cacert.pem` file you downloaded in the Prerequisites.
kubectl create secret generic lightstep.cacert --from-file=cacert.pem

# 4. Set variables
export ACCESS_TOKEN=[YOUR ACCESS TOKEN]
export SATELLITE_ENDPOINT="ingest.lightstep.com:443"

# 5. Set up istiod
helm template \
   --set global.tag=$ISTIO_VERSION \
   --set pilot.traceSampling=100 \
   --set global.proxy.tracer="lightstep" \
   --set meshConfig.defaultConfig.tracing.tlsSettings.mode="SIMPLE" \
   --set meshConfig.defaultConfig.tracing.tlsSettings.caCertificates="/etc/lightstep/cacert.pem" \
   --set global.tracer.lightstep.address=$SATELLITE_ENDPOINT \
   --set global.tracer.lightstep.accessToken=$ACCESS_TOKEN \
   manifests/charts/istio-control/istio-discovery \
   --name-template istio \
   --namespace istio-system > ./my-istio.yaml

On-Premise Microsatellites (non-TLS)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# NOTE: If you are using TLS for your on-premise pool,
# follow the config in the public satellite pool, except
# use your own cert and your own pool host:port

# 4. Set variables
export ACCESS_TOKEN=[YOUR ACCESS TOKEN]
export SATELLITE_ENDPOINT=[YOUR SATELLITE HOST:PORT]

# 5. Set up istiod
helm template \
   --set global.tag=$ISTIO_VERSION \
   --set pilot.traceSampling=100 \
   --set global.proxy.tracer="lightstep" \
   --set global.tracer.lightstep.address=$SATELLITE_ENDPOINT \
   --set global.tracer.lightstep.accessToken=$ACCESS_TOKEN \
   manifests/charts/istio-control/istio-discovery \
   --name-template istio \
   --namespace istio-system > ./my-istio.yaml

End code tabs

1
2
3
4
5
6
7
8
9
10
11

# 6. Install istiod
kubectl apply -f ./my-istio.yaml

# 7. Instal istio-ingress
helm install --namespace istio-system istio-ingress manifests/charts/gateways/istio-ingress --set global.tag=$ISTIO_VERSION

# 8. Install istio-egress
helm install --namespace istio-system istio-egress manifests/charts/gateways/istio-egress --set global.tag=$ISTIO_VERSION

# 9. Check pods are running
kubectl get pods -n istio-system

Istio is now configured to work with Lightstep Observability!

To see example traces, follow the next section to install Istio’s Bookfinder sample app.

Install the Bookfinder App and Send Traces

Istio’s Bookfinder app uses four microservices to display information about a book.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# 1. Enable sidecar injection in the default namespace.
kubectl label namespace default istio-injection=enabled

# 2. Deploy the application to Kubernetes.
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

# 3. Create the ingress gateway for the application.
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

# 4. Set the `INGRESS_HOST` and `INGRESS_PORT` variables for accessing the gateway.
export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}')

# 5. Set the GATEWAY_URL.
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

Open the app by visiting http://$GATEWAY_URL/productpage in your browser. 

Click the browser’s refresh button a few times to create some spans.

You now have sent data to Lightstep Observability created by the refresh requests (or anything else you’ve clicked in the browser).

View traces in Lightstep Observability

Now that you have some trace data created, go into Lightstep Observability to see it!

If you open Lightstep Observability and don’t see any data, it’s likely because too much time has gone by since you last refreshed the browser (causing a request). Refresh the Bookfinder window a few times and then go back into Lightstep Observability.

  1. Open Lightstep Observability. You’ll see the three services from the Bookfinder app listed on the Services page.

  2. Click the Explorer tab to view the generated latency histogram.

    You can see all the requests broken down by service and operation.

  3. Click on the first row in the table of example traces below the latency histogram to see span details corresponding to that operation. Each span corresponds to a Bookinfo service invoked during the execution of a /productpage request.

    Each RPC request results in two spans - one for the client and one for the server. For example, the call from productpage to reviews starts with the reviews.default.svc.cluster.local:9080/* operation and the productpage.default: proxy client service. This service represents the client-side span of the call. It took 21.7 ms.

    The second span for the reviews.default.svc.cluster.local:9080/* operation and reviews.default: proxy server service is a child of the first span and represents the server-side span of the call. It took 20.4 ms.

    On the right, you can see the tags that Istio provides by default.

    Now you can deploy your own app and generate your own trace data. Here are instructions for uninstalling the Bookfinder app.

cacert.pem File

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

DST Root CA X3
==============
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/MSQwIgYDVQQK
ExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMTDkRTVCBSb290IENBIFgzMB4X
DTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVowPzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1
cmUgVHJ1c3QgQ28uMRcwFQYDVQQDEw5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmT
rE4Orz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEqOLl5CjH9
UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9bxiqKqy69cK3FCxolkHRy
xXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40d
utolucbY38EVAjqr2m7xPi71XAicPNaDaeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0T
AQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQ
MA0GCSqGSIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69ikug
dB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXrAvHRAosZy5Q6XkjE
GB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZzR8srzJmwN0jP41ZL9c8PDHIyh8bw
RLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubS
fZGL+T0yjWW06XyxV3bqxbYoOb8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----