Once you start sending data to Lightstep Satellites, it is recommended practice to monitor their performance. The Satellites expose statsd metrics that can be sent to Prometheus, a popular open source systems monitoring and alerting toolkit.

In this tutorial, we will setup a Lightstep Satellite and Prometheus in a local Kubernetes cluster using Docker for Desktop on Mac.

Prerequisites

We will run this example in in Kubernetes. Before proceeding make sure you have:

  • kubectl
    • Install using gcloud components install kubectl
  • Access to a Kubernetes cluster configured with gcloud auth init OR Local Kubernetes cluster deployment:
  • Lightstep satellite key
    • This key is required to run a satellite and you can create one in Lightstep’s admin settings

Directory structure

1
2
3
4
5
6
7
8
9
10

satellite_prometheus_example /
---- satellite /
-------- secret.yaml
-------- configmap.yaml
-------- deployment.yaml
-------- service.yaml
---- prometheus /
-------- configmap.yaml
-------- deployment.yaml
-------- service.yaml

Setup Lightstep Satellite

1. Create a Lightstep secret

We will use this to run the satellite.

1
2
3
4
5
6
7
8
9

apiVersion: v1
kind: Secret
metadata:
  name: lightstep-credentials
  labels:
    app: lightstep-satellite
type: Opaque
stringData:
  satelliteKey: "<SATELLITE_KEY_HERE>"

2. Create a ConfigMap for statsd

This will map the statsd metrics reporting from the satellite to a Prometheus friendly format.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

apiVersion: v1
kind: ConfigMap
metadata:
  name: statsd-cm
  labels:
    name: statsd-cm
    app: lightstep-satellite
data:
  statsd_mapping.yml: |
    mappings:
      - match: "lightstep.client.spans.dropped.*"
        name: "lightstep_client_spans_dropped"
        labels:
          project: "$1"
      - match: "lightstep.satellite.access_tokens.invalid.*"
        name: "lightstep_satellite_access_tokens_invalid"
        labels:
          project: "$1"
      - match: "lightstep.satellite.spans.received.*"
        name: "lightstep_satellite_spans_received"
        labels:
          project: "$1"
      - match: "lightstep.satellite.spans.dropped.*"
        name: "lightstep_satellite_spans_dropped"
        labels:
          project: "$1"
      - match: "lightstep.satellite.index.queue.length.*"
        name: "lightstep_satellite_index_queue_length"
        labels:
          project: "$1"
      - match: "lightstep.satellite.index.queue.bytes.*"
        name: "lightstep_satellite_index_queue_bytes"
        labels:
          project: "$1"
      - match: "lightstep.satellite.spans.indexed.*"
        name: "lightstep_satellite_spans_indexed"
        labels:
          project: "$1"
      - match: "lightstep.satellite.current.recall.seconds.*"
        name: "lightstep_satellite_current_recall_seconds"
        labels:
          project: "$1"

3. Create a Deployment for the Satellite

In this deployment, we will run a sidecar container for the statsd-exporter which will listen to metrics emitted by the satellite and transform them using the ConfigMap above.

It’s important to note that in this configuration, each Satellite you run will have a statsd-exporter sidecar alongside to report its metrics.

The following example is used for testing, refer to the Satellite Configuration Parameters for more details on options available for the satellite.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lightstep-satellite-deployment
  labels:
    app: lightstep-satellite
spec:
  replicas: 1 # only one for testing
  selector:
    matchLabels:
      app: lightstep-satellite
  template:
    metadata:
      labels:
        app: lightstep-satellite
    spec:
      containers:
        - image: prom/statsd-exporter:latest
          name: statsd-exporter
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: statsd-volume # mount the config map
              mountPath: /tmp/
          args: ["--statsd.mapping-config=/tmp/statsd_mapping.yml"]
          ports:
            - containerPort: 9102 # to expose metrics to Prometheus
              name: metrics
              protocol: TCP
            - containerPort: 9125 # to receive metrics from Satellite
              name: statsd
              protocol: UDP
          resources:
            limits:
              cpu: 100m
              memory: 100Mi
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /metrics
              port: metrics
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
        - image: lightstep/collector:latest
          name: lightstep-satellite
          imagePullPolicy: "IfNotPresent"
          env:
            - name: COLLECTOR_SATELLITE_KEY
              valueFrom:
                secretKeyRef:
                  key: satelliteKey
                  name: lightstep-credentials
            - name: COLLECTOR_POOL
              value: k8s-test-pool
            - name: COLLECTOR_REPORTER_BYTES_PER_PROJECT
              value: "100000000"
            - name: COLLECTOR_DIAGNOSTIC_PORT
              value: "8000"
            - name: COLLECTOR_ADMIN_PLAIN_PORT
              value: "8180"
            - name: COLLECTOR_HTTP_PLAIN_PORT
              value: "8181"
            - name: COLLECTOR_STATSD_HOST
              value: "127.0.0.1"
            - name: COLLECTOR_STATSD_PORT # report to the sidecar
              value: "9125"
            - name: COLLECTOR_STATSD_EXPORT_STATSD
              value: "true"
            - name: COLLECTOR_STATSD_PREFIX
              value: "lightstep"
          ports:
            - containerPort: 8000
              name: diagnostics
              protocol: TCP
            - containerPort: 8180
              name: admin
              protocol: TCP
            - containerPort: 8181 # receive traffic from tracers
              name: http
              protocol: TCP
          resources:
            limits:
              memory: 1Gi # just for testing, use 16Gi in prod
              cpu: 1 # just for testing, use 2 CPU in prod
            requests:
              memory: 1Gi # just for testing, use 16Gi in prod
              cpu: 1 # just for testing, use 2 CPU in prod
          readinessProbe:
            failureThreshold: 10
            httpGet:
              path: /_ready
              port: admin
              scheme: HTTP
            timeoutSeconds: 15
            initialDelaySeconds: 5
            periodSeconds: 20
            successThreshold: 1
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /_live
              port: admin
              scheme: HTTP
            timeoutSeconds: 15
            initialDelaySeconds: 30
            periodSeconds: 20
            successThreshold: 1
      volumes:
        - name: statsd-volume # mount the config map
          configMap:
            defaultMode: 420
            name: statsd-cm

4. Expose the Service for the satellite and statsd-exporter

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

apiVersion: v1
kind: Service
metadata:
  name: lightstep-satellite
  labels:
    app: lightstep-satellite
spec:
  type: LoadBalancer
  selector:
    app: lightstep-satellite
  ports:
    - name: http # send test data to satellite here
      port: 8181
      targetPort: 8181

---
apiVersion: v1
kind: Service
metadata:
  name: lightstep-satellite-statsd
  labels:
    app: lightstep-satellite
spec:
  type: LoadBalancer
  selector:
    app: lightstep-satellite
  ports:
    - name: metrics-prom # Prometheus will scrape this
      port: 9102
      targetPort: 9102

5. Run the satellite

1

$ kubectl apply -f satellite/**.yaml

6. Report data to Lightstep

Use one of our guides to report telemetry data to the satellite at localhost:8181

7. Verify statsd metrics

Go to localhost:9102/metrics and verify that lightstep_* metrics are showing up.

Now, we are ready to send these metrics to Prometheus.

Setup Prometheus

1. Setup ConfigMap for Prometheus

In our test we are only running one Satellite, but in a production environment each satellite will be reporting individual metrics via a statsd-exporter sidecar. We can setup a ConfigMap to differentiate between the Satellites based on their node metadata.

We only need one Prometheus instance as it is capable of scraping multiple satellites.

The following ConfigMap assumes that Prometheus runs in the same cluster and namespace as the satellites. For more advanced configurations, refer to Prometheus docs.

Make sure to set the app label and metrics port name to the correct config so that Prometheus can discover the endpoints for scraping.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

apiVersion: v1
metadata:
  name: prometheus-cm
  labels:
    name: prometheus-cm
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s

    scrape_configs:
      - job_name: lightstep-satellite-metrics
        honor_labels: true
        honor_timestamps: true
        scrape_interval: 1m
        scrape_timeout: 59s
        metrics_path: /metrics
        scheme: http
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - default # specify name space
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_label_app]
          separator: ;
          regex: lightstep-satellite # Set your app label
          replacement: $1
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_port_name]
          separator: ;
          regex: metrics-prom # Set your port name
          replacement: $1
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
          separator: ;
          regex: Node;(.*)
          target_label: node
          replacement: ${1}
          action: replace
        - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
          separator: ;
          regex: Pod;(.*)
          target_label: pod
          replacement: ${1}
          action: replace
        - source_labels: [__meta_kubernetes_namespace]
          separator: ;
          regex: (.*)
          target_label: namespace
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_service_name]
          separator: ;
          regex: (.*)
          target_label: service
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_pod_name]
          separator: ;
          regex: (.*)
          target_label: pod
          replacement: $1
          action: replace
kind: ConfigMap

2. Setup a Deployment for Prometheus

We will also enable Prometheus to run a Web UI on port 9090 for testing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  labels:
    app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:latest
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: prometheus-vol # mount the configmap
              mountPath: /etc/prometheus/
          command:
            - "prometheus"
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--web.listen-address=:9090"
            - "--log.level=debug"
          ports:
            - containerPort: 9090
      volumes:
        - name: prometheus-vol # mount the configmap
          configMap:
            defaultMode: 420
            name: prometheus-cm

3. Expose the Prometheus UI as a Service

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
spec:
  type: LoadBalancer
  selector:
    app: prometheus
  ports:
    - name: http
      port: 9090
      targetPort: 9090

4. Run Prometheus

1

$ kubectl apply -f prometheus/**.yaml

5. Verify metrics in Prometheus

Navigate to localhost:9090/targets to test that Prometheus has registered one satellite for scraping the metrics (multiple if you chose to run multiple replicas). This may take a few seconds.

prometheus-targets

Navigate to localhost:9090/graph and choose one of the lightstep_* metrics to graph. If you are running multiple replicas, you will observe an individual time series in the graph for each satellite.

prometheus-graph

Next Steps

If you are an experienced Prometheus user, you may want to send the metrics to Grafana directly for visualization, aggregation, and further calculations and also set up monitors on the important metrics in Prometheus Alertmanager.