The OpenTelemetry Collector, when configured with a StatsD receiver, provides an integration with Airflow to scrape StatsD metrics. The Collector uses the StatsD Receiver to fetch metrics from the Airflow webserver endpoint. From there, the metrics are processed and exported to Cloud Observability.
To complete the integration, perform the following steps:
You need to configure Airflow to report metrics to the StatsD receiver’s endpoint. Here’s an example of configuration:
1 2 3 4 5 [metrics] statsd_on = True statsd_host = otel-collector statsd_port = 8125 statsd_prefix = airflow
In the Collector configuration file, specify the Airflow webserver endpoint as a scrape target.
1 2 3 4 5 6 7 8 9 10 receivers: statsd: endpoint: "otel-collector:8125" aggregation_interval: 60s is_monotonic_counter: true timer_histogram_mapping: - statsd_type: "histogram" observer_type: "summary" - statsd_type: "timing" observer_type: "summary"
The OpenTelemetry repository readme provides additional details about StatsD receiver configuration.
After the Airflow receiver is configured, enable it by adding it to one or more pipelines as described in the Collector configuration documentation.
You can validate that metrics are reporting to Cloud Observability on the Metrics details page in Project settings.
In Cloud Observability, click Project settings > Metric details.
Search for Airflow metric names.
See the Airflow documentation for a detailed description of provided metrics.
If needed, select the metric to edit the description and how the units are displayed in Cloud Observability.
Use the Cloud Observability Terraform Provider to create a dashboard for the metrics.
Updated May 30, 2023