Ingest metrics from Airflow

The OpenTelemetry Collector, when configured with a StatsD receiver, provides an integration with Airflow to scrape StatsD metrics. The Collector uses the StatsD Receiver to fetch metrics from the Airflow webserver endpoint. From there, the metrics are processed and exported to Cloud Observability.

To complete the integration, perform the following steps:

Configure Airflow
Configure the Collector to use StatsD receiver
Enable the integration by adding it to a pipeline

Prerequisites

You’ve configured the Collector to export metric data to Cloud Observability.

Configure Airflow reporting

You need to configure Airflow to report metrics to the StatsD receiver’s endpoint. Here’s an example of configuration:

[metrics]
statsd_on = True
statsd_host = otel-collector
statsd_port = 8125
statsd_prefix = airflow

Configure the Collector receiver

In the Collector configuration file, specify the Airflow webserver endpoint as a scrape target.

receivers:
  statsd:
    endpoint: "otel-collector:8125"
    aggregation_interval: 60s
    is_monotonic_counter: true
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "summary"
      - statsd_type: "timing"
        observer_type: "summary"

The OpenTelemetry repository readme provides additional details about StatsD receiver configuration.

Enable the Collector receiver

After the Airflow receiver is configured, enable it by adding it to one or more pipelines as described in the Collector configuration documentation.

Validate metrics are reporting to Cloud Observability

You can validate that metrics are reporting to Cloud Observability on the Metrics details page in Settings.

In Cloud Observability, click Settings > Metric details.
Search for Airflow metric names.

See the Airflow documentation for a detailed description of provided metrics.
If needed, select the metric to edit the description and how the units are displayed in Cloud Observability.

Create a dashboard for the metrics

Use the Cloud Observability Terraform Provider to create a dashboard for the metrics.

Additional resources

For a complete example that’s ready to run, see the Airflow integration in Cloud Observability OpenTelemetry Examples.