Ingest metrics from Fluentd

The OpenTelemetry Collector, when configured with a Prometheus receiver, provides an integration with Fluentd to scrape Prometheus metrics. The Collector uses the Prometheus Receiver to fetch metrics from the Fluentd metrics endpoint. From there, the metrics are processed and exported to Cloud Observability.

To complete the integration, perform the following steps:

  • Configure the Fluentd to enable export of the required metrics
  • Configure the Collector to use the Fluentd Prometheus plugin endpoint as a scrape target for the Prometheus receiver
  • Enable the integration by adding it to a pipeline

Prerequisites

  • Fluentd running in a Kubernetes environment.
  • You’ve configured the Collector to export metric data to Cloud Observability.

Configure Fluentd reporting

Fluentd requires configuration to expose metrics in Prometheus endpoint. See the Fluentd example for an integration. And here’s the Fluentd documentation for the reference.

Configure the Collector receiver

In the Collector configuration file, specify the Fluentd Prometheus plugin endpoint as a scrape target.

1
2
3
4
5
6
7

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-fluentd
          static_configs:
            - targets: [fluentd:24224]

The OpenTelemetry repository readme provides additional details about Prometheus receiver configuration.

Complete information about the Prometheus scrape configuration can be found in the Prometheus documentation.

Enable the Collector receiver

After the Fluentd receiver is configured, enable it by adding it to one or more pipelines as described in the Collector configuration documentation.

Validate metrics are reporting to Cloud Observability

You can validate that metrics are reporting to Cloud Observability on the Metrics details page in Settings.

  1. In Cloud Observability, click Settings > Metric details.

  2. Search for Fluentd metric names. Arrow points to the metrics search box.

    See the Fluentd documentation for a detailed description of provided metrics.

  3. If needed, select the metric to edit the description and how the units are displayed in Cloud Observability.

Create a dashboard for the metrics

Use the Cloud Observability Terraform Provider to create a dashboard for the metrics.

Additional resources

  • For a complete example that’s ready to run, see the Fluentd integration in Cloud Observability OpenTelemetry Examples.

See also

Create and manage dashboards

Create alerts

Updated May 3, 2023