The OpenTelemetry Collector provides an integration with Hadoop. The Collector fetches Hadoop metrics from JMX. From there the metrics are processed and exported to Cloud Observability.
To complete the integration, you will:
The Hadoop JMX receiver periodically fetches metrics using a built-in OpenTelemetry helper utilizing a Groovy script. The script is configured to poll both the JMX Agent and the OpenTelemetry JMX Metric Gatherer operation.
For more details regarding configuring the Hadoop remote connection, see the official Hadoop documentation.
In the Collector configuration file, add JMX as a receiver and set the following:
jar_path: The path to the
endpoint: The hostname/IP address and port or unix socket file path of the Hadoop instance
target_system: The metrics to collect. Valid values for this integration are
1 2 3 4 5 6 receivers: jmx: jar_path: /opt/opentelemetry-jmx-metrics.jar endpoint: "localhost:9999" target_system: jvm,hadoop
The OpenTelemetry repo’s readme provides additional details about JMX configuration.
Once the Hadoop receiver is configured, enable it by adding it to one or more pipelines as described in the Collector configuration documentation.
You can validate that metrics are reporting to Cloud Observability on the Metrics details page in Project settings.
In Cloud Observability, click Project settings > Metric details.
Search for Hadoop metric names.
See the OpenTelemetry repo for a complete list of emitted metrics.
Use the Cloud Observability Terraform Provider to create a dashboard for the metrics.
For a more complete example that’s ready to run, see the Hadoop integration in Cloud Observability OpenTelemetry Examples.
Updated Dec 1, 2022