The OpenTelemetry Collector provides an integration with Hadoop. The Collector fetches Hadoop metrics from JMX. From there the metrics are processed and exported to Lightstep Observability.
To complete the integration, you will:
- Configure the JMX receiver for the Collector to ingest Hadoop metrics
- Enable the integration by adding it to a pipeline
Prerequisites
- Java 8, to run Hadoop:
- Apache Hadoop v3 or later
- A running Hadoop single cluster instance
- Download OpenTelemetry JMX Metric JAR release version 1.9 or later.
- The path to the
opentelemetry-jmx-metrics.jar
file. - You’ve configured the Collector to export metric data to Lightstep Observability.
Hadoop reporting
The Hadoop JMX receiver periodically fetches metrics using a built-in OpenTelemetry helper utilizing a Groovy script. The script is configured to poll both the JMX Agent and the OpenTelemetry JMX Metric Gatherer operation.
For more details regarding configuring the Hadoop remote connection, see the official Hadoop documentation.
Configure the Collector receiver
In the Collector configuration file, add JMX as a receiver and set the following:
jar_path
: The path to theopentelemetry-jmx-metrics.jar
file.endpoint
: The hostname/IP address and port or unix socket file path of the Hadoop instancetarget_system
: The metrics to collect. Valid values for this integration arejvm
andhadoop
1
2
3
4
5
6
receivers:
jmx:
jar_path: /opt/opentelemetry-jmx-metrics.jar
endpoint: "localhost:9999"
target_system: jvm,hadoop
The OpenTelemetry repo’s readme provides additional details about JMX configuration.
Enable the Collector receiver
Once the Hadoop receiver is configured, enable it by adding it to one or more pipelines as described in the Collector configuration documentation.
Validate metrics are reporting to Lightstep
You can validate that metrics are reporting to Lightstep on the Metrics details page in Project settings.
-
In Lightstep, click Project settings > Metric details.
-
Search for Hadoop metric names.
See the OpenTelemetry repo for a complete list of emitted metrics.
- If needed, click on the metric to edit the description and how the units are displayed in Lightstep.
Create a dashboard for the metrics
Use the Lightstep Terraform Provider to create a dashboard for the metrics.
Additional resources
For a more complete example that’s ready to run, see the Hadoop integration in Lightstep OpenTelemetry Examples.