Hashicorp Nomad metrics

The OpenTelemetry Collector, when configured with a Prometheus receiver, provides an integration with Hashicorp Nomad to scrape Prometheus metrics. The Collector uses the Prometheus Receiver to fetch metrics from the configured path in the Nomad configuration file. From there, the metrics are processed and exported to Cloud Observability.

To complete the integration, you will:

Configure Hashicorp Nomad to use the Prometheus exporter.
Configure the Collector to use the Hashicorp Nomad endpoint as a scrape target for the Prometheus receiver.
Enable the integration by adding it to a pipeline.

Prerequisites

Hashicorp Nomad v1.3.3 or later.
You’ve configured the Collector to export metric data to Cloud Observability.

Configure Hashicorp Nomad reporting

To enable Prometheus metrics, you need to enable telemetry on the Nomad operator. In the ./nomad/config/local.json file, add the following:

"telemetry": {
  "publish_allocation_metrics": true,
  "publish_node_metrics": true,
  "collection_interval": "1s",
  "disable_hostname": true,
  "prometheus_metrics": true
}

Setting prometheus_metrics to true allows the agent to make Prometheus metrics available at /v1/metrics?format=prometheus. You can test that metrics are enabled using the following:

curl -s 'localhost:4646/v1/metrics?format=prometheus'

For more details about using Hashicorp Nomad with the Prometheus exporter, see the Nomad documentation.

Configure the Collector receiver

In the Collector configuration file, configure the Prometheus receiver to use the Hashicorp Nomad Prometheus endpoint as a scrape target.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'nomad-server'
          scrape_interval: 10s
          scrape_timeout: 5s
          metrics_path: '/v1/metrics'
          params:
            format: ['prometheus']
          static_configs:
            - targets: ['localhost:4646']

The OpenTelemetry repo’s README provides additional details about Prometheus receiver configuration.

More details about the Prometheus scrape configuration can be found here.

Enable the Collector receiver

Once the Hashicorp Nomad receiver is configured, enable it by adding it to one or more pipelines as described in the Collector configuration documentation.

Validate metrics are reporting to Cloud Observability

You can validate that metrics are reporting to Cloud Observability on the Metrics details page in Settings.

In Cloud Observability, click Settings > Metric details.
Search for Hashicorp Nomad metric names.

See the Nomad docs for a complete list of emitted metrics.
If needed, click on the metric to edit the description and how the units are displayed in Cloud Observability.

Create a dashboard for the metrics

Use the Cloud Observability Terraform Provider to create a dashboard for the metrics.

Additional resources

For a more complete example that’s ready to run, see the Hashicorp Nomad integration in Cloud Observability OpenTelemetry Examples.