The OpenTelemetry Collector, when configured with a Prometheus receiver, provides an integration with Hashicorp Nomad to scrape Prometheus metrics. The Collector uses the Prometheus Receiver to fetch metrics from the configured path in the Nomad configuration file. From there, the metrics are processed and exported to Lightstep Observability.

To complete the integration, you will:

  • Configure Hashicorp Nomad to use the Prometheus exporter.
  • Configure the Collector to use the Hashicorp Nomad endpoint as a scrape target for the Prometheus receiver.
  • Enable the integration by adding it to a pipeline.

Prerequisites

  • Hashicorp Nomad v1.3.3 or later.
  • You’ve configured the Collector to export metric data to Lightstep Observability.

Configure Hashicorp Nomad reporting

To enable Prometheus metrics, you need to enable telemetry on the Nomad operator. In the ./nomad/config/local.json file, add the following:

1
2
3
4
5
6
7
"telemetry": {
  "publish_allocation_metrics": true,
  "publish_node_metrics": true,
  "collection_interval": "1s",
  "disable_hostname": true,
  "prometheus_metrics": true
}

Setting prometheus_metrics to true allows the agent to make Prometheus metrics available at /v1/metrics?format=prometheus. You can test that metrics are enabled using the following:

1
curl -s 'localhost:4646/v1/metrics?format=prometheus'

For more details about using Hashicorp Nomad with the Prometheus exporter, see the Nomad documentation.

Configure the Collector receiver

In the Collector configuration file, configure the Prometheus receiver to use the Hashicorp Nomad Prometheus endpoint as a scrape target.

1
2
3
4
5
6
7
8
9
10
11
12
13
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'nomad-server'
          scrape_interval: 10s
          scrape_timeout: 5s
          metrics_path: '/v1/metrics'
          params:
            format: ['prometheus']
          static_configs:
            - targets: ['localhost:4646']

The OpenTelemetry repo’s README provides additional details about Prometheus receiver configuration.

More details about the Prometheus scrape configuration can be found here.

Enable the Collector receiver

Once the Hashicorp Nomad receiver is configured, enable it by adding it to one or more pipelines as described in the Collector configuration documentation.

Validate metrics are reporting to Lightstep

You can validate that the metrics are reporting to Lightstep from the Metric details section of the Project Settings page.

  1. Open the Project settings page and scroll to the Metric details section.

  2. Search for Hashicorp Nomad metric names. Search for metric

    See the Nomad docs for a complete list of emitted metrics.

  3. If needed, click on the metric to edit the description and how the units are displayed in Lightstep.

Create a dashboard for the metrics

Use the Lightstep Terraform Provider to create a dashboard for the metrics.

Additional resources

For a more complete example that’s ready to run, see the Hashicorp Nomad integration in Lightstep OpenTelemetry Examples.