Create ingestion pipelines

Transform your logs with a basic ingestion pipeline.

The steps below create a pipeline and transformation for dropping INFO logs. For a complete list of transformations and more examples, visit Ingestion pipeline reference.

Before you begin

To complete the steps below, set up the OpenTelemetry (OTel) Collector with a filelog receiver.

While this page uses the OTel Collector, you can also use the Datadog Agent with log ingestion pipelines.

Step 1: Choose a pipeline

Follow these steps to access log ingestion pipelines and select a pipeline:

  1. In Cloud Observability, select Settings > Log ingestion pipelines.
  2. To create a pipeline for logs from the OTel Collector, select the OTLP tab.

Cloud Observability directs all logs from the OTel Collector through the OTLP pipeline. To learn more about the different pipelines, visit Ingestion pipeline reference.

Step 2: Create a transformation

Follow these steps to create a Drop matching transformation:

  1. In the OTLP pipeline, select Add transformation.
  2. Fill out the form:
    • Name - Identify the transformation: Drop INFO logs.
    • Configuration - Specify the transformation type: Drop matching.
    • Filter - Specify the filter in filter-expression format: sev == “INFO”.
  3. To activate your transformation, select Done > Save > Save.

Cloud Observability displays Configuration saved successfully. It may take up to 10 minutes for Cloud Observability to implement your changes.

Step 3: Verify a pipeline

Follow these steps to test your Drop matching transformation:

  1. Paste the logs below into space.log and save the file.

    (space.log is the file you created in Get started with logs.)

    1
2
3
4
5

     INFO Spaceship 'Lightstep' reached orbit around Saturn.
 INFO Astronaut training completed in sector 4.
 ERROR Communication with mission control lost: Signal interference detected.
 ERROR Navigation malfunction: Unable to plot course through asteroid field.
 FATAL Reactor core meltdown. Evacuate the ship immediately!
  2. In Cloud Observability, select Logs.
  3. View three of the five logs you pasted into space.log.

You’re all set! Cloud Observability now drops all incoming logs with the INFO severity level.

To remove the transformation from Cloud Observability, select Settings > Log ingestion pipelines. Next, point to the relevant transformation, select X, and then select Save > Save. You won’t be able to recover the transformation.

Next steps

Now that you know the basics, visit Ingestion pipeline reference to learn about the different transformation types and source-specific pipelines.

See also

Ingestion pipeline reference

Log ingestion pipelines

Log integrations

Updated Apr 24, 2024