Ingestion pipeline reference

Learn about the different transformation types and pipeline order.

Transformation types

Parse and process incoming logs with transformations.

Drop matching

Block logs based on certain conditions.

Inputs

  • Name - Identify the transformation.
  • Configuration - Select the transformation type: Drop matching.

  • Filter - Specify the filter in filter-expression format.

    Cloud Observability doesn’t ingest logs that match the filter. All other logs continue through the pipeline. Log ingestion pipelines support every filter expression except phrase_match.

Examples

Drop INFO logs

Block all INFO logs with this Drop matching transformation:

  • Name: Drop INFO logs
  • Configuration: Drop matching
  • Filter: sev == "INFO"

Logs before:

1
2

INFO Spaceship 'Lightstep' reached orbit around Saturn.
ERROR Navigation malfunction: Unable to plot course through asteroid field.

Logs after:

1

ERROR Navigation malfunction: Unable to plot course through asteroid field.

Expandable end

Drop INFO logs for a service

Block INFO logs for the StellarNav service with this Drop matching transformation:

  • Name: Drop INFO logs for StellarNav
  • Configuration: Drop matching
  • Filter: sev == "INFO" && service == "StellarNav"

Logs before:

1
2

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn.
ERROR StellarNav Navigation malfunction: Unable to plot course through asteroid field.

Logs after:

1

ERROR StellarNav Navigation malfunction: Unable to plot course through asteroid field.

Expandable end

Keep matching

Ingest logs based on certain conditions.

Inputs

  • Name - Identify the transformation.
  • Configuration - Select the transformation type: Keep matching.

  • Filter - Specify the filter in filter-expression format.

    Cloud Observability only ingests logs that match the filter. It drops all other logs. Log ingestion pipelines support every filter expression except phrase_match.

Examples

Keep logs where body contains “sect 8”

Only keep sect 8 logs with this Keep matching transformation:

  • Name: Keep sector 8 logs
  • Configuration: Keep matching
  • Filter: contains(body, "sect 8")

Logs before:

1
2
3

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn in sect 8.
ERROR Lightstep Navigation malfunction in sect 4: Unable to plot course through asteroid field.
FATAL Juno core meltdown. Evacuate the ship immediately! Everything in sect 8 is compromised.

Logs after:

1
2

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn in sect 8.
FATAL Juno core meltdown. Evacuate the ship immediately! Everything in sect 8 is compromised.

Expandable end

Parse JSON

Parse and restructure JSON logs into logical attributes.

Inputs

  • Name - Identify the transformation.
  • Configuration - Select the transformation type: Parse JSON.
  • Target field - Specify the name of the JSON field you want to parse, for example, body.

Optional Parse JSON inputs

These inputs are optional for Parse JSON transformations:

Drop target field

Toggle this option to remove the target field once it’s parsed.

Filter

Specify the filter in filter-expression format. Log ingestion pipelines support every filter expression except phrase_match.

Cloud Observability only parses logs that match the filter. Other logs are unaffected by the transformation.

Fields to promote

Extract specific JSON fields and promote them to the top level of the log.

For example, enter action to promote the action field in the target body field. If you specify a prefix in Prefix new fields (see below), Cloud Observability adds the prefix to action. Select Add field to promote multiple fields.

Prefix new fields

Add context to the front of new top-level field names.

For example, station results in field names like station.destination and station.launch_pad.

JSON max depth

Specify the maximum nesting level for Cloud Observability to parse.

For example, if JSON max depth is 3, Cloud Observability only parses up to 3 levels in the JSON structure.

Expandable end

Examples

Parse body

Parse the body field with this Parse JSON transformation:

  • Name: Parse body
  • Configuration: Parse JSON
  • Target field: body

Log before:

1

{"severity": "INFO", "message": {"action": "Spacecraft launched", "details": {"launch_pad": "LC-39A", "destination": "Mars"}}}

Log after:

1

{"severity": "INFO", "message.action": "Spacecraft launched", "message.details.launch_pad": "LC-39A", "message.details.destination": "Mars"}

Expandable end

Parse body and add context

Parse the body field and add information with this Parse JSON transformation:

  • Name: Parse body
  • Configuration: Parse JSON
  • Target field: body
  • Prefix new fields: context

Log before:

1

{"severity": "INFO", "message": {"action": "Spacecraft launched", "details": {"launch_pad": "LC-39A", "destination": "Mars"}}}

Log after:

1

{"context.severity": "INFO", "context.message.action": "Spacecraft launched", "context.message.details.launch_pad": "LC-39A", "context.message.details.destination": "Mars"}

Expandable end

Promote specific fields

Parse body and promote its message field with the Parse JSON transformation below.

Because the transformation sets Drop target field to true, the other body fields (event_type and location) don’t appear in Cloud Observability.

  • Name: Promote message
  • Configuration: Parse JSON
  • Target field: body
  • Drop target field: True
  • Fields to promote: message

Log before:

1

{"message": "Launch successful", "event_type": "Launch", "location": "Sector 7G"}

Log after:

1

{"message": "Launch successful"}

Expandable end

Pipeline order

In log ingestion pipelines, the order of operations affects how Cloud Observability transforms your data.

Pipeline tabs

In Cloud Observability, the log ingestion pipeline page has three tabs:

  • Datadog - The pipeline for logs from the Datadog Agent.
  • OTLP - The pipeline for logs from the OTel Collector.
  • All - The pipeline for all incoming logs, unless the Datadog or OTLP pipelines have transformations.

The All pipeline is a catch-all for all incoming logs. The source-specific pipelines – Datadog and OTLP – take precedence over the All pipeline for logs from those sources.

For example, if you only have an All pipeline, all logs flow through that pipeline. If you then create an OTLP pipeline, Cloud Observability directs logs from the OTel Collector through the OTLP pipeline.

Transformation filters

Filter behavior determines how logs flow through pipelines.

In Keep matching and Drop matching transformations, filters are destructive. The filters decide which logs appear in Cloud Observability and which logs are dropped. For example, if a Keep matching transformation uses sev == "INFO", Cloud Observability ingests INFO logs and drops all other logs.

In other transformations, such as Parse JSON, filters determine which logs the transformation applies to. For example, if a Parse JSON transformation uses sev == "INFO", Cloud Observability only applies the transformation to INFO logs. Other logs, such as ERROR or DEBUG logs, pass through unaffected.

Transformation order

If a pipeline has several transformations, Cloud Observability runs the transformations in order. To change the order, point to a transformation and select the up or down arrow.

Examples

Drop JSON logs

To drop JSON logs where destination == Earth, parse the logs first and drop the logs second. If you reverse the transformation order, both sample logs appear in Cloud Observability.

  • Parse body (transformation 1):
    • Name: Parse body
    • Configuration: Parse JSON
    • Target field: body
  • Drop matching (transformation 2):
    • Name: Drop Earth
    • Configuration: Drop matching
    • Filter: message.details.destination == "Earth"

Logs before:

1
2

{"severity": "INFO", "message": {"action": "Spacecraft launched", "details": {"launch_pad": "LC-39A", "destination": "Mars"}}}
{"severity": "INFO", "message": {"action": "Spacecraft launched", "details": {"launch_pad": "LC-39A", "destination": "Earth"}}}

Logs after:

1

{"severity": "INFO", "message.action": "Spacecraft launched", "message.details.launch_pad": "LC-39A", "message.details.destination": "Mars"}

Expandable end

See also

Create ingestion pipelines

Log ingestion pipelines

Log integrations

Updated Apr 24, 2024