Ingestion pipeline reference

Learn about the different transformation types and pipeline order.

Transformation types

Parse and process incoming logs with transformations.

Drop matching

Block logs based on certain conditions.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Drop matching.

  • Filter: Specify the filter in filter-expression format.

    Cloud Observability doesn’t ingest logs that match the filter. All other logs continue through the pipeline. Transformations support every filter expression except phrase_match.

Examples

Drop INFO logs

Block all INFO logs with this Drop matching transformation:

  • Name: Drop INFO logs
  • Configuration: Drop matching
  • Filter: sev == "INFO"

Logs before:

1
2

INFO Spaceship 'Lightstep' reached orbit around Saturn.
ERROR Navigation malfunction: Unable to plot course through asteroid field.

Logs after:

1

ERROR Navigation malfunction: Unable to plot course through asteroid field.

Expandable end

Drop INFO logs for a service

Block INFO logs for the StellarNav service with this Drop matching transformation:

  • Name: Drop INFO logs for StellarNav
  • Configuration: Drop matching
  • Filter: sev == "INFO" && service == "StellarNav"

Logs before:

1
2

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn.
ERROR StellarNav Navigation malfunction: Unable to plot course through asteroid field.

Logs after:

1

ERROR StellarNav Navigation malfunction: Unable to plot course through asteroid field.

Expandable end

Flatten

Move nested attributes in a JSON object to the top level.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Flatten.
  • Target attribute: Specify the name of the JSON attribute you want to flatten, for example, message.

For Flatten transformations, the target attribute must be a JSON object. If the target attribute isn’t a JSON object – for example, a string – the attribute goes through the log ingestion pipeline untouched.

Optional Flatten inputs

These inputs are optional for Flatten transformations:

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only flattens logs that match the filter. Other logs are unaffected by the transformation.

Delimiter

Specify the character that separates the flattened attribute keys.

For example, if you set Delimiter to -, {"message": {"action": "Spacecraft launched"}} flattens to {message-action:"Spacecraft launched"}. The default delimiter is ..

Max depth

Specify the maximum nesting level for Cloud Observability to flatten. By default, Cloud Observability flattens all levels.

For example, if Max depth is 1, Cloud Observability flattens only the first level in the JSON object. Cloud Observability leaves deeper levels unchanged.

Expandable end

Examples

Flatten message

Move every nested attribute in message to the top level:

  • Name: Flatten message
  • Configuration: Flatten
  • Target attribute: message

Logs before:

1
2
3
4
5
6
7
8

"message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A",
    "payload": "Exploration Rover"
  }
}

Logs after:

1
2
3
4

"message.action": "Spacecraft launched",
"message.details.destination": "Mars",
"message.details.launch_pad": "LC-39A",
"message.details.payload": "Exploration Rover"

Expandable end

Flatten one level

Move only attributes at level 1 to the top level:

  • Name: Flatten message
  • Configuration: Flatten
  • Target attribute: message
  • Max depth: 1

Logs before:

1
2
3
4
5
6
7
8

"message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A",
    "payload": "Exploration Rover"
  }
}

Logs after:

1
2
3
4
5
6

"message.action": "Spacecraft launched",
"message.details": {
  "destination": "Mars",
  "launch_pad": "LC-39A",
  "payload": "Exploration Rover"
}

Expandable end

Flatten body (OTel Collector example)

The OpenTelemetry (OTel) Collector sends log information to Cloud Observability as a string in body. To flatten attributes in body, use the Parse JSON transformation to turn body attributes into JSON objects. Then use the Flatten transformation on the JSON-object attributes.

  • Parse body (transformation 1):

    • Name: Parse body
    • Configuration: Parse JSON
    • Target attribute: body

  • Flatten message (transformation 2):

    • Name: Flatten message
    • Configuration: Flatten
    • Target attribute: message

Logs before:

1

"body": "{\"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\", \"payload\": \"Exploration Rover\"}}}"

Logs after transformation 1:

1
2
3
4
5
6
7
8
9

"body": "{\"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\", \"payload\": \"Exploration Rover\"}}}",
"message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A",
    "payload": "Exploration Rover"
  }
}

Logs after transformation 2:

1
2
3
4
5

"body": "{\"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\", \"payload\": \"Exploration Rover\"}}}",
"message.action": "Spacecraft launched",
"message.details.destination": "Mars",
"message.details.launch_pad": "LC-39A",
"message.details.payload": "Exploration Rover"

Expandable end

Keep matching

Ingest logs based on certain conditions.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Keep matching.

  • Filter: Specify the filter in filter-expression format.

    Cloud Observability only ingests logs that match the filter. It drops all other logs. Transformations support every filter expression except phrase_match.

Examples

Keep logs where body contains “sect 8”

Only keep sect 8 logs with this Keep matching transformation:

  • Name: Keep sector 8 logs
  • Configuration: Keep matching
  • Filter: contains(body, "sect 8")

Logs before:

1
2
3

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn in sect 8.
ERROR Lightstep Navigation malfunction in sect 4: Unable to plot course through asteroid field.
FATAL Juno core meltdown. Evacuate the ship immediately! Everything in sect 8 is compromised.

Logs after:

1
2

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn in sect 8.
FATAL Juno core meltdown. Evacuate the ship immediately! Everything in sect 8 is compromised.

Expandable end

Parse JSON

Parse JSON-encoded strings into logical attributes.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Parse JSON.
  • Target attribute: Specify the name of the attribute you want to parse, for example, body.

Optional Parse JSON inputs

These inputs are optional for Parse JSON transformations:

Drop target attribute

Toggle this option to remove the target attribute once it’s parsed.

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only parses logs that match the filter. Other logs are unaffected by the transformation.

Attributes to promote

Extract specific JSON attributes and promote them to the top level of the log.

For example, enter action to promote the action attribute in the target body attribute. If you specify a prefix in Prefix new attributes (see below), Cloud Observability adds the prefix to action. Select Add attribute to promote multiple attributes.

Prefix new attributes

Add context to the front of new top-level attribute names.

For example, station results in attribute names like station.destination and station.launch_pad.

JSON max depth

Specify the maximum nesting level for Cloud Observability to parse.

For example, if JSON max depth is 3, Cloud Observability only parses up to 3 levels in the JSON structure.

Expandable end

Examples

Parse body

Parse the body attribute with this Parse JSON transformation:

  • Name: Parse body
  • Configuration: Parse JSON
  • Target attribute: body

Log before:

1

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}"

Log after:

1
2
3
4
5
6
7
8
9

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}",
"message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A"
  }
},
"severity": "INFO"

Expandable end

Parse body and add context

Parse the body attribute and add information with this Parse JSON transformation:

  • Name: Parse body
  • Configuration: Parse JSON
  • Target attribute: body
  • Prefix new attributes: context

Log before:

1

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}"

Log after:

1
2
3
4
5
6
7
8
9

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}",
"context.message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A"
  }
},
"context.severity": "INFO"

Expandable end

Promote specific attributes

Parse body and promote its message attribute with the Parse JSON transformation below.

Because the transformation sets Drop target attribute to true, the other body attributes (event_type and location) don’t appear in Cloud Observability.

  • Name: Promote message
  • Configuration: Parse JSON
  • Target attribute: body
  • Drop target attribute: True
  • Attributes to promote: message

Log before:

1

"body": "{\"message\": \"Launch successful\", \"event_type\": \"Launch\", \"location\": \"Sector 7G\"}"

Log after:

1

"message": "Launch successful"

Expandable end

Rename

Give new names to specific attributes.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Rename.
  • Rename rules: Specify the attribute’s current and new name. Select Add rule to rename more attributes.

Optional Rename inputs

This input is optional for Rename transformations:

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only renames attributes in logs that match the filter. Other logs are unaffected by the transformation.

Expandable end

Examples

Rename message to body

Rename the message attribute to body with the Rename transformation below. Because body is tokenized, Cloud Observability automatically tokenizes any attributes you rename to body.

  • Name: Rename message to body
  • Configuration: Rename
  • Rename rules: message -> body

Logs before:

1

"message": "Spacecraft launched"

Logs after:

1

"body": "Spacecraft launched"

Expandable end

Pipeline order

In log ingestion pipelines, the order of operations affects how Cloud Observability transforms your data.

Pipeline tabs

In Cloud Observability, the log ingestion pipeline page has three tabs:

  • Datadog - The pipeline for logs from the Datadog Agent.
  • OTLP - The pipeline for logs from the OTel Collector.
  • All - The pipeline for all incoming logs, unless the Datadog or OTLP pipelines have transformations.

The All pipeline is a catch-all for all incoming logs. The source-specific pipelines – Datadog and OTLP – take precedence over the All pipeline for logs from those sources.

For example, if you only have an All pipeline, all logs flow through that pipeline. If you then create an OTLP pipeline, Cloud Observability directs logs from the OTel Collector through the OTLP pipeline.

Transformation filters

Filter behavior determines how logs flow through pipelines.

In Keep matching and Drop matching transformations, filters are destructive. The filters decide which logs appear in Cloud Observability and which logs are dropped. For example, if a Keep matching transformation uses sev == "INFO", Cloud Observability ingests INFO logs and drops all other logs.

In other transformations, such as Parse JSON, filters determine which logs the transformation applies to. For example, if a Parse JSON transformation uses sev == "INFO", Cloud Observability only applies the transformation to INFO logs. Other logs, such as ERROR or DEBUG logs, pass through unaffected.

Transformation order

If a pipeline has several transformations, Cloud Observability runs the transformations in order. To change the order, point to a transformation and select the up or down arrow.

Example

Drop JSON logs

To drop JSON logs where destination == Earth, parse the logs first and drop the logs second. If you reverse the transformation order, both sample logs appear in Cloud Observability.

  • Parse body (transformation 1):

    • Name: Parse body
    • Configuration: Parse JSON
    • Target attribute: body

  • Drop matching (transformation 2):

    • Name: Drop Earth
    • Configuration: Drop matching
    • Filter: message.details.destination == "Earth"

Logs before:

1
2

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}",
"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Earth\"}}}"

Logs after:

1

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}"

Expandable end

See also

Create ingestion pipelines

Log ingestion pipelines

Log integrations

Updated Jun 11, 2024