Ingestion pipeline reference

Learn about the different transformation types and pipeline order.

Transformation types

Parse and process incoming logs with transformations.

Drop matching

Block logs based on certain conditions.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Drop matching.

  • Filter: Specify the filter in filter-expression format.

    Cloud Observability doesn’t ingest logs that match the filter. All other logs continue through the pipeline. Transformations support every filter expression except phrase_match.

Examples

Drop INFO logs

Block all INFO logs with this Drop matching transformation:

  • Name: Drop INFO logs
  • Configuration: Drop matching
  • Filter: sev == "INFO"

Logs before:

1
2

INFO Spaceship 'Lightstep' reached orbit around Saturn.
ERROR Navigation malfunction: Unable to plot course through asteroid field.

Logs after:

1

ERROR Navigation malfunction: Unable to plot course through asteroid field.

Expandable end

Drop INFO logs for a service

Block INFO logs for the StellarNav service with this Drop matching transformation:

  • Name: Drop INFO logs for StellarNav
  • Configuration: Drop matching
  • Filter: sev == "INFO" && service == "StellarNav"

Logs before:

1
2

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn.
ERROR StellarNav Navigation malfunction: Unable to plot course through asteroid field.

Logs after:

1

ERROR StellarNav Navigation malfunction: Unable to plot course through asteroid field.

Expandable end

Flatten

Move nested attributes in a JSON object to the top level.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Flatten.
  • Target attribute: Specify the name of the JSON attribute you want to flatten, for example, message.

For Flatten transformations, the target attribute must be a JSON object. If the target attribute isn’t a JSON object – for example, a string – the attribute goes through the log ingestion pipeline untouched.

Optional Flatten inputs

These inputs are optional for Flatten transformations:

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only flattens logs that match the filter. Other logs are unaffected by the transformation.

Delimiter

Specify the character that separates the flattened attribute keys.

For example, if you set Delimiter to -, {"message": {"action": "Spacecraft launched"}} flattens to {message-action:"Spacecraft launched"}. The default delimiter is ..

Max depth

Specify the maximum nesting level for Cloud Observability to flatten. By default, Cloud Observability flattens all levels.

For example, if Max depth is 1, Cloud Observability flattens only the first level in the JSON object. Cloud Observability leaves deeper levels unchanged.

Expandable end

Examples

Flatten message

Move every nested attribute in message to the top level:

  • Name: Flatten message
  • Configuration: Flatten
  • Target attribute: message

Logs before:

1
2
3
4
5
6
7
8

"message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A",
    "payload": "Exploration Rover"
  }
}

Logs after:

1
2
3
4

"message.action": "Spacecraft launched",
"message.details.destination": "Mars",
"message.details.launch_pad": "LC-39A",
"message.details.payload": "Exploration Rover"

Expandable end

Flatten one level

Move only attributes at level 1 to the top level:

  • Name: Flatten message
  • Configuration: Flatten
  • Target attribute: message
  • Max depth: 1

Logs before:

1
2
3
4
5
6
7
8

"message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A",
    "payload": "Exploration Rover"
  }
}

Logs after:

1
2
3
4
5
6

"message.action": "Spacecraft launched",
"message.details": {
  "destination": "Mars",
  "launch_pad": "LC-39A",
  "payload": "Exploration Rover"
}

Expandable end

Flatten body (OTel Collector example)

The OpenTelemetry (OTel) Collector sends log information to Cloud Observability as a string in body. To flatten attributes in body, use the Parse JSON transformation to turn body attributes into JSON objects. Then use the Flatten transformation on the JSON-object attributes.

  • Transformation 1: Parse body:

    • Name: Parse body
    • Configuration: Parse JSON
    • Target attribute: body

  • Transformation 2: Flatten message:

    • Name: Flatten message
    • Configuration: Flatten
    • Target attribute: message

Logs before:

1

"body": "{\"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\", \"payload\": \"Exploration Rover\"}}}"

Logs after transformation 1:

1
2
3
4
5
6
7
8
9

"body": "{\"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\", \"payload\": \"Exploration Rover\"}}}",
"message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A",
    "payload": "Exploration Rover"
  }
}

Logs after transformation 2:

1
2
3
4
5

"body": "{\"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\", \"payload\": \"Exploration Rover\"}}}",
"message.action": "Spacecraft launched",
"message.details.destination": "Mars",
"message.details.launch_pad": "LC-39A",
"message.details.payload": "Exploration Rover"

Expandable end

Format

Aggregate attributes into a single attribute.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Format.
  • Attribute: Specify the attribute key for the aggregated values.
  • Overwrite existing attribute: Toggle this option to replace existing values with the new values. Otherwise, the current values stay unchanged.
  • Format string: Aggregate attributes using the syntax %{attribute-name}. For example, if moon==europa and planet==saturn, %{moon}_%{planet} generates europa_saturn.

Format doesn’t work with attributes such as signal{strength}%. If your attribute keys include {,}, or %, consider using a different transformation, such as Rename, to change those keys first.

Optional Format inputs

This input is optional for Format transformations:

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only applies Format to logs matching the filter. Other logs are unaffected by the transformation.

Expandable end

Examples

Aggregate two attributes

Aggregate two existing attributes into a new attribute called mission_event.

  • Name: Aggregate mission and event
  • Configuration: Format
  • Attribute: mission_event
  • Overwrite existing attribute: False
  • Format string: %{mission}_%{event}

Logs before:

1
2
3
4

{
  "event": "LunarLanding",
  "mission":"ServiceNow24"
}

Logs after:

1
2
3
4
5

{
  "event": "LunarLanding",
  "mission":"ServiceNow24",
  "mission_event": "ServiceNow24_LunarLanding"
}

Expandable end

Aggregate nested attributes in body

Create two transformations to aggregate attributes nested in body. First, use Parse JSON to move the attributes to the top level of the log. Second, use Format to aggregate the top-level attributes into a new attribute.

Format only works on top-level attributes. It doesn’t support nested syntax, for example, %{body{mission}}_%{body{event}}.

  • Transformation 1: Parse body
    • Name: Parse body
    • Configuration: Parse JSON
    • Target attribute: body
  • Transformation 2: Aggregate attributes
    • Name: Aggregate mission and event
    • Configuration: Format
    • Attribute: mission_event
    • Overwrite existing attribute: False
    • Format string: %{mission}_%{event}

Logs before:

1
2
3

{
  "body":"{"mission":"ServiceNow24","event": "LunarLanding"}"
}

Logs after transformation 1:

1
2
3
4
5

{
  "body":"{"mission":"ServiceNow24","event": "LunarLanding"}",
  "event": "LunarLanding",
  "mission":"ServiceNow24"
}

Logs after transformation 2:

1
2
3
4
5
6

{
  "body":"{"mission":"ServiceNow24","event": "LunarLanding"}",
  "event": "LunarLanding",
  "mission":"ServiceNow24",
  "mission_event": "ServiceNow24_LunarLanding"
}

Expandable end

Keep matching

Ingest logs based on certain conditions.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Keep matching.

  • Filter: Specify the filter in filter-expression format.

    Cloud Observability only ingests logs that match the filter. It drops all other logs. Transformations support every filter expression except phrase_match.

Examples

Keep logs where body contains “sect 8”

Only keep sect 8 logs with this Keep matching transformation:

  • Name: Keep sector 8 logs
  • Configuration: Keep matching
  • Filter: contains(body, "sect 8")

Logs before:

1
2
3

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn in sect 8.
ERROR Lightstep Navigation malfunction in sect 4: Unable to plot course through asteroid field.
FATAL Juno core meltdown. Evacuate the ship immediately! Everything in sect 8 is compromised.

Logs after:

1
2

INFO StellarNav Spaceship 'Lightstep' reached orbit around Saturn in sect 8.
FATAL Juno core meltdown. Evacuate the ship immediately! Everything in sect 8 is compromised.

Expandable end

Parse JSON

Parse JSON-encoded strings into logical attributes.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Parse JSON.
  • Target attribute: Specify the name of the attribute you want to parse, for example, body.

Optional Parse JSON inputs

These inputs are optional for Parse JSON transformations:

Drop target attribute

Toggle this option to remove the target attribute once it’s parsed.

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only parses logs that match the filter. Other logs are unaffected by the transformation.

Attributes to promote

Extract specific JSON attributes and promote them to the top level of the log.

For example, enter action to promote the action attribute in the target body attribute. If you specify a prefix in Prefix new attributes (see below), Cloud Observability adds the prefix to action. Select Add attribute to promote multiple attributes.

Prefix new attributes

Add context to the front of new top-level attribute names.

For example, station results in attribute names like station.destination and station.launch_pad.

JSON max depth

Specify the maximum nesting level for Cloud Observability to parse.

For example, if JSON max depth is 3, Cloud Observability only parses up to 3 levels in the JSON structure.

Expandable end

Examples

Parse body

Parse the body attribute with this Parse JSON transformation:

  • Name: Parse body
  • Configuration: Parse JSON
  • Target attribute: body

Log before:

1

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}"

Log after:

1
2
3
4
5
6
7
8
9

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}",
"message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A"
  }
},
"severity": "INFO"

Expandable end

Parse body and add context

Parse the body attribute and add information with this Parse JSON transformation:

  • Name: Parse body
  • Configuration: Parse JSON
  • Target attribute: body
  • Prefix new attributes: context

Log before:

1

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}"

Log after:

1
2
3
4
5
6
7
8
9

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}",
"context.message": {
  "action": "Spacecraft launched",
  "details": {
    "destination": "Mars",
    "launch_pad": "LC-39A"
  }
},
"context.severity": "INFO"

Expandable end

Promote specific attributes

Parse body and promote its message attribute with the Parse JSON transformation below.

Because the transformation sets Drop target attribute to true, the other body attributes (event_type and location) don’t appear in Cloud Observability.

  • Name: Promote message
  • Configuration: Parse JSON
  • Target attribute: body
  • Drop target attribute: True
  • Attributes to promote: message

Log before:

1

"body": "{\"message\": \"Launch successful\", \"event_type\": \"Launch\", \"location\": \"Sector 7G\"}"

Log after:

1

"message": "Launch successful"

Expandable end

Rename

Give new names to specific attributes.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Rename.
  • Rename rules: Specify the attribute’s current and new name. Select Add rule to rename more attributes.

Optional Rename inputs

This input is optional for Rename transformations:

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only renames attributes in logs that match the filter. Other logs are unaffected by the transformation.

Expandable end

Examples

Rename message to body

Rename the message attribute to body with the Rename transformation below. Because body is tokenized, Cloud Observability automatically tokenizes any attributes you rename to body.

  • Name: Rename message to body
  • Configuration: Rename
  • Rename rules: message -> body

Logs before:

1

"message": "Spacecraft launched"

Logs after:

1

"body": "Spacecraft launched"

Expandable end

Set

Assign values to new or existing attributes.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Set.

  • Target attribute: Specify a new or existing attribute key to set values for.

    If your target attribute includes periods (.), use quotations: "client.browser".

  • Overwrite existing attribute: Toggle this option to replace existing values with the new values. Otherwise, the current values stay unchanged.
  • Value: Specify the value to assign to the attribute. You can enter strings, numbers (integers or floats), or boolean values.

Optional Set inputs

This input is optional for Set transformations:

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only applies Set to logs matching the filter. Other logs are unaffected by the transformation.

Expandable end

Example

Assign statuses based on HTTP status codes

To assign log statuses (OK, Notice, Warning, Error, and Info) based on HTTP status codes, create the five Set transformations below:

Transformation input Transformation
1		 Transformation
2		 Transformation
3		 Transformation
4		 Transformation
5
Name Set OK Set Notice Set Warning Set Error Set Info
Configuration Set Set Set Set Set
Target attribute status status status status status
Overwrite existing attribute True True True True True
Filter: "\"http.status_code\"" >= 200 && "\"http.status_code\"" <= 299 "\"http.status_code\"" >= 300 && "\"http.status_code\"" <= 399 "\"http.status_code\"" >= 400 && "\"http.status_code\"" <= 499 "\"http.status_code\"" >= 500 && "\"http.status_code\"" <= 599 "\"http.status_code\"" <= 100
Value OK Notice Warning Error Info

If an attribute key has a period (.), use this syntax: "\"<attribute.name>\"". The syntax indicates a full attribute key, not a nested JSON attribute. For example, the transformations above use "\"http.status_code\"".

Logs before:

1
2
3
4
5
6
7
8
9
10
11
12

{
  "http.status_code": 200,
  "message": "Mission status: Observability has landed."
}
{
  "http.status_code": 302,
  "message": "Mission status: Redirected for data relay."
}
{
  "http.status_code": 404,
  "message": "Error: Satellite communication lost."
}

Logs after:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
  "http.status_code": 200,
  "status": "OK",
  "message": "Mission status: Observability has landed."
}
{
  "http.status_code": 302,
  "status": "Notice",
  "message": "Mission status: Redirected for data relay."
}
{
  "http.status_code": 404,
  "status": "Warning",
  "message": "Error: Satellite communication lost."
}

Expandable end

Regex extract

Parse attributes using regular expressions.

Inputs

  • Name: Identify the transformation.
  • Configuration: Select the transformation type: Regex extract.
  • Attribute: Specify the attribute to parse. For example, body.

  • Regular expression: Use a regular expression to match patterns in logs and extract attribute values.

    Regular expressions use the RE2 syntax. For example, user ID '(?<userID>\w+)' captures a user ID from a string and assigns it to the userID attribute.

Optional Regex extract inputs

This input is optional for Regex extract transformations:

Filter

Specify the filter in filter-expression format. Transformations support every filter expression except phrase_match.

Cloud Observability only applies Regex extract to logs matching the filter. Other logs are unaffected by the transformation.

Expandable end

Example

Extract IDs from attributes

Capture two IDs from message and assign them to the userID and resourceID attributes:

  • Name: Extract the user and resource IDs for auth
  • Configuration: Regex extract
  • Attribute: message
  • Filter: "\"service.name\"" == auth
  • Regular expression: user ID '(?<userID>\w+)'.* resource ID '(?<resourceID>\w+)'

If an attribute key has a period (.), use this syntax: "\"<attribute.name>\"". The syntax indicates a full attribute key, not a nested JSON attribute. For example, the transformation above uses "\"service.name\"".

Logs before:

1
2
3
4
5
6
7
8
9
10
11
12

{
  "message": "user ID 'commander123' successfully authenticated for resource ID 'mars_42'",
  "service.name":"auth"
}
{
  "message": "user ID 'pilot890' requested control of resource ID 'satellite_beta'",
  "service.name":"auth"
}
{
  "message": "user ID 'navigator124' initiated diagnostics on resource ID 'space_station_clouds'",
  "service.name":"billing"
}

Logs after:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
  "message": "user ID 'commander123' successfully authenticated for resource ID 'mars_42'",
  "service.name":"auth",
  "resourceID": "mars_42",
  "userID": "commander123"
}
{
  "message": "user ID 'pilot890' requested control of resource ID 'satellite_beta'",
  "service.name":"auth",
  "resourceID": "satellite_beta",
  "userID": "pilot890"
}
{
  "message": "user ID 'navigator124' initiated diagnostics on resource ID 'space_station_clouds'",
  "service.name":"billing"
}

Expandable end

Pipeline order

In log ingestion pipelines, the order of operations affects how Cloud Observability transforms your data.

Pipeline tabs

In Cloud Observability, the log ingestion pipeline page has three tabs:

  • Datadog - The pipeline for logs from the Datadog Agent.
  • OTLP - The pipeline for logs from the OTel Collector.
  • All - The pipeline for all incoming logs, unless the Datadog or OTLP pipelines have transformations.

The All pipeline is a catch-all for all incoming logs. The source-specific pipelines – Datadog and OTLP – take precedence over the All pipeline for logs from those sources.

For example, if you only have an All pipeline, all logs flow through that pipeline. If you then create an OTLP pipeline, Cloud Observability directs logs from the OTel Collector through the OTLP pipeline.

Transformation filters

Filter behavior determines how logs flow through pipelines.

In Keep matching and Drop matching transformations, filters are destructive. The filters decide which logs appear in Cloud Observability and which logs are dropped. For example, if a Keep matching transformation uses sev == "INFO", Cloud Observability ingests INFO logs and drops all other logs.

In other transformations, such as Parse JSON, filters determine which logs the transformation applies to. For example, if a Parse JSON transformation uses sev == "INFO", Cloud Observability only applies the transformation to INFO logs. Other logs, such as ERROR or DEBUG logs, pass through unaffected.

Transformation order

If a pipeline has several transformations, Cloud Observability runs the transformations in order. To change the order, point to a transformation and select the up or down arrow.

Example

Drop JSON logs

To drop JSON logs where destination == Earth, parse the logs first and drop the logs second. If you reverse the transformation order, both sample logs appear in Cloud Observability.

  • Transformation 1: Parse body:

    • Name: Parse body
    • Configuration: Parse JSON
    • Target attribute: body

  • Transformation 2: Drop matching:

    • Name: Drop Earth
    • Configuration: Drop matching
    • Filter: message.details.destination == "Earth"

Logs before:

1
2

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}",
"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Earth\"}}}"

Logs after:

1

"body": "{\"severity\": \"INFO\", \"message\": {\"action\": \"Spacecraft launched\", \"details\": {\"launch_pad\": \"LC-39A\", \"destination\": \"Mars\"}}}"

Expandable end

See also

Create ingestion pipelines

Log ingestion pipelines

Log integrations

Updated Sep 25, 2024