Introducing log ingestion pipelines


Log ingestion pipelines let you process and parse incoming logs in a unified interface. Use log ingestion pipelines to manage, secure, and optimize the logs you see in Cloud Observability.

Log ingestion pipeline example with four transformations

What are log ingestion pipelines?

Log ingestion pipelines help you manage incoming logs. Within ingestion pipelines, you create transformations to process and parse logs to suit your specific needs.

Cloud Observability’s log ingestion pipelines come with these features:

  • Customized pipelines: Create pipelines for specific data sources or unify log management with one pipeline.
  • Supported tools: Use ingestion pipelines on logs from OpenTelemetry or Datadog.
  • Transformations: Process and parse logs with transformations, including Drop matching, Keep matching, and Parse JSON.

Key benefits

Some benefits of using log ingestion pipelines include:

  • Centralized log management: Process and manage logs from multiple sources in a unified interface.
  • Cost optimization: Control storage costs by keeping only the logs that matter to your operations.
  • Security: Safeguard sensitive information by removing it from incoming logs.

Get started

Explore the new documentation to start managing and transforming your logs to suit your needs:

Updated May 1, 2024