Log concepts

Learn concepts and general guidelines for sending logs to Cloud Observability.

Integrations

Send logs to Cloud Observability with tools such as Logstash, OpenTelemetry Collector, and Vector. Visit Log integrations for the list of integrations and setup instructions.

If you’re using OpenTelemetry (OTel) or Datadog, learn how to parse and filter incoming logs with log ingestion pipelines.

General guidelines

Before sending logs to Cloud Observability, review the sections below to help optimize your logging experience.

Log `body` tokenization

Cloud Observability tokenizes only the body attribute of a log. It parses the attribute value and stores it as distinct strings. For example, Cloud Observability stores body="space launch" as body=["space","launch"].

Cloud Observability tokenizes the body attribute to improve query performance and help you find information.

If you’re using OTel or Datadog, use log ingestion pipelines to rename an incoming attribute to body. If you can’t change the attribute name to body or want to use a different name, contact your customer success representative about attribute remapping.

Connect logs to traces

To help you explore and resolve issues, Cloud Observability lets you connect logs and traces in the logs tab and Trace view.

To use those features, when sending log data to Cloud Observability, include span IDs in one of the attribute keys below. Example: span_id=25bd1104506ec466.

span_id
SpanId
tags.span_id

How it works

Select the tabs to learn about logging authorization, indexing, and ingest format.

Access tokens let users and tools send data to Cloud Observability.

Access tokens are project-specific. You create access tokens in a project. And you include them in API requests or tool configurations to send data to that project.

There are two ways to configure access tokens in requests:
- Set the access token in a lightstep-access-token header.
- Using basic authentication, set the access token as the password.
  
  Cloud Observability Logs ignores the username, so you can set it to anything, for example, HTTP_User spacecat.
Indexes are a collection of logs. Indexes, as Elasticsearch uses them, are equivalent to projects in Cloud Observability.

When sending data to Cloud Observability in the Elasticsearch format, your access token identifies the project. If you must include an index name, set the name to anything you want – Cloud Observability ignores the setting.
Cloud Observability Logs supports Elasticsearch’s Bulk API format for ingesting logs. Several integrations rely on existing Elasticsearch output exporters, plugins, and sinks.