Log integrations

This page shows how to send logs to Cloud Observability. See the sections below for a list of integrations and sample configurations.

Access to Cloud Observability Logging is available on request. Reach out to your Account Manager to get started.

Before you begin

To send logs to Cloud Observability, you need an access token. Create and find access tokens in Project settings > Access tokens.

The sections below also assume you’ve read Cloud Observability’s logging best practices.

Datadog

Use the Datadog Agent to send logs to Cloud Observability.

Basic Datadog Agent configuration

To only send logs to Cloud Observability, paste the content below in your Datadog Agent configuration file. Replace YOUR-ACCESS-TOKEN with your own value.

1
2
3
4
5
6
7
api_key: YOUR-ACCESS-TOKEN

logs_enabled: true
logs_config:
  logs_dd_url: logingest.lightstep.com
  use_v2_api: false
  force_use_http: true

To send logs to both Datadog and Cloud Observability (dual-shipping), use the following configuration. Replace YOUR-ACCESS-TOKEN with your own value.

1
2
3
4
5
6
7
logs_config:
  use_v2_api: false
  use_http: true
  additional_endpoints:
  - api_key: "YOUR-ACCESS-TOKEN"
    host: "logingest.lightstep.com"
    port: 443

Fluent Bit

Use Fluent Bit to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your Fluent Bit configuration file. Replace YOUR-ACCESS-TOKEN with your own value.

1
2
3
4
5
6
7
8
9
10
11
[OUTPUT]
    Name  es
    Match *
    Host  logingest.lightstep.com
    Port  443
    tls On
    
    Time_Key _ts 
    
    HTTP_User lightstep 
    HTTP_Passwd YOUR-ACCESS-TOKEN

Visit Fluent Bit’s Elasticsearch output plugin documentation to learn more about the configuration settings. That plugin works with Cloud Observability because Cloud Observability supports much of Elasticsearch’s API.

Sample Fluent Bit configuration

This Fluent Bit configuration sends dummy logs to Cloud Observability:

1
2
3
4
5
6
7
8
9
10
11
12
[INPUT]
    Name   dummy
    Tag    dummy.log
[OUTPUT]
    Name  es
    Match *
    Host  logingest.lightstep.com
    Port  443
    tls On
    Time_Key _ts
    HTTP_User lightstep
    HTTP_Passwd abcdEfGHIjkLmnoPqrSTuvwvyz1234567890

After restarting Fluent Bit, the logs appear in Cloud Observability. To view them, go to Cloud Observability and click Logs.

Expandable end

Fluentd

Use Fluentd to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your Fluentd configuration file. Replace YOUR-LOG-TAG and YOUR-ACCESS-TOKEN with your own values.

1
2
3
4
5
6
7
8
<match YOUR-LOG-TAG>
  @type elasticsearch
  host logingest.lightstep.com
  port 443
  scheme https
  user lightstep
  password YOUR-ACCESS-TOKEN
</match>

Visit Fluentd’s Elasticsearch output plugin documentation to learn more about the configuration settings. That plugin works with Cloud Observability because Cloud Observability supports much of Elasticsearch’s API.

Sample Fluentd configuration

This Fluentd configuration sends sample events to Cloud Observability:

1
2
3
4
5
6
7
8
9
10
11
12
13
<source>
  @type sample
  sample {"hello":"world"}
  tag sample
</source>

<match sample>
  @type elasticsearch
  host logingest.lightstep.com
  port 443
  scheme https
  custom_headers "Lightstep-access-token: abcdEfGHIjkLmnoPqrSTuvwvyz1234567890"
</match>

After restarting Fluentd, the logs appear in Cloud Observability. To view them, go to Cloud Observability and click Logs.

Expandable end

Logstash

Use Logstash to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your Logstash configuration file. Replace YOUR-ACCESS-TOKEN with your own value.

1
2
3
4
5
6
7
8
9
output {
  elasticsearch {
    hosts => "https://logingest.lightstep.com:443"

   custom_headers => {
        "Lightstep-access-token" => "YOUR-ACCESS-TOKEN"
    }
  }
}

Visit Logstash’s Elasticsearch output plugin documentation to learn more about the configuration settings. That plugin works with Cloud Observability because Cloud Observability supports much of Elasticsearch’s API.

Sample Logstash configuration

This configuration uses Logstash’s file input plugin to send logs to Cloud Observability:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
input {
  file {
    path => "/Users/space.cat/nebula.txt"
  }
}

output {
  elasticsearch {
    hosts => "https://logingest.lightstep.com:443"

   custom_headers => {
        "Lightstep-access-token" => "abcdEfGHIjkLmnoPqrSTuvwvyz1234567890"
    }
  }
}

After restarting Logstash, the logs appear in Cloud Observability. To view them, go to Cloud Observability and click Logs.

Expandable end

OpenTelemetry Collector

Follow our quick start to automatically configure the OpenTelemetry Collector to send Kubernetes logs, metrics, and traces to Cloud Observability.

Use the OpenTelemetry Collector to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your OpenTelemetry Collector configuration file. Replace YOUR-ACCESS-TOKEN with your own values.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
exporters:   
  otlp:
    endpoint: "ingest.lightstep.com:443"
    headers:
      "lightstep-access-token": "YOUR-ACCESS-TOKEN"

processors:
  batch:
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25

service:
  pipelines:
    logs:
      exporters: [otlp]
      processors: [memory_limiter, batch]
      receivers: [otlp] # update with your receiver name

Vector

Use Vector to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your Vector configuration file. Replace YOUR-ACCESS-TOKEN with your own values.

1
2
3
4
5
6
7
8
9
10
11
12
sinks:
  lightstep_logs:
    type: elasticsearch
    inputs: ["..."] # replace with your inputs
    endpoints: ["https://logingest.lightstep.com"]
    healthcheck:
      enabled: false
    batch:
      max_bytes: 2e+06
    request:
      headers:
        "lightstep-access-token": "YOUR-ACCESS-TOKEN"

Visit Vector’s Elasticsearch sink documentation to learn more about the configuration settings.

See also

Log concepts

Explore logs

Updated Sep 27, 2023