Log integrations

This page shows how to send logs to Cloud Observability. See the sections below for a list of integrations and sample configurations.

As of March 12, 2024, all non-logging customers have trial access to Cloud Observability logs. Use trial access to explore logging before committing to a full subscription. To get full access, contact Customer Success.

Before you begin

To send logs to Cloud Observability, you need an access token. Create and find access tokens in Settings > Access tokens.

The sections below also assume you’ve read Cloud Observability’s general guidelines for logs.

OpenTelemetry Collector

Follow the quickstart to configure the OpenTelemetry Collector to send Kubernetes logs, metrics, and traces to Cloud Observability.

Use the OpenTelemetry Collector to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your OpenTelemetry Collector configuration file. Replace YOUR-ACCESS-TOKEN and the receivers configuration with your own values.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
exporters:   
  otlp:
    endpoint: "ingest.lightstep.com:443"
    headers:
      "lightstep-access-token": "YOUR-ACCESS-TOKEN"

processors:
  batch:

service:
  pipelines:
    logs:
      receivers: [otlp] # FIXME update with your receiver name
      processors: [batch]
      exporters: [otlp]

Datadog

Use the Datadog Agent to send logs to Cloud Observability.

Basic Datadog Agent configuration

To only send logs to Cloud Observability, paste the content below in your Datadog Agent configuration file. Replace YOUR-ACCESS-TOKEN with your own value.

1
2
3
4
5
6
7
8
api_key: YOUR-ACCESS-TOKEN

logs_enabled: true
logs_config:
  logs_dd_url: logingest.lightstep.com # US data center
  # logs_dd_url: logingest.eu.lightstep.com # EU data center
  use_v2_api: false
  force_use_http: true

To send logs to both Datadog and Cloud Observability (dual-shipping), use the following configuration. Replace YOUR-ACCESS-TOKEN with your own value.

1
2
3
4
5
6
7
8
logs_config:
  use_v2_api: false
  use_http: true
  additional_endpoints:
  - api_key: "YOUR-ACCESS-TOKEN"
    host: "logingest.lightstep.com" # US data center
    # host: "logingest.eu.lightstep.com" # EU data center
    port: 443

Fluent Bit

Use Fluent Bit to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your Fluent Bit configuration file. Replace YOUR-ACCESS-TOKEN with your own value.

1
2
3
4
5
6
7
8
9
10
11
12
[OUTPUT]
    Name  es
    Match *
    Host  logingest.lightstep.com # US data center
    # Host logingest.eu.lightstep.com # EU datacenter
    Port  443
    tls On
    
    Time_Key _ts 
    
    HTTP_User lightstep 
    HTTP_Passwd YOUR-ACCESS-TOKEN

Visit Fluent Bit’s Elasticsearch output plugin documentation to learn more about the configuration settings. That plugin works with Cloud Observability because Cloud Observability supports much of Elasticsearch’s API.

Sample Fluent Bit configuration

This Fluent Bit configuration sends dummy logs to Cloud Observability:

1
2
3
4
5
6
7
8
9
10
11
12
13
[INPUT]
    Name   dummy
    Tag    dummy.log
[OUTPUT]
    Name  es
    Match *
    Host  logingest.lightstep.com # US data center
    # Host logingest.eu.lightstep.com # EU data center
    Port  443
    tls On
    Time_Key _ts
    HTTP_User lightstep
    HTTP_Passwd abcdEfGHIjkLmnoPqrSTuvwvyz1234567890

After restarting Fluent Bit, the logs appear in Cloud Observability. To view them, go to Cloud Observability and click Logs.

Expandable end

Fluentd

Use Fluentd to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your Fluentd configuration file. Replace YOUR-LOG-TAG and YOUR-ACCESS-TOKEN with your own values.

1
2
3
4
5
6
7
8
9
<match YOUR-LOG-TAG>
  @type elasticsearch
  host logingest.lightstep.com # US data center
  # host logingest.eu.lightstep.com # EU data center
  port 443
  scheme https
  user lightstep
  password YOUR-ACCESS-TOKEN
</match>

Visit Fluentd’s Elasticsearch output plugin documentation to learn more about the configuration settings. That plugin works with Cloud Observability because Cloud Observability supports much of Elasticsearch’s API.

Sample Fluentd configuration

This Fluentd configuration sends sample events to Cloud Observability:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<source>
  @type sample
  sample {"hello":"world"}
  tag sample
</source>

<match sample>
  @type elasticsearch
  host logingest.lightstep.com # US data center
  # host logingest.eu.lightstep.com # EU data center
  port 443
  scheme https
  custom_headers "Lightstep-access-token: abcdEfGHIjkLmnoPqrSTuvwvyz1234567890"
</match>

After restarting Fluentd, the logs appear in Cloud Observability. To view them, go to Cloud Observability and click Logs.

Expandable end

Logstash

Use Logstash to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your Logstash configuration file. Replace YOUR-ACCESS-TOKEN with your own value.

1
2
3
4
5
6
7
8
9
10
output {
  elasticsearch {
    hosts => "https://logingest.lightstep.com:443" // US data center
    // hosts => "https://logingest.eu.lightstep.com:443" # EU data center

   custom_headers => {
        "Lightstep-access-token" => "YOUR-ACCESS-TOKEN"
    }
  }
}

Visit Logstash’s Elasticsearch output plugin documentation to learn more about the configuration settings. That plugin works with Cloud Observability because Cloud Observability supports much of Elasticsearch’s API.

Sample Logstash configuration

This configuration uses Logstash’s file input plugin to send logs to Cloud Observability:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
input {
  file {
    path => "/Users/space.cat/nebula.txt"
  }
}

output {
  elasticsearch {
    hosts => "https://logingest.lightstep.com:443" // US data center
    // hosts => "https://logingest.eu.lightstep.com:443" # EU data center

   custom_headers => {
        "Lightstep-access-token" => "abcdEfGHIjkLmnoPqrSTuvwvyz1234567890"
    }
  }
}

After restarting Logstash, the logs appear in Cloud Observability. To view them, go to Cloud Observability and click Logs.

Expandable end

Vector

Use Vector to send logs to Cloud Observability.

Basic Cloud Observability configuration

To send logs to Cloud Observability, paste the content below in your Vector configuration file. Replace YOUR-ACCESS-TOKEN with your own values.

1
2
3
4
5
6
7
8
9
10
11
12
13
sinks:
  lightstep_logs:
    type: elasticsearch
    inputs: ["..."] # replace with your inputs
    endpoints: ["https://logingest.lightstep.com"] # US data center
    # endpoints: ["https://logingest.eu.lightstep.com"] # EU data center
    healthcheck:
      enabled: false
    batch:
      max_bytes: 2e+06
    request:
      headers:
        "lightstep-access-token": "YOUR-ACCESS-TOKEN"

Visit Vector’s Elasticsearch sink documentation to learn more about the configuration settings.

Sample Vector configuration

This Vector configuration sends demo logs to Cloud Observability:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
sources:
  sample_logs:
    type: demo_logs
    format: apache_common
    lines:
      - line1
transforms:
  parse_logs:
    type: remap
    inputs:
      - sample_logs
    source: |
      . = parse_apache_log!(string!(.message), "common") 
      .body = del(.message)
      
sinks:
  lightstep_logs:
    type: elasticsearch
    inputs: ["parse_logs"]
    endpoints: ["https://logingest.lightstep.com:443"]
    healthcheck:
      enabled: false
    bulk:
      index: lightstep
    request:
      headers:
        "lightstep-access-token": "abcdEfGHIjkLmnoPqrSTuvwvyz1234567890"

After restarting Vector, the logs appear in Cloud Observability. To view them, go to Cloud Observability and click Logs.

Expandable end

See also

Log concepts

Explore logs

Log ingestion pipelines

Updated Mar 19, 2024