Lightstep Observability’s metrics integration with Amazon uses CloudWatch Metric Streams to send metric data into its system. In addition, Lightstep Observability supports collecting resource metadata from the following AWS products (click the link to view the metrics Lightstep ingests):
|Elastic File System||Kinesis||Lambda||RDS|
|S3||SNS||SQS||...more coming soon|
You can add EC2 resource tags and other useful EC2 metadata as labels to any metric from an EC2 container, and you can also add ECS, RDS, ElastiCache, ELB, and ApplicationELB resource tags as labels to CloudWatch metrics.
Once Lightstep Observability is ingesting metrics from CloudWatch, you can use Terraform to create a number of pre-built dashboards.
In order to take full advantage of Lightstep Observability’s Change Intelligence features, tag your resources with an appropriate
You need to create a Cloudwatch Metric Stream to push metric data to Lightstep Observability. Additionally, you need to create an AWS role and policy for Lightstep Observability and then grant Lightstep Observability read-only access to your resource metadata. Lightstep strongly recommends using Terraform for this integration. If you are unable to use Terraform, please contact your Technical Account Manager for manual integration instructions.
About CloudWatch metric streams
Amazon offers CloudWatch Metric Streams as a way to continually stream metrics to Lightstep Observability and similar services. Metric streams offer near-real-time delivery and an expected latency of as low as 3 minutes. AWS has embraced the OpenTelemetry standard and currently streams metrics in OTLP v0.7. Metric streams are a reliable and cost-effective way to deliver your CloudWatch metrics to Lightstep Observability.
Use Terraform to integrate
The Terraform repository creates a metric stream, a kinesis firehose, along with the necessary roles and policies for the stream as well as role and the associated policy that grant Lightstep Observability access to your AWS resource metadata.
You need the following to use the Terraform repository:
- Locally configured AWS credentials.
- Basic knowledge of Github.
- Your Lightstep Observability access token from the project into which you would like to stream your metrics.
Download and run the Terraform repository
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
# You may also download and unzip via the command line. % wget https://github.com/lightstep/aws-cloudwatch-metric-stream-terraform/archive/main.zip % unzip main.zip % cd aws-cloudwatch-metric-stream-terraform-main # Configure your AWS credentials. % export AWS_ACCESS_KEY_ID=<access-key-id> % export AWS_SECRET_ACCESS_KEY=<secret-access-key> # Initialize your terraform backend. % terraform init # Apply terraform, entering your Lightstep Observability project access token when prompted. % terraform apply var.lightstep_access_token Lightstep Observability project access token Enter a value: <your-lightstep-access-token> # ----- # The above is the minimal install with default values. # For custom install, copy example.tfvars (eg: `cp example.tfvars my-vars.tfvars`), edit the new file, then run: % terraform apply -var-file="my-vars.tfvars"
The script outputs an Integration Role ARN and an External ID (a random string used to provide extra security). Copy these as you’ll need to paste them into the UI in the next step.
Add credentials to Lightstep Observability
To complete the integration, you need to add the AWS credential information to Lightstep Observability.
In Lightstep Observability, click Settings in the left navigation menu to open the Project Settings page.
On the Project Settings page, scroll down to Metric integrations.
- Click Enable and enter the following AWS credential information in the dialog:
- The Integration Role ARN
These values are included in the output after running the Terraform script.
Your metric data is now enabled in Lightstep Observability. To verify metrics are reporting, search for the metrics in the Metric details section of the Project Settings page.
It may take up to 30 minutes for metric data to appear in Lightstep Observability using the default configuration. This may vary if you set custom values for
It may take up to 30 minutes for changes to resource metadata to propagate to Lightstep Observability. For example, if you add a new custom tag to an EC2 instance, that tag may not appear in Lightstep Observability for up to 30 minutes.
Update the integration
You may reconfigure your integration at any time by re-running
terraform apply with different configuration values.
We recommend using an external
1 2 3 4 5 6 % cd /<path>/<to>/aws-cloudwatch-metric-stream-terraform-main % cp example.tfvars my-vars.tfvars # Edit my-vars.tfvars in your favorite text editor. % terraform apply -var-file="my-vars.tfvars"
It may take up to 30 minutes for configuration changes to propagate to Lightstep Observability. This may vary if you set custom values for
Remove the integration
You may remove your metric stream integration at any time using terraform.
1 2 % cd /<path>/<to>/aws-cloudwatch-metric-stream-terraform-main % terraform destroy
- Percentiles are not included in CloudWatch Metric Streams.
- CloudWatch Metric Streams do not include metrics with timestamps older than two hours - for example, metrics that are aggregated on a daily basis.
If you have previously integrated your CloudWatch metrics with Lightstep Observability prior to the release of our metric streams integration, we strongly recommend transitioning to the streaming integration. It is lower latency, more reliable, and cheaper! Please contact your Technical Account Manager who will be happy to guide you through the upgrade.