Ingest metrics from Amazon

Cloud Observability’s metrics integration with Amazon uses CloudWatch Metric Streams to send metric data into its system. From the AWS integrations page, you can access AWS CloudFormation and choose the projects that you want to integrate. A built-in template provides most of the needed configuration (you can edit as needed), including any resource enrichments.

Once the integration is configured, you can click a button to create one of the pre-built dashboards and see that data in Cloud Observability.

You can also use Terraform to create pre-built dashboards.

In order to take full advantage of Cloud Observability’s correlation feature, tag your resources with an appropriate service.name tag.

AWS tag management form sets service.name as the key.

About CloudWatch metric streams

Amazon offers CloudWatch Metric Streams as a way to continually stream metrics to Cloud Observability and similar services. Metric streams offer near-real-time delivery and an expected latency of as low as 3 minutes. AWS has embraced the OpenTelemetry standard and currently streams metrics in OTLP v0.7. Metric streams are a reliable and cost-effective way to deliver your CloudWatch metrics to Cloud Observability.

Requirements

You need the following to integrate with AWS metrics:

  • An AWS account.
  • AWS credentials with these permissions:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

"cloudformation:*",
"cloudwatch:DeleteMetricStream",
"cloudwatch:GetMetricStream",
"cloudwatch:PutMetricData",
"cloudwatch:PutMetricStream",
"cloudwatch:StartMetricStreams",
"cloudwatch:StopMetricStreams",
"ec2:DescribeRegions"
"firehose:CreateDeliveryStream",
"firehose:DeleteDeliveryStream",
"firehose:DescribeDeliveryStream",
"iam:AttachRolePolicy",
"iam:CreateRole",
"iam:DeleteRole",
"iam:DeleteRolePolicy",
"iam:DetachRolePolicy",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:PassRole",
"iam:PutRolePolicy",
"lambda:CreateFunction",
"lambda:DeleteFunction",
"lambda:GetFunction",
"lambda:InvokeFunction",
"s3:AbortMultipartUpload",
"s3:CreateBucket",
"s3:DeleteBucket",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutBucketPublicAccessBlock",
"s3:PutLifecycleConfiguration",
"s3:PutObject"
"tag:GetResources"

Create an AWS integration

Cloud Observability uses AWS CloudFormation to configure the CloudWatch metric steam and Kinesis Data Firehose to send metrics. It also creates an IAM role for Cloud Observability in order to enrich the metric data with resource tags.

  1. In Cloud Observability, click Settings > AWS integration > Create an integration.

  2. Click Generate and copy API key. This API key is exclusively for all AWS integrations and is automatically copied to your clipboard. Box highlights Generate and copy API key button in Cloud Observability.

  3. Click Go to AWS CloudFormation.

    Log into your AWS account, if not already logged in. You’re taken to the Quick create stack page in CloudFormation where you can continue the integration. Your AWS resources will be created in whatever region you run the stack from, so ensure that you are in your desired region. CloudFormation page has many sections, including template, stack name, and parameters.

  4. Enter the copied API key into the Cloud Observability API Key field.

  5. For Lightstep instance, select the appropriate region for your organization (either US or EU).

    Cloud Observability generates default values for the other fields. You can change most of them as needed, but they must be unique for the project.

    Don’t change the Cloud Observability Project or Organization fields. These are tied to the API key and will break the integration if changed.

  6. At the bottom of the page, acknowledge the creation of IAM resources, and click Create stack. Box highlights the check box and Create stack button in AWS.

  7. Once data is flowing into Cloud Observability, click Create an AWS dashboard and choose the integration you need.

    Pre-built dashboards are not available for all AWS integrations. You can also use Terraform to create your dashboards.

    The dashboard is added to your project. You can edit the dashboard as needed, for example, to add more labels, rearrange the charts, edit, add and delete charts, and more.

The integration now appears on the AWS Integrations page. Table shows AWS account IDs, stack names, and created-on dates for integrations.

You can verify that the metrics are reporting by visiting the Metric details page in Settings.

Add additional AWS integrations

Your CloudWatch metric stream will only send metrics emitted from the same AWS region and account that the stream exists in. To stream data from other AWS regions and/or accounts, navigate to your desired AWS account and region, and repeat the previous steps to create another integration.

Once you have metrics reporting, you can read about the actual metrics ingested here.

Update an integration

You can update or delete any integration in CloudFormation. Click the More ( ⋮ ) icon and choose View in CloudFormation. You can make any changes there, including deleting the integration. Box highlights the Go to CloudFormation option.

Notes

  • Percentiles are not included in CloudWatch Metric Streams.
  • CloudWatch Metric Streams do not include metrics with timestamps older than two hours - for example, metrics that are aggregated on a daily basis.
  • Tags are refreshed at a five-minute interval.

If you have previously integrated your CloudWatch metrics with Cloud Observability prior to the release of our metric streams integration, we strongly recommend transitioning to the streaming integration. It is lower latency, more reliable, and cheaper! Please contact your Technical Account Manager who will be happy to guide you through the upgrade.

See also

Create and manage dashboards

Create alerts

Troubleshoot missing data in Cloud Observability

Updated Jun 4, 2024