OpenObserve Docs
IntegrationCloudAws

AWS CloudWatch Logs Monitoring - Centralized AWS Log Management

Monitor AWS CloudWatch Logs with comprehensive log streaming from CloudWatch to OpenObserve using Amazon Kinesis Firehose. This guide explains how to stream application logs and infrastructure logs from AWS CloudWatch for centralized log management, cloud observability, and real-time log analysis.

Overview

Forward AWS application logs and infrastructure logs from CloudWatch Logs to OpenObserve in near real-time for AWS monitoring and cloud observability. The CloudWatch Logs integration uses CloudWatch Logs subscriptions to stream log data into Amazon Kinesis Firehose, which delivers logs to OpenObserve over HTTP for centralized AWS log management.

This approach enables centralized log analysis, alerting, and visualization in OpenObserve without storing logs in S3 or managing Lambda transformations.

Steps to Integrate

Prerequisites
  • OpenObserve account (Cloud or Self-Hosted)
  • AWS account with:
    • CloudWatch Log Group(s)
    • IAM permissions to create Firehose streams and set up log subscriptions
Step 1: Get OpenObserve Ingestion URL and Access Key

  1. In OpenObserve: go to Data Sources → Recommended → AWS

  2. Copy the ingestion HTTP URL and Access Key

    Fetch OpenObserve Ingestion URL

Example format:

https://<your-openobserve-domain>/aws/default/cloudwatch-logs/_kinesis_firehose
Step 2: Create a Kinesis Firehose Delivery Stream

  1. In AWS Kinesis Firehose, Create delivery stream.

  2. Set Source: Direct PUT and Destination: HTTP Endpoint.

  3. Provide OpenObserve's HTTP Endpoint URL and Access Key, and set an S3 backup bucket.

  4. Give the stream a meaningful name and Create it.

    Kinesis Firehose Delivery Stream

Step 3: Attach CloudWatch Logs to Firehose

  1. Go to CloudWatch → Log Groups

  2. Select the log group you want to stream

  3. Click Actions → Create subscription filter

  4. Choose:

    • Destination: Kinesis Firehose
    • Delivery stream: the stream created in Step 2

  5. Set a filter pattern (e.g., "" to send all logs)

  6. Click Start streaming

    Attach CloudWatch Logs to Firehose

Step 4: Verify Logs in OpenObserve

  1. Go to Logs → select your log stream → Set time range → Click Run Query

    Verify Logs in OpenObserve

Troubleshooting

Not seeing logs in OpenObserve?

Check the following:

  • CloudWatch Logs
    • Ensure log events are being generated and the correct log group is selected.
    • Verify that the subscription filter is active.
  • Firehose
    • Confirm the delivery stream is active and not reporting errors.
    • Review the Monitoring tab for delivery status.
    • Check the backup S3 bucket (if configured) for failed records.
  • OpenObserve
    • Validate the HTTP ingestion URL and access key.
    • In the Logs view, select the correct stream and expand the time range.

Start Streaming Fails after creating Subscription Filter

Check the IAM role attached to the filter has full acess to Kinesis Firehose.

Was this page helpful?

Last updated on

AWS Network Firewall

Stream AWS Network Firewall flow and alert logs to OpenObserve via CloudWatch Logs and Kinesis Firehose for real-time network traffic monitoring.

AWS Cloudwatch metrics

Stream AWS CloudWatch metrics to OpenObserve via Metric Streams and Kinesis Data Streams for real-time infrastructure monitoring.

On this page

AWS CloudWatch Logs Monitoring - Centralized AWS Log ManagementOverviewSteps to Integrate