What is AWS CloudTrail? How to Monitor CloudTrail Logs and Data Events for Real-Time Insights
Chaitanya Sistla
November 23, 2024
6 min read
Don’t forget to share!
What is AWS CloudTrail?
AWS CloudTrail is a vital service for auditing and monitoring your AWS account activity (individual or across all the organizations). It records API calls, data events, user activities, and resource changes, providing visibility into your AWS environment. This ensures compliance, security, and operational efficiency.
Why is CloudTrail Important?
- Audit Trail: Helps maintain a record of all API calls made within your account, ensuring accountability.
- Security: Identifies unauthorized access attempts or unusual behavior, bolstering your cloud security.
- Compliance: Aids in adhering to regulatory requirements by maintaining detailed logs of actions in your account.
- Operational Insights: Offers visibility into resource usage and access patterns for better management.
How to Enable AWS CloudTrail?
Enabling CloudTrail in your AWS environment involves setting up trails to deliver logs to an S3 bucket for storage and further analysis.
Log in to AWS Console: Navigate to the CloudTrail service.
Create a Trail:
- Click on Trails and then Create trail.
- Provide a name for your trail.
S3 Bucket for Storage:
- Select an existing S3 bucket or create a new one for storing CloudTrail logs.
- Configure permissions to allow CloudTrail to write logs to the bucket.
- It is recommended to select Enable for all accounts in my organization for a more centralized management.
Enable Log Events:
- Choose to log management events, data events, or both.
- Optionally include read and write events for detailed insights.
Save and Activate: Once configured, activate the trail.
Your logs will now be delivered to the specified S3 bucket, ready for further processing and analysis.
Using a CloudFormation Template for Observability Pipeline
You can download the cloudformation template from github by clicking this line.
The CloudFormation template automates the setup by creating the following resources:
- IAM Roles:
- An IAM role for Kinesis Firehose with permissions to access S3 buckets and send data to a specified HTTP endpoint.
- A Lambda execution role with permissions to process logs from S3 and send them to Kinesis Firehose.
- Lambda Function:
- Automatically triggered when CloudTrail logs are added to the S3 bucket.
- Processes and extracts individual records from the logs.
- Sends the processed records to a Kinesis Firehose delivery stream.
- Kinesis Firehose:
- A delivery stream configured to send CloudTrail logs to an HTTP endpoint (like OpenObserve) for analysis.
- Supports buffering, retries, and logging of delivery operations.
- Backs up any failed data to a secondary S3 bucket.
- S3 Notification:
- Configures the S3 bucket to notify the Lambda function whenever new logs are created.
How to Use the Template
⚠️ Important Note ⚠️
The provided CloudFormation stack assumes that CloudTrail logs are already enabled in your AWS account. You will need to provide the name of the S3 bucket where CloudTrail logs are stored. Setting up and enabling CloudTrail as part of the CloudFormation stack is outside the scope of this blog.
The stream name in the HTTP endpoint should match aws_orgs_cloudtrail or you will not see data in the dashboard. Alternatively, ypu can also replace aws_orgs_cloudtrail with your custom stream name after downloading the dashboard if you want to use a different stream name
Deploy the Template:
Upload the CloudFormation template to your AWS account.
Provide parameters such as:
- The HTTP endpoint name and URL.
- The access key for the HTTP endpoint.
- Names of the CloudTrail S3 bucket and backup bucket.
Verify the Deployment:
- Ensure the IAM roles, Lambda function, Kinesis Firehose, and S3 configurations are created successfully.
Enable trigger for Lambda:
- Cloudformation does not support enabling trigger on existing bucket so you will need to add the trigger manually as below.
Analyzing Logs in OpenObserve
Once the above steps are completed, logs are ingested into OpenObserve, you can visualize and analyze them using dashboards. You can download the dashboards here.
Key Features of the Dashboards
- Log Analysis:
- View detailed API activity logs with filters for event names, users, and resources.
- Security Monitoring:
- Track unusual activity patterns or unauthorized access attempts.
- Compliance Reporting:
- Generate reports for audits with specific event details.
- Resource Insights:
- Monitor resource creation, deletion, and usage trends.
Benefits of the Automated Setup
- Efficiency: Automates the log ingestion pipeline from CloudTrail to OpenObserve.
- Scalability: Handles large volumes of logs seamlessly.
- Insights: Provides a real-time view of your AWS environment.
Comparing AWS CloudTrail Logs: With vs. Without Enhanced Observability
| Feature | Without OpenObserve | With OpenObserve |
|---|---|---|
| Log Storage | Stored in S3 with basic search capabilities | Centralized in OpenObserve with advanced indexing and search |
| Real-Time Monitoring | Not available, relies on manual log analysis | Real-time log ingestion, monitoring, and visualization |
| Search Capabilities | Limited to S3 and CloudTrail console filtering | Full-text search, filtering, and custom queries |
| Data Correlation | Requires manual efforts and external tools | Correlate logs and data events seamlessly in dashboards |
| Alerting and Notifications | Requires custom scripts or additional AWS services | Integrated real-time alerts based on log patterns |
| Visualization and Insights | Requires external visualization tools like QuickSight | Built-in dashboards and panels for actionable insights |
| Scalability | Limited by S3 and manual processing pipelines | Scales efficiently for large volumes of logs |
| Security Incident Response | Slower due to manual log retrieval and analysis | Faster with real-time alerts and context-rich dashboards |
Achieve AWS Security Goals with OpenObserve
AWS CloudTrail is indispensable for monitoring and securing your AWS environment. By leveraging a CloudFormation template, you can automate the setup of an efficient log pipeline to OpenObserve. This not only saves time but also provides a powerful way to analyze and visualize your AWS activities. With detailed dashboards and log analysis, you can ensure compliance, detect anomalies, and optimize your cloud operations. Get started with OpenObserve and gain control of your AWS cloudtrail logs.
