AWS ALB Logs Monitoring: Complete Setup and Real-Time Insights
Chaitanya Sistla
January 24, 2025
6 min read
Don’t forget to share!
What is AWS ALB?
AWS Application Load Balancer (ALB) is a fully managed service provided by Amazon Web Services (AWS) that automatically distributes incoming application traffic across multiple targets, such as EC2 instances, containers, and IP addresses. It operates at the application layer (Layer 7 of the OSI model) and provides advanced request routing capabilities based on HTTP/HTTPS headers, paths, and hostnames. ALB also supports WebSockets, gRPC, and enhanced security features, making it a preferred choice for modern applications.
Key features of AWS ALB:
- Intelligent Traffic Routing: Routes requests based on URL paths, headers, and cookies.
- High Availability & Scalability: Automatically scales based on incoming traffic.
- Integrated Security Features: Works with AWS WAF and supports TLS termination.
- Access Logging & Monitoring: Provides detailed access logs stored in S3 for auditing and analysis.
- Support for Microservices & Containers: Easily integrates with ECS, EKS, and Fargate for containerized applications.
Why Monitor AWS ALB?
Monitoring AWS ALB logs is critical for several reasons:
Traffic Analysis
- Gain insights into request patterns, sources, and geographic distribution.
- Identify high-traffic endpoints and optimize performance accordingly.
Security & Compliance
- Detect and prevent malicious activities, such as DDoS attacks and unauthorized access.
- Meet compliance requirements by storing detailed logs for audits.
Performance Optimization
- Identify latency and response time issues affecting end-user experience.
- Detect and troubleshoot backend service failures.
Cost Management
- Analyze request patterns to optimize ALB scaling and reduce unnecessary costs.
- Identify inefficient load balancing that may lead to excessive AWS expenses.
Prerequisites
Before you begin setting up AWS ALB log monitoring, ensure you have the following:
- Running AWS ALB: You must have an Application Load Balancer (ALB) already set up in your AWS environment.
- OpenObserve (Self-Hosted or Cloud): OpenObserve should be available for log ingestion, either as a self-hosted instance or via the OpenObserve Cloud.
- Required Access in AWS: Ensure you have the necessary AWS permissions, including:
- Permissions to create and manage S3 buckets for log storage.
- CloudFormation access to deploy the automation template.
- IAM Role Management to assign permissions for log forwarding.
- Access to AWS ALB Console to enable access logging.
Steps to Configure AWS ALB to Ingest Logs to HTTP Endpoint in Under 5 Minutes
1. Downloading the CloudFormation Template
AWS provides an automated way to configure ALB log ingestion using a CloudFormation template. Download the pre-built template from: https://github.com/openobserve/cloudformation-templates/blob/main/aws_alb/alb.yaml
2. Configuring CloudFormation with Parameters
- Navigate to AWS CloudFormation in the AWS Management Console.
- Click Create Stack → With new resources (standard).
- Upload the downloaded
alb.yamltemplate. - Provide required parameters such as:
- ALB name
- Target bucket for storing logs
- IAM role for S3 access
- OpenObserve auth credentials
- For ELBAccountId, you need to follow https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html
- Click Next and proceed with default settings unless customization is needed.
- Click Create Stack.
3. Validating CloudFormation Deployment
- Once deployed, go to CloudFormation Stacks and ensure the status is
CREATE_COMPLETE.
- Navigate to S3 and verify that the created bucket exists.
- Check Lambda function.
4. Enabling Monitoring in AWS ALB
Once the S3 bucket is created via CloudFormation:
- Go to AWS ALB Console.
- Select the Application Load Balancer.
- Click on Edit load balancer attributes.
- Under Access Logs and Connection Logs, enable logging and select the S3 bucket created and add the prefix that you provided during the CloudFormation steps.
- Save the configuration.
5. Verifying Logs in OpenObserve
Once logs are flowing to the S3 bucket, the lambda function will be invoked and start ingesting the logs to OpenObserve endpoint.
6. Adding VRL Functions in OpenObserve to Parse Logs
By default ALB logs are unstructured and are ingested as below.
"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
To enhance log analysis, add three VRL (Vector Remap Language) functions in OpenObserve for parsing:
- Parse logs using parse_aws_alb_log
VRL supports parse_aws_alb_log that helps transform the logs.
. = parse_aws_alb_log!(.log)
.
- Remove ports associated with IPs
.client_host = split!(.client_host, ":")[0]
.
- Fetch Geo-location from IPs
if exists(.client_host) && (is_ipv4(string!(.client_host)) || is_ipv6(string!(.client_host))) {
.geo_city_data = get_enrichment_table_record!("maxmind_city", {"ip": string!(.client_host)})
if exists(.geo_city_data) {
.geo_city = .geo_city_data.city
.geo_country = .geo_city_data.country
.geo_latitude = .geo_city_data.latitude
.geo_longitude = .geo_city_data.longitude
}
.geo_asn_data = get_enrichment_table_record!("maxmind_asn", {"ip": string!(.client_host)})
if exists(.geo_asn_data) {
.geo_asn = .geo_asn_data.asn
.geo_org = .geo_asn_data.organization
}
}
.
Finally create a pipeline that can transform your logs in real-time.
7. Uploading Pre-Built Dashboard
To visualize ALB logs effectively:
- Download the pre-built ALB dashboard from: https://github.com/openobserve/dashboards/tree/main/AWS_ALB
- Import it into OpenObserve’s Dashboard section.
- You must replace alb_demo inside the json file with the stream name that you have provided while configuring the HTTP endpoint in cloudformation deployment.
OpenObserve vs. Traditional ALB Log Monitoring
To understand the benefits of using OpenObserve for ALB log monitoring, let’s compare it with traditional methods:
| Feature | Without OpenObserve | With OpenObserve |
|---|---|---|
| Log Storage | Logs stored in S3, requiring additional processing | Real-time log ingestion with query capabilities |
| Data Parsing | Requires custom scripts to parse logs | Prebuilt VRL functions simplify parsing |
| Visualization | Needs third-party tools (e.g., Kibana, Grafana) | Built-in dashboards for instant insights |
| Alerting | No built-in alerting; needs external setup | Real-time alerts on anomalies |
Conclusion
Monitoring AWS ALB logs is crucial for security, performance optimization, and compliance. Traditional log monitoring methods often involve complex setups and additional tools for parsing and visualization. However, with OpenObserve, organizations can streamline log ingestion, parsing, and visualization in real time, reducing complexity and improving insights.
By leveraging CloudFormation templates, VRL functions, and pre-built dashboards, teams can efficiently monitor AWS ALB logs, ensuring high availability and proactive issue detection. Setting up AWS ALB log monitoring with OpenObserve takes just minutes but provides long-term benefits in operational efficiency, security, and cost optimization.
