Monitor Amazon ECS logs using AWS FireLens

Introduction to AWS FireLens

Are you struggling to keep tabs on your Amazon ECS container logs? You're not alone. AWS FireLens offers a powerful solution for routing your ECS logs to various destinations, streamlining your monitoring and troubleshooting processes.

This article dives into using FireLens to gain valuable insights into your containerized applications running on ECS. From understanding its core functionalities to configuration steps, we'll equip you to leverage FireLens and achieve comprehensive log management for your ECS deployments.

Let's dive into the world of FireLens, a log router for Amazon ECS and AWS Fargate.

FireLens is a log router that helps you collect, process, and route logs from your applications running on Amazon ECS and AWS Fargate. It's designed to simplify log management by providing a centralized way to collect and route logs from your applications to various destinations, such as CloudWatch Logs, Splunk, or ELK.

FireLens plays a crucial role in log routing and management by providing a centralized way to collect and route logs from your applications. It helps you:

Collect logs: FireLens collects logs from your applications running on ECS and Fargate.
Process logs: FireLens can process logs using built-in filters and transformations.
Route logs: FireLens routes logs to various destinations, such as OpenObserve, CloudWatch Logs, Splunk, or ELK.

In summary, FireLens is a powerful tool for AWS log routing and management.

AWS FireLens provides an easy way to send ECS container logs to OpenObserve.

Book a Free Demo with OpenObserve

Benefits and Capabilities of FireLens

Here are the benefits and capabilities of FireLens.

Simplified Interface

FireLens provides a simplified interface for log filtering, metadata addition, and routing, making it easy to manage and process logs.

Direct Streaming

FireLens allows for direct logs streaming to destinations like OpenObserve, Amazon CloudWatch, Amazon Elasticsearch Service, S3, and Kinesis Data Streams, ensuring efficient and reliable log management.

Flexibility

FireLens offers flexibility through ECS task definition parameters, allowing users to configure destinations and routing rules according to their specific needs.

Partnership Integrations

FireLens integrates with various partners to provide enhanced log management solutions, including AWS services like CloudWatch and Elasticsearch, as well as third-party tools like OpenObserve, Splunk and ELK.

FireLens Architecture

The FireLens architecture consists of the following components:

FireLens Agent: This is the core component that runs on the host where the logs are generated. It captures the logs and sends them to the FireLens configuration.

FireLens Configuration: This is the configuration file that defines how the logs should be processed and forwarded. It includes settings for log processing, filtering, and forwarding.

FireLens Processor: This component processes the logs according to the configuration. It can perform tasks such as log filtering, parsing, and transformation.

FireLens Forwarder: This component forwards the processed logs to the specified destination, such as OpenObserve, Amazon S3, Amazon Elasticsearch, or Amazon CloudWatch Logs.

Utilizing FireLens in Amazon ECS

The following points help you utilize Firelens in Amazon ECS.

Use with AWS Services or APN Destination: FireLens can be used with AWS services for enhanced log storage and analytics. It can also be used with APN (Amazon Partner Network) destinations for more advanced analytics and integration.

Name Log Container and Start Order Dependency: This ensures that the log container is started before the application container, allowing for proper log capture and forwarding.

IAM Roles and Permissions: This includes setting up roles for the FireLens agent and configuring permissions for access to the desired AWS services.

Centralized Logging Solutions for Containers: FireLens can be used with OpenObserve, a popular open-source logging agent, to provide centralized logging solutions for containers, including Windows-based containers. This allows for efficient log collection, processing, and forwarding from containers to various destinations.

AWS fireLens provides an easy way to send ECS container logs to OpenObserve.

Book a Free Demo with OpenObserve

Security and Networking Considerations

Here are some key security and networking considerations for AWS FireLens:

Security Considerations

Restricting Inbound Traffic: It is essential to restrict inbound traffic to FireLens on port 24224 to prevent unauthorized access. This can be achieved by configuring security groups to allow only necessary traffic to reach FireLens.
Security Group Configurations: The security group configurations for different ECS network modes are crucial for ensuring secure operations. For example, in the awsvpc mode, you need to configure the security group to allow traffic from the VPC to reach FireLens. In the bridge mode, you need to configure the security group to allow traffic from the bridge to reach FireLens.
Dependency Conditions in Task Definitions: Dependency conditions in task definitions can be used to ensure secure operations. For example, you can specify that a task should only start if a specific condition is met, such as the availability of a specific resource or the completion of a specific task.

Networking Considerations

Network Isolation: Ensure that FireLens is deployed in a network-isolated environment to prevent unauthorized access.
Network Segmentation: Use network segmentation to isolate FireLens from other components of your application and prevent unauthorized access.
Network Monitoring: Monitor network traffic to detect and respond to potential security threats.

Configuring FireLens for Advanced Use Cases

Configuration Options

FireLensConfiguration

Type: FireLensConfiguration (required)
LogConfiguration:
LogGroup (required): The name of the log group where logs will be sent.
LogStream (optional): The name of the log stream where logs will be sent.
Format (optional): The format of the logs. Supported formats include JSON, JSON Lines, and Text.

CloudWatchLogs
Enabled (optional): Whether CloudWatch logs are enabled. Default is true.

CloudWatchMetrics
Enabled (optional): Whether CloudWatch metrics are enabled. Default is true.
Region (optional): The AWS region where logs will be sent. Default is the region of the task.

Example Configurations for Routing Logs

yaml
task_definition:
  container_definitions:
    - name: my-container
      image: my-image
      firelens_configuration:
        type: FireLensConfiguration
        options:
          log_configuration:
            log_group: my-log-group
            log_stream: my-log-stream
            format: JSON
          cloud_watch_logs:
            enabled: true
          cloud_watch_metrics:
            enabled: true
          region: us-west-2
          region: eu-west-1

Monitor ECS Logs with Integration Platforms

To monitor ECS logs with integration platforms, you can use AWS FireLens with platforms like OpenObserve. Here are the key steps:

Configure AWS FireLens

Set the awsfirelens log driver in your ECS task definition to use FireLens for log forwarding.
Configure the logConfiguration in the task definition to send logs to Amazon CloudWatch.

Forward logs to external platforms

Use a Lambda function to collect logs from CloudWatch and forward them to the desired monitoring platform, such as OpenObserve.
The Lambda function acts as a bridge to send logs from CloudWatch to the external platform of your choice.

Automate log routing

Configure the ECS task definition to use the awsfirelens log driver.
Set the logConfiguration to send logs to CloudWatch.
The Lambda function automatically collects logs from CloudWatch and forwards them to the external monitoring platform, providing an automated log routing solution.

Watch this Video on Youtube: Monitor Performance of Amazon ECS Applications

OpenObserve: Ingest logs from Amazon ECS using AWS Firelens

Ingest logs from Amazon ECS using AWS Firelens

Image Credit: Open Observe

Getting Started with FireLens

To get started with FireLens, the key steps are:

Identify the log locations for your container by updating the logLocations array in your analysis.json file and enabling dynamic logging.
Configure the log deployment parameters in your deployment.json file, including setting enableFirelensLogging to true and specifying the desired log destinations (e.g. CloudWatch, Firehose, Kinesis) with the appropriate regex filters and stream names.

Are you a Developer?

Join OpenObserve on Github

Best Practices

Use custom log parsers with FireLens to structure your logs and enable more efficient querying at the destination.
Leverage FireLens integrations with observability platforms like OpenObserve, Splunk and SignalFx to gain real-time visibility and alerting from your container logs.

Examples for deploying FireLens

Here are some examples for deploying FireLens:

Routing Logs to Amazon Kinesis Data Firehose

json
{
  "family": "firelens-example-firehose",
  "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role",
  "containerDefinitions": [
    {
      "essential": true,
      "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:stable",
      "name": "log_router",
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "enable-ecs-log-metadata": "true"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "firelens-container",
          "awslogs-region": "us-west-2",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "firelens"
        }
      },
      "memoryReservation": 50
    },
    {
      "essential": true,
      "image": "httpd",
      "name": "app",
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "firehose",
          "region": "us-west-2",
          "delivery_stream": "my-stream",
          "log-driver-buffer-limit": "2097152"
        }
      },
      "memoryReservation": 100
    }
  ]
}

This example defines a log router container using Fluent Bit to send logs to Amazon Kinesis Data Firehose. The application container uses the awsfirelens log driver to route its logs to Firehose.

Are you a Developer?

Join OpenObserve on Github

Routing Logs to Amazon CloudWatch Logs

json
{
  "family": "firelens-example-cloudwatch",
  "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role",
  "containerDefinitions": [
    {
      "essential": true,
      "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:stable",
      "name": "log_router",
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "enable-ecs-log-metadata": "true"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "firelens-container",
          "awslogs-region": "us-west-2",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "firelens"
        }
      },
      "memoryReservation": 50
    },
    {
      "essential": true,
      "image": "httpd",
      "name": "app",
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "cloudwatch",
          "region": "us-west-2",
          "log_group_name": "firelens-cloudwatch",
          "log_stream_prefix": "firelens-"
        }
      },
      "memoryReservation": 100
    }
  ]
}

This example is similar to the previous one, but it routes logs from the application container to Amazon CloudWatch Logs instead of Firehose.

Ingest logs from Amazon ECS using AWS firelens

To configure Firelens to send logs to OpenObserve, you need to modify your ECS task definition to add the fluentbit sidecar. Here is an example task definition:

json
{
  "taskDefinition": {
    "family": "nginx_firelens_zo1",
    "taskRoleArn": null,
    "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "containerDefinitions": [
      {
        "name": "nginx",
        "image": "nginx:latest",
        "essential": true,
        "firelensConfiguration": {
          "type": "fluentbit",
          "options": {
            "config-file-type": "file",
            "config-file-value": "/fluent-bit.conf",
            "enable-ecs-log-metadata": "false"
          }
        },
        "logConfiguration": {
          "logDriver": "awsfirelens",
          "options": {
            "Name": "http",
            "Match": "*",
            "Host": "openobserve.example.com",
            "Port": "8080",
            "URI": "/api/v1/logs",
            "Header": "X-Openobserve-Token:abc123"
          }
        }
      }
    ],
    "volumes": [],
    "placementConstraints": [],
    "requiresCompatibilities": [
      "FARGATE"
    ],
    "cpu": "256",
    "memory": "512"
  }
}

This configures the Firelens log driver and specifies the HTTP output plugin configuration for fluentbit to send logs to OpenObserve.

OpenObserve makes it easy to ingest logs from Amazon ECS containers using AWS Firelens. By modifying the ECS task definition to add a fluentbit sidecar and configure it to send logs to OpenObserve, you can easily view and analyze your container logs in OpenObserve.

Are you a Developer?

Join OpenObserve on Github

Conclusion

The ever-growing volume of logs from your Amazon ECS containers can quickly become overwhelming. AWS FireLens and integration platforms like OpenObserve unlocks a world of streamlined log management, empowering you to gain valuable insights into your containerized applications.

Now, you are equipped with the basics of using AWS FireLens to Monitor Amazon ECS and integration platforms, transforming your ECS log management from a chaotic storm to a well-oiled machine, providing clear visibility into your containerized applications.

For more details: Contact Team OpenObserve

Where OpenObserve Shines

OpenObserve steps in as a powerful backend solution, seamlessly integrating with FireLens to empower you with advanced log management capabilities:

Storage and Scalability: OpenObserve boasts a highly scalable architecture, efficiently storing and managing your container logs regardless of their volume.
Advanced Analytics: OpenObserve goes beyond simple log storage. Leverage its powerful analytics tools to unearth patterns, identify trends, and diagnose issues within your containerized applications.
Real-time Visibility: Gain instant insights into your container health with OpenObserve's real-time log visualization. Quickly pinpoint anomalies and identify potential problems before they escalate.
Cost-Effectiveness: OpenObserve offers a cost-effective solution compared to traditional log management options, keeping your monitoring expenses under control.
Incorporating OpenObserve with FireLens unlocks a comprehensive log management experience, providing a centralized platform for collecting, analyzing, and visualizing your ECS container logs.

This empowers you to make data-driven decisions, optimize your container performance, and ensure the smooth operation of your applications.

AWS Firelens provides an easy way to send ECS container logs to OpenObserve. We configured AWS firelens in a few steps to send logs to OpenObserve / OpenObserve Cloud. to easily view and analyze logs.

Book a Free Demo with OpenObserve

About Us | Open Source Observability Platform

Are you a Developer?

Join OpenObserve on Github

Additional Resources

Set up FireLens to send logs to CloudWatch Logs

Send Amazon ECS logs to an AWS service or AWS Partner

Ingest logs from Amazon ECS using AWS firelens