Ingest logs from Amazon ECS using AWS firelens
In order to send logs from
tasks running in ECS (on fargate and ec2 for linux) to OpenObserve, AWS firelens is the recommended mechanism. AWS firelens is a log router for Amazon ECS that sends log data from containers running in ECS tasks to fluentbit (or fluentd) sidecar container which can then send data to wherever fluentbit supports sending data. A sidecar container is simply an additional container running along side the main container in a task that performs some ancillary services - e.g. collecting logs, keep configuration up to date, etc. We recommend that you use fluentbit instead of fluentd due to its much lower resource requirements.
If you have existing ECS tasks from which you need to send logs to OpenObserve, then you will need to modify their task definition to add fluentbit sidecar. Let's take a look at how to accomplish this.
- A OpenObserve Cloud account or a OpenObserve self hosted setup.
- A running ECS cluster that supports fargate. If you don't already have one, create one by following the documentation.
We will run our tasks using fargate for this demonstration.
Get OpenObserve Cloud / OpenObserve Configuration
Before you can start with setting up the configuration of your ECS task you will need the details of your OpenObserve where you will send the logs.
You can either use a self hosted OpenObserve or OpenObserve Cloud for following this guide. You can get started with OpenObserve Cloud for free at https://cloud.openobserve.ai that has a generous free tier.
You can find the config details under ingestion/fluentbit
You can use the configuration details from this section in your task definition
Create ECS task definition
Create the following file and save it as "nginx_firelens_zo_task_def.json"
logDriver in this case is
awsfirelens. All the logs for nginx container will be sent to fluentbit using
options section has http output plugin configuration for fluentbit. Configure this section with the values you got from OpenObserve.
Register the task definition using the below command:
We will also need to provide network configuration when using networkMode as
awsvpc during service creation. Let's create a json file for that:
|Network configuration - network_config.json|
Do not set
assignPublicIp on line
10 as ENABLED for real world scenarios. You do not want to access tasks directly. We are doing this now only for demonstration. You should always either front the services with a load balancer or AWS
Create ECS service
Now let's create an ECS
service that will use this task definition.
Assuming the name of your cluster -
ecs1_fargate_cluster1 run below command:
If all goes well, you should see a running service on ECS console:
Now click on the tasks tab. You should see a task running as part of the service:
Now click on the task:
You should see the
Public IP for the task.
open address. You should see the following page:
Now head on the OpenObserve / OpenObserve Cloud and see the logs flowing in there.
AWS firelens provides an easy way to send ECS container logs to OpenObserve. We configured AWS firelens in few steps to send logs to OpenObserve / OpenObserve Cloud. to easily view and analyze logs.