Scaling on AWS ECS Fargate for application and events with SQS & Lambda

Introduction

AWS provides a set of fully managed services that can be used to make a serverless application. Know more about how we used some of these services to set up a fully managed, scalable video processing application on AWS and how we did scaling on AWS ECS Fargate for application and events with SQS & Lambda

IN this article we discuss how to create a serverless application for long running processing tasks using AWS Lambda, SQS and ECS Fargate. While fargate does provide time and cloud watch-based task scheduling, it doesn’t provide a way to trigger the task creation based on application event.

Our service receives some events in form of SQS messages, this event consists of a URL of video which needs to be processed using machine learning algorithm.
This problem statement can further be broken down into following tasks:

1. Uploading video to the cloud
2. Passing the uploaded video information for further processing.
3. Launching Machine learning tasks with the received information in required compute engines.

The article has below sections to explain on how you could use scaling on AWS ECS Fargate for application and events with SQS & Lambda

1. Architecture and design
2. Steps to Implement
   • Setting up Fargate based task Definition
   • Integrating the SQS and ECS service with Help of AWS lambda
   • Write code for lambda Function
   • Create a jar for application code to upload the Lambda Function
   • Settings, Update, debug and test the application

Architecture and Design for Scaling on AWS ECS Fargate with SQS & Lambda

Following AWS services were used to meet above use cases and launch this application:

1. AWS Simple Storage Service (S3) : Videos uploaded are stored on AWS S3 (Reference)
2. AWS Simple Queue Service (SQS): The task of passing the information related to uploaded video like the video url, video id is done with the help of this service (Reference)
3. AWS Elastic Container Service (ECS) and AWS Fargate: ECS is a fully managed container orchestration service,  AWS Fargate is a serverless compute for running containers, it removes the need to provision and manage servers, lets you specify and pay for resources used while running the container, and improves security through application isolation by design (Reference). For our use-case we created a docker image for running our machine learning models and ECS cluster along with Fargate helped to set up the required computing component with the advantage of not having to worry about any server management.
4. AWS Lambda: The task of passing the received SQS message and launching a task in Fargate was achieved with the help of this service. It lets you run code without provisioning or managing servers. You pay only for the compute time you consume (Reference)

Our service overall architecture looked like this:

Steps to Implement Scaling on AWS ECS Fargate with SQS & Lambda

Below are the steps to implement the system assuming ECS with Fargate is already configured. So, we are starting from the step where you need to setup a Task Definition.

Setting up Fargate based Task Definition

Step 1: Select launch type compatibility:

We selected Fargate based setup for our ECS tasks since it provides the option to run containers without having to manage servers

Step 2: Configure task and container definition

Provide the task definition details and create a new task definition in AWS ECS service

• Provide basic task details like name, task role.

• Provide the task execution role and the memory, CPU requirements

• Provide the container image to run on launching the task

Integrating the SQS and ECS service with the help of AWS lambda and add a Trigger

Create SQS queue which will receive the event messages. Whenever a video gets uploaded on S3, ensure that all its required meta information is pushed to this queue as well. (Reference)

Once that’s done follow below steps to create AWS lambda with SQS trigger

Step 1: Choose runtime and create lambda function
There are multiple runtimes available for this, we created a java application to perform required action.

Note in this step you also provide the execution role to the lambda function, for this you need to ensure that the service role provided has access to following three policies:

Step 2: Once you create the function, the next step is to add the trigger for it.

There are multiple trigger options available in lambda, we went with SQS for our setup. Multiple triggers can also be added to a single lambda function

Write code for lambda function.

We created a maven-based java project to perform the task for us. Following are the steps required to setup the code for your use-case :

A. Ensure the entry point in your lambda application extends from RequestHandler<I, O> which is a Lambda request handler, here “I” refers to input parameter and “O” refers to output parameter.
In our case “I” is SQSEvent and “O” is Void.
This interface provides a handleRequest function which needs to be implemented.

Here we don’t need to explicitly write any code for receiving messages from sqs and deleting those messages. This will be automatically handled by the linking we did in step 2. However, the point to note is that the time it takes for your message to process should be less than or equal to the visibility timeout of the message in the queue.

public class Application implements RequestHandler<SQSEvent, Void> 

B. Write the function to parse message received from sqs

public Void handleRequest(SQSEvent sqsEvent, Context context) {
   for (SQSEvent.SQSMessage msg : sqsEvent.getRecords()) {
       JSONObject inputJSON;
       if (msg == null) {
           System.out.println("HandleRequest : Empty message received");
           continue;       }
       try {
           inputJSON = new JSONObject(msg.getBody());
           System.out.println(" HandleRequest: Msg Id " + msg.getMessageId()
                             +" Message JSON Object is: " + inputJSON.toString());
           sendRequestForEcsTask(inputJSON);
       } catch (JSONException e) {
           System.out.println(" HandleRequest: Failed to create JSON Object " + e.toString() + " , msg "
                              + msg.toString());       }
       finally {
           System.out.println(" HandleRequest : Finished processing Msg Id " + msg.getMessageId());
       }   }
   return null;}

C. Launch ECS task definition for the message received.

For this we used com.amazonAWS.services.ecs package to create the ecs client and provide the task request to it. Following are the set of inputs which were required:

• Amazon ECS Cluster Name
• Launch type (Fargate or EC2)
• ECS Task definition name
• Subnet Ids and security group for network configuration
• Container name
• Container environment key-value pairs

(i) Creating ecs task request instance and populating it with required options

* @param inputJSON Message received for processing
*/
private void sendRequestForEcsTask(JSONObject inputJSON) {
   //Read the keys from received sqs msg body
   String videoId = inputJSON.optString(VIDEO_ID_SQS_KEY);
   String videoName = inputJSON.optString(VIDEO_NAME_SQS_KEY);
   //Create the ecs task definition request
   RunTaskRequest runTaskRequest = new RunTaskRequest();
   runTaskRequest.setCluster(EcsTaskConstants.CLUSTER_NAME);
   runTaskRequest.setLaunchType(EcsTaskConstants.LAUNCH_TYPE);
   runTaskRequest.setTaskDefinition(EcsTaskConstants.TASK_DEFINITION);
  NetworkConfiguration networkConfiguration = getNetworkConfiguration();
   runTaskRequest.setNetworkConfiguration(networkConfiguration);
   TaskOverride taskOverride = getTaskOverride(videoId, videoName);
   runTaskRequest.setOverrides(taskOverride);
   //Create a ecs client
   AmazonECS client = AmazonECSClientBuilder.standard().build();
   RunTaskResult taskResult = client.runTask(runTaskRequest);
   System.out.println("RunTask Task result " + taskResult.toString());
}

(ii) Setting up network configuration for ecs task request

   NetworkConfiguration networkConfiguration = new NetworkConfiguration();
   AWSVpcConfiguration AWSVpcConfiguration = new AWSVpcConfiguration();
   AWSVpcConfiguration.setSubnets(EcsTaskConstants.SUBNET_IDS);   AWSVpcConfiguration.setSecurityGroups(EcsTaskConstants.SECURITY_GROUPS);
   networkConfiguration.setAWSvpcConfiguration(AWSVpcConfiguration);
   return networkConfiguration;
}

(iii) Setting up environment variables to be passed to the container when ecs task runs.

   TaskOverride taskOverride = new TaskOverride();
   ContainerOverride containerOverride = new ContainerOverride();
   containerOverride.setName(EcsTaskConstants.CONTAINER_NAME);
   KeyValuePair videoIdInfo = new KeyValuePair();
   videoIdInfo.setName(VIDEO_ID_KEY);
   videoIdInfo.setValue(videoId);
   KeyValuePair videoNameInfo = new KeyValuePair();
   videoNameInfo.setName(VIDEO_NAME_KEY);
   videoNameInfo.setValue(videoName);
   containerOverride.setEnvironment(Arrays.asList(videoIdInfo, videoNameInfo));
   taskOverride.setContainerOverrides(Collections.singletonList(containerOverride));
   return taskOverride;
}

Create a Jar for application code to upload to the Lambda function

Next step is to create a jar for your application code and upload it to the lambda function. This option is available in Function code.

Settings Update, Debug and testing of the setup

Update the basic settings like handler method, memory, timeout, description etc. for the function.

Debug and Test the application:

Once all the above things are done, try to send messages to your queue and check in cloudwatch logs and in ECS Tasks whether required tasks has been triggered or not.

As shown below ECS tasks are triggered.

Summary & Benefits on how to use Scaling on AWS ECS Fargate with SQS & Lambda

To summarize, this architecture helped us to achieve serverless design goal by running long running tasks in ECS Fargate service in form of containers tasks on basis of application events. We only spin up container based on application events; we used SQS which triggers the lambda which in turn creates the container thus making complete solution serverless and scalable as needed.

Benefit achieved are:

1. Machine learning and container-based service is best optimized using ECS and Fargate for long running and compute intensive task
2. It provides a scalable infrastructure based on application events
3. It does not incur any cost if not task is running so cost is based on consumption, if this is compared to EC2 it would have cost for even an idle resource.

References:

Read more about Simple storage services

https://aws.amazon.com/s3/

Read more about Simple Queue services

https://aws.amazon.com/sqs/

Read more about ECS

https://aws.amazon.com/ecs/

Read more about Lambda functions

https://aws.amazon.com/lambda/

Integrating the SQS and ECS service with the help of AWS lambda and add a Trigger

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-create-queue.html