Scaling on AWS ECS Fargate for application and events with SQS & Lambda
Introduction
AWS provides a set of fully managed services that can be used to make a serverless application. Know more about how we used some of these services to set up a fully managed, scalable video processing application on AWS and how we did scaling on AWS ECS Fargate for application and events with SQS & Lambda
IN this article we discuss how to create a serverless application for long running processing tasks using AWS Lambda, SQS and ECS Fargate. While fargate does provide time and cloud watch-based task scheduling, it doesn’t provide a way to trigger the task creation based on application event.
Our service receives some events in form of SQS messages, this event consists of a URL of video which needs to be processed using machine learning algorithm. This problem statement can further be broken down into following tasks:
Uploading video to the cloud
Passing the uploaded video information for further processing.
Launching Machine learning tasks with the received information in required compute engines.
The article has below sections to explain on how you could use scaling on AWS ECS Fargate for application and events with SQS & Lambda
Architecture and design
Steps to Implement
Setting up Fargate based task Definition
Integrating the SQS and ECS service with Help of AWS lambda
Write code for lambda Function
Create a jar for application code to upload the Lambda Function
Settings, Update, debug and test the application
Architecture and Design for Scaling on AWS ECS Fargate with SQS & Lambda
Following AWS services were used to meet above use cases and launch this application:
AWS Simple Storage Service (S3) : Videos uploaded are stored on AWS S3 (Reference)
AWS Simple Queue Service (SQS): The task of passing the information related to uploaded video like the video url, video id is done with the help of this service (Reference)
AWS Elastic Container Service (ECS) and AWS Fargate: ECS is a fully managed container orchestration service, AWS Fargate is a serverless compute for running containers, it removes the need to provision and manage servers, lets you specify and pay for resources used while running the container, and improves security through application isolation by design (Reference). For our use-case we created a docker image for running our machine learning models and ECS cluster along with Fargate helped to set up the required computing component with the advantage of not having to worry about any server management.
AWS Lambda: The task of passing the received SQS message and launching a task in Fargate was achieved with the help of this service. It lets you run code without provisioning or managing servers. You pay only for the compute time you consume (Reference)
Our service overall architecture looked like this:
Steps to Implement Scaling on AWS ECS Fargate with SQS & Lambda
Below are the steps to implement the system assuming ECS with Fargate is already configured. So, we are starting from the step where you need to setup a Task Definition.
Setting up Fargate based Task Definition
Step 1: Select launch type compatibility:
We selected Fargate based setup for our ECS tasks since it provides the option to run containers without having to manage servers
Step 2: Configure task and container definition
Provide the task definition details and create a new task definition in AWS ECS service
Provide basic task details like name, task role.
Provide the task execution role and the memory, CPU requirements
Provide the container image to run on launching the task
Integrating the SQS and ECS service with the help of AWS lambda and add a Trigger
Create SQS queue which will receive the event messages. Whenever a video gets uploaded on S3, ensure that all its required meta information is pushed to this queue as well. (Reference)
Once that’s done follow below steps to create AWS lambda with SQS trigger
Step 1: Choose runtime and create lambda function There are multiple runtimes available for this, we created a java application to perform required action.
Note in this step you also provide the execution role to the lambda function, for this you need to ensure that the service role provided has access to following three policies:
Step 2: Once you create the function, the next step is to add the trigger for it.
There are multiple trigger options available in lambda, we went with SQS for our setup. Multiple triggers can also be added to a single lambda function
Write code for lambda function.
We created a maven-based java project to perform the task for us. Following are the steps required to setup the code for your use-case :
A. Ensure the entry point in your lambda application extends from RequestHandler<I, O> which is a Lambda request handler, here “I” refers to input parameter and “O” refers to output parameter. In our case “I” is SQSEvent and “O” is Void. This interface provides a handleRequest function which needs to be implemented.
Here we don’t need to explicitly write any code for receiving messages from sqs and deleting those messages. This will be automatically handled by the linking we did in step 2. However, the point to note is that the time it takes for your message to process should be less than or equal to the visibility timeout of the message in the queue.
public class Application implements RequestHandler<SQSEvent, Void>
B. Write the function to parse message received from sqs
/** * @param sqsEvent - Sqs event received from configured queue * @param context - Context of the request received * @return Handles the sqs event message received for triggering an ECS task */
C. Launch ECS task definition for the message received.
For this we used com.amazonAWS.services.ecs package to create the ecs client and provide the task request to it. Following are the set of inputs which were required:
Amazon ECS Cluster Name
Launch type (Fargate or EC2)
ECS Task definition name
Subnet Ids and security group for network configuration
Container name
Container environment key-value pairs
(i) Creating ecs task request instance and populating it with required options
/*** Create a ecs task request and run it on the ecs client
* @param inputJSON Message received for processing */ private void sendRequestForEcsTask(JSONObject inputJSON) { //Read the keys from received sqs msg body String videoId = inputJSON.optString(VIDEO_ID_SQS_KEY); String videoName = inputJSON.optString(VIDEO_NAME_SQS_KEY); //Create the ecs task definition request RunTaskRequest runTaskRequest = new RunTaskRequest(); runTaskRequest.setCluster(EcsTaskConstants.CLUSTER_NAME); runTaskRequest.setLaunchType(EcsTaskConstants.LAUNCH_TYPE); runTaskRequest.setTaskDefinition(EcsTaskConstants.TASK_DEFINITION); NetworkConfiguration networkConfiguration = getNetworkConfiguration(); runTaskRequest.setNetworkConfiguration(networkConfiguration); TaskOverride taskOverride = getTaskOverride(videoId, videoName); runTaskRequest.setOverrides(taskOverride); //Create a ecs client AmazonECS client = AmazonECSClientBuilder.standard().build(); RunTaskResult taskResult = client.runTask(runTaskRequest); System.out.println("RunTask Task result " + taskResult.toString()); }
(ii) Setting up network configuration for ecs task request
TaskOverride taskOverride = new TaskOverride(); ContainerOverride containerOverride = new ContainerOverride(); containerOverride.setName(EcsTaskConstants.CONTAINER_NAME); KeyValuePair videoIdInfo = new KeyValuePair(); videoIdInfo.setName(VIDEO_ID_KEY); videoIdInfo.setValue(videoId); KeyValuePair videoNameInfo = new KeyValuePair(); videoNameInfo.setName(VIDEO_NAME_KEY); videoNameInfo.setValue(videoName); containerOverride.setEnvironment(Arrays.asList(videoIdInfo, videoNameInfo)); taskOverride.setContainerOverrides(Collections.singletonList(containerOverride)); return taskOverride; }
Create a Jar for application code to upload to the Lambda function
Next step is to create a jar for your application code and upload it to the lambda function. This option is available in Function code.
Settings Update, Debug and testing of the setup
Update the basic settings like handler method, memory, timeout, description etc. for the function.
Debug and Test the application:
Once all the above things are done, try to send messages to your queue and check in cloudwatch logs and in ECS Tasks whether required tasks has been triggered or not.
As shown below ECS tasks are triggered.
Summary & Benefits on how to use Scaling on AWS ECS Fargate with SQS & Lambda
To summarize, this architecture helped us to achieve serverless design goal by running long running tasks in ECS Fargate service in form of containers tasks on basis of application events. We only spin up container based on application events; we used SQS which triggers the lambda which in turn creates the container thus making complete solution serverless and scalable as needed.
Benefit achieved are:
Machine learning and container-based service is best optimized using ECS and Fargate for long running and compute intensive task
It provides a scalable infrastructure based on application events
It does not incur any cost if not task is running so cost is based on consumption, if this is compared to EC2 it would have cost for even an idle resource.
Multiple Of Our Clients Are Transforming Their Businesses Every Day Using Cutting Edge IT Technology. Contact Our Experts And Start Your Own Journey Today.