AWS provides a set of fully managed services that can be used to make a serverless application. Know more about how we used some of these services to set up a fully managed, scalable video processing application on AWS and how we did scaling on AWS ECS Fargate for application and events with SQS & Lambda
IN this article we discuss how to create a serverless application for long running processing tasks using AWS Lambda, SQS and ECS Fargate. While fargate does provide time and cloud watch-based task scheduling, it doesn’t provide a way to trigger the task creation based on application event.
Our service receives some events in form of SQS messages, this event consists of a URL of video which needs to be processed using machine learning algorithm.
This problem statement can further be broken down into following tasks:
The article has below sections to explain on how you could use scaling on AWS ECS Fargate for application and events with SQS & Lambda
Following AWS services were used to meet above use cases and launch this application:
Our service overall architecture looked like this:
Below are the steps to implement the system assuming ECS with Fargate is already configured. So, we are starting from the step where you need to setup a Task Definition.
Step 1: Select launch type compatibility:
We selected Fargate based setup for our ECS tasks since it provides the option to run containers without having to manage servers
Step 2: Configure task and container definition
Provide the task definition details and create a new task definition in AWS ECS service
Create SQS queue which will receive the event messages. Whenever a video gets uploaded on S3, ensure that all its required meta information is pushed to this queue as well. (Reference)
Once that’s done follow below steps to create AWS lambda with SQS trigger
Step 1: Choose runtime and create lambda function
There are multiple runtimes available for this, we created a java application to perform required action.
Note in this step you also provide the execution role to the lambda function, for this you need to ensure that the service role provided has access to following three policies:
Step 2: Once you create the function, the next step is to add the trigger for it.
There are multiple trigger options available in lambda, we went with SQS for our setup. Multiple triggers can also be added to a single lambda function
We created a maven-based java project to perform the task for us. Following are the steps required to setup the code for your use-case :
A. Ensure the entry point in your lambda application extends from RequestHandler<I, O> which is a Lambda request handler, here “I” refers to input parameter and “O” refers to output parameter.
In our case “I” is SQSEvent and “O” is Void.
This interface provides a handleRequest function which needs to be implemented.
Here we don’t need to explicitly write any code for receiving messages from sqs and deleting those messages. This will be automatically handled by the linking we did in step 2. However, the point to note is that the time it takes for your message to process should be less than or equal to the visibility timeout of the message in the queue.
public class Application implements RequestHandler<SQSEvent, Void> |
B. Write the function to parse message received from sqs
/* public Void handleRequest(SQSEvent sqsEvent, Context context) { |
C. Launch ECS task definition for the message received.
For this we used com.amazonAWS.services.ecs package to create the ecs client and provide the task request to it. Following are the set of inputs which were required:
(i) Creating ecs task request instance and populating it with required options
/*** Create a ecs task request and run it on the ecs client * @param inputJSON Message received for processing |
(ii) Setting up network configuration for ecs task request
private NetworkConfiguration getNetworkConfiguration() { NetworkConfiguration networkConfiguration = new NetworkConfiguration(); |
(iii) Setting up environment variables to be passed to the container when ecs task runs.
private TaskOverride getTaskOverride(String videoId, String videoName) { TaskOverride taskOverride = new TaskOverride(); |
Next step is to create a jar for your application code and upload it to the lambda function. This option is available in Function code.
Update the basic settings like handler method, memory, timeout, description etc. for the function.
Debug and Test the application:
Once all the above things are done, try to send messages to your queue and check in cloudwatch logs and in ECS Tasks whether required tasks has been triggered or not.
As shown below ECS tasks are triggered.
To summarize, this architecture helped us to achieve serverless design goal by running long running tasks in ECS Fargate service in form of containers tasks on basis of application events. We only spin up container based on application events; we used SQS which triggers the lambda which in turn creates the container thus making complete solution serverless and scalable as needed.
Benefit achieved are:
Read more about Simple storage services
Read more about Simple Queue services
Read more about ECS
Read more about Lambda functions
https://aws.amazon.com/lambda/
Integrating the SQS and ECS service with the help of AWS lambda and add a Trigger
This Article talks about how you could use SAP S/4 Hana to Modify Email template and move changes to production.
This Article talks about how you could use SAP S/4 Hana to Modify Email template and move changes to production.
This article talks about how you could use cloud services for scaling on AWS ECS Fargate for applications and events with SQS & Lambda.