AWS Custom Application Development Product Atmosolve Tonkin + Taylor

The project explores using AWS for custom app development using ESRI ArcGIS to read data and generate Hazard reports for a property that the user can search using Google Maps

Technologies

Frontend
React
Software Development

Project Types

Cloud Consulting

Industries

Land Engineering

Implementation

About Client

The customer’s (Tonkin + Taylor) business is involved in environmental consulting or meteorological services, focuses on providing high-resolution meteorological data for various applications, including air quality analysis, weather forecasting, and climate risk assessment. Their offerings are centered around advanced data modeling using the Weather Research Forecasting (WRF) model, which requires significant computational resources due to its ability to generate detailed meteorological datasets.

Project Background - AWS Custom product for Weather research forecasting

Peritos was hired to address these challenges by developing a comprehensive system that could:

Efficiently run the WRF model using HPC cluster.
Automatically create and manage HPC cluster jobs on receiving new data requests.
Automatically manage data resolution adjustments.
Provide a seamless experience for customers through an easy-to-use online platform.
Enable the commercialization of the datasets, ensuring that the customer could capitalize on the broad applicability of their data across multiple disciplines

Scope & Requirement

Implementation

Technology and Architecture

The architecture of this application efficiently handles the computational intensity of the WRF model, scales dynamically with demand, and provides a seamless experience for users. The integration of various AWS services ensures that the solution is robust, secure, and scalable.

Overall Workflow

User Request: Users input data parameters and request pricing. If satisfied, they proceed with the purchase.
Processing Trigger: Upon payment confirmation, the system triggers the data processing workflow.
WRF and WPS Processing: The ParallelCluster performs the necessary computations to generate the meteorological data.
Post-Processing: Any additional processing is done before the final data is stored.
Download and Notification: Users are notified and provided with a link to download their processed data.

Technology

Integrations

Cost Optimization

Peritos enhanced Tonkin + Taylor’s FinOps capabilities by designing a cost-efficient, scalable AWS architecture. We optimized compute resources using AWS ParallelCluster, implemented serverless automation with Lambda and Step Functions, and used Amazon S3 and FSx for Lustre for cost-effective data storage. The solution allowed Tonkin + Taylor to scale on demand, reduce infrastructure costs, and gain visibility into cloud spending. This enabled efficient monetization of meteorological data while maintaining control over operational expenses.

High-Performance Computing (HPC) Environment

AWS ParallelCluster: Provides the compute infrastructure needed to run the WRF model and WPS processes. This cluster is set up dynamically and scaled according to the computational demands of the task, ensuring efficient resource usage.
Head Node and Compute Fleet: The head node manages the compute fleet, which executes the high-compute WRF and WPS processes.
Head Node and Compute Fleet: The head node manages the compute fleet, which executes the high-compute WRF and WPS processes.

Processing and Orchestration

AWS Lambda Functions: Used extensively for orchestrating various steps in the data processing workflow.
AWS Step Functions: Orchestrates the entire workflow by coordinating Lambda functions, managing state transitions, and handling retries or errors.

Features of Application

The solution leverages AWS cloud services to generate, process, and distribute high-resolution meteorological data.
Users interact via an interface hosted on AWS Amplify, secured by AWS WAF and Shield, with APIs managed by Amazon API Gateway.
The system orchestrates data processing using AWS Lambda functions and AWS Step Functions, coordinating tasks such as WRF and WPS processing on an AWS ParallelCluster.
FSx for Lustre provides high-performance storage, while Amazon S3 and Aurora DB handle data storage and transaction management.
Post-processing is done on EC2 instances, with notifications sent via SNS. The solution efficiently manages the high computational demands of the WRF model, scales dynamically, and ensures secure, seamless data access for internal and external users.

Challenges

Challenge 1: High Computational Demand: The WRF model’s capacity to produce highly detailed meteorological datasets necessitates extensive computational power, which made running it on the customer’s existing local infrastructure impractical. The challenge was to find a solution that could efficiently handle large-scale data generation with optimum costing.
Solution: This challenge was met by implementing an AWS-based high-performance computing (HPC) cluster, specifically AWS ParallelCluster, which provided the necessary computational resources to run the WRF model efficiently. The jobs on ParallelCluster were created and managed dynamically using AWS Stepfunction and AWS Lambda by utilizing Slurm APIs
Challenge 2: User Experience and Commercialization: To monetize their meteorological data, the customer needed to create an accessible, user-friendly portal where external users could easily select regions, adjust data resolution, and purchase datasets. The portal needed to be intuitive, efficient, and fully capable of handling secure transactions, which was essential for the success of the customer’s business model.
Solution: The customer addressed this challenge by developing a web-based portal using AWS Amplify, integrated with AWS WAF and Shield for security, and managed via Amazon API Gateway. This platform provided a seamless user experience, enabling external customers to effortlessly interact with the system, select their data parameters, and complete purchases, thereby facilitating the commercialization of their datasets and enhancing revenue streams.

Project Completion

Duration

Deliverables

Jan 2024 – Aug 2024 ~ Implementation and Support
Custom web application was developed by the Peritos team working closely with the client’s product owner and completing any changes, bugs and adding critical features prior to Go live to ensure we have a smooth release.
We are still working on the handover documents and preparing for the final go Live

Testimonial

Awaited

Next Phase

We are now looking at the next phase of the project which involves:

Ongoing Support and adding new features every Quarter with minor bug fixes
Adding support for more countries

If You Are Looking For Similar Services? Please Get In Touch

Testimonial

Edward Yarashev

Product Manager | Tonkin + Taylor

Peritos is a team of highly skilled developers, technical experts, and delivery managers. We’ve been very impressed with their commitment. Their developers and delivery managers have conducted themselves with professionalism and diligence at all times, and the quality of the work they have performed has been excellent. Many times, they proposed better solutions, which resulted in better and faster products. Peritos is a reliable AWS Partner. You can trust and be satisfied.

(For services and support)

Richard Canny

Ultimation industries | President

“We are very happy with the results of this major IT project with Peritos Solutions. Their staff gained a good knowledge of our business and processes. They were able to work with many of our diverse team members and launch Business Central ERP on time and on budget according to our requirements. We will be using them again for future projects and additional development actions. Thank you!”

(For services and support)

Grant Peterson

Software Development Manager | Insight Software

The project was completed on time and as per the agreed budget. The communication was excellent, as was the dedication to quickly turning around the required report development. The overall experience was good, and I would definitely work with the Peritos Team again.

(For services and support)

Share us

Get A Quote

Get in touch and Know more!

Email Us

info@peritossolutions.com

Proud projects that
make us stand out

Recent Home

services

Azure Cloud Consulting Services

Automate mundane HR tasks from pre-boarding to post-retirement with our award-winning HRMS suite, driven by…

Recent Home

services

Cloud Consulting Services

Unlock agility, scalability, and innovation by modernizing your legacy systems through our tailored cloud solutions.…

Download our Checklist on Selecting Azure/ AWS vs Other Hosting providers

Recent Home

Downloads

Download our Checklist on Selecting Azure/ AWS vs Other Hosting providers

Cloud Hosting The checklist contains the below details on how you could decide between whether…

Recent Home

Open CV - Image processing

Border detection and Image processing

Home Category Open CV – Image processing ID card Border detection and image processing using…

Thinking about a project? Get in touch with us.

Connect with Us Today through the Details Below or Fill
Out the Form for a Prompt Response

Book Free Consultation

Guaranteed response within 8 business hours.

Trusted by Startups and Fortune 500 companies

07+ years of experience

We can handle projects of all complexities.

100+ satisfied customers

Startups to Fortune 500, we have worked with all.

20+ in-house team

Top 1% industry talent to ensure your digital success.

Software Consulting, Outsourcing & Training

Assessments

Cloud Consulting

Industry Types

Products & Offerings

AWS Custom Application Development Product Atmosolve Tonkin + Taylor