AWS Custom Application Development Product Atmosolve Tonkin + Taylor
AWS Custom Application Development Product Atmosolve Tonkin + Taylor
The project explores using AWS for custom app development using ESRI ArcGIS to read data and generate Hazard reports for a property that the user can search using Google Maps
Implementation
About Client
Project Background - AWS Custom product for Weather research forecasting
- Efficiently run the WRF model using HPC cluster.
- Automatically create and manage HPC cluster jobs on receiving new data requests.
- Automatically manage data resolution adjustments.
- Provide a seamless experience for customers through an easy-to-use online platform.
- Enable the commercialization of the datasets, ensuring that the customer could capitalize on the broad applicability of their data across multiple disciplines
Scope & Requirement
Implementation
Technology and Architecture
Overall Workflow
- User Request: Users input data parameters and request pricing. If satisfied, they proceed with the purchase.
- Processing Trigger: Upon payment confirmation, the system triggers the data processing workflow.
- WRF and WPS Processing: The ParallelCluster performs the necessary computations to generate the meteorological data.
- Post-Processing: Any additional processing is done before the final data is stored.
- Download and Notification: Users are notified and provided with a link to download their processed data.
Technology
- The web app was deployed with the below technological component
- Backend Code: .NET, C#, Python
- Web App code: Nextjs
- Database: PostgreSQL
- Cloud: AWS
Integrations
- Google APIs
- Stripe
- Auth0
- SendGrid
- Slurm APIs
Cost Optimization
Peritos enhanced Tonkin + Taylor’s FinOps capabilities by designing a cost-efficient, scalable AWS architecture. We optimized compute resources using AWS ParallelCluster, implemented serverless automation with Lambda and Step Functions, and used Amazon S3 and FSx for Lustre for cost-effective data storage. The solution allowed Tonkin + Taylor to scale on demand, reduce infrastructure costs, and gain visibility into cloud spending. This enabled efficient monetization of meteorological data while maintaining control over operational expenses.
High-Performance Computing (HPC) Environment
- AWS ParallelCluster: Provides the compute infrastructure needed to run the WRF model and WPS processes. This cluster is set up dynamically and scaled according to the computational demands of the task, ensuring efficient resource usage.
- Head Node and Compute Fleet: The head node manages the compute fleet, which executes the high-compute WRF and WPS processes.
- Head Node and Compute Fleet: The head node manages the compute fleet, which executes the high-compute WRF and WPS processes.
Processing and Orchestration
- AWS Lambda Functions: Used extensively for orchestrating various steps in the data processing workflow.
- AWS Step Functions: Orchestrates the entire workflow by coordinating Lambda functions, managing state transitions, and handling retries or errors.
Features of Application
- The solution leverages AWS cloud services to generate, process, and distribute high-resolution meteorological data.
- Users interact via an interface hosted on AWS Amplify, secured by AWS WAF and Shield, with APIs managed by Amazon API Gateway.
- The system orchestrates data processing using AWS Lambda functions and AWS Step Functions, coordinating tasks such as WRF and WPS processing on an AWS ParallelCluster.
- FSx for Lustre provides high-performance storage, while Amazon S3 and Aurora DB handle data storage and transaction management.
- Post-processing is done on EC2 instances, with notifications sent via SNS. The solution efficiently manages the high computational demands of the WRF model, scales dynamically, and ensures secure, seamless data access for internal and external users.
Challenges
- Challenge 1: High Computational Demand: The WRF model’s capacity to produce highly detailed meteorological datasets necessitates extensive computational power, which made running it on the customer’s existing local infrastructure impractical. The challenge was to find a solution that could efficiently handle large-scale data generation with optimum costing.
- Solution: This challenge was met by implementing an AWS-based high-performance computing (HPC) cluster, specifically AWS ParallelCluster, which provided the necessary computational resources to run the WRF model efficiently. The jobs on ParallelCluster were created and managed dynamically using AWS Stepfunction and AWS Lambda by utilizing Slurm APIs
- Challenge 2: User Experience and Commercialization: To monetize their meteorological data, the customer needed to create an accessible, user-friendly portal where external users could easily select regions, adjust data resolution, and purchase datasets. The portal needed to be intuitive, efficient, and fully capable of handling secure transactions, which was essential for the success of the customer’s business model.
- Solution: The customer addressed this challenge by developing a web-based portal using AWS Amplify, integrated with AWS WAF and Shield for security, and managed via Amazon API Gateway. This platform provided a seamless user experience, enabling external customers to effortlessly interact with the system, select their data parameters, and complete purchases, thereby facilitating the commercialization of their datasets and enhancing revenue streams.
Project Completion
Duration
Deliverables
- Jan 2024 – Aug 2024 ~ Implementation and Support
- Custom web application was developed by the Peritos team working closely with the client’s product owner and completing any changes, bugs and adding critical features prior to Go live to ensure we have a smooth release.
- We are still working on the handover documents and preparing for the final go Live
Testimonial
Next Phase
We are now looking at the next phase of the project which involves:
- Ongoing Support and adding new features every Quarter with minor bug fixes
- Adding support for more countries
If You Are Looking For Similar Services? Please Get In Touch
Testimonial
Peritos is a team of highly skilled developers, technical experts, and delivery managers. We’ve been very impressed with their commitment. Their developers and delivery managers have conducted themselves with professionalism and diligence at all times, and the quality of the work they have performed has been excellent. Many times, they proposed better solutions, which resulted in better and faster products. Peritos is a reliable AWS Partner. You can trust and be satisfied.
“We are very happy with the results of this major IT project with Peritos Solutions. Their staff gained a good knowledge of our business and processes. They were able to work with many of our diverse team members and launch Business Central ERP on time and on budget according to our requirements. We will be using them again for future projects and additional development actions. Thank you!”
The project was completed on time and as per the agreed budget. The communication was excellent, as was the dedication to quickly turning around the required report development. The overall experience was good, and I would definitely work with the Peritos Team again.
Share us
Get A Quote
Proud projects that
make us stand out
Recent Home
Recent Home
Recent Home
Recent Home
Thinking about a project? Get in touch with us.
Connect with Us Today through the Details Below or Fill
Out the Form for a Prompt Response
Book Free Consultation
Guaranteed response within 8 business hours.
Trusted by Startups and Fortune 500 companies
We can handle projects of all complexities.
Startups to Fortune 500, we have worked with all.
Top 1% industry talent to ensure your digital success.
AWS Environment Setup For Tonkin+Taylor
AWS Environment Setup For Tonkin+Taylor
Read the project details which talks about the set up of AWS environment for the client. We used Control Tower, Adhering to AWS NIST Compliance for multi account, Multi environment use
Implementation
About Client
AWS Environment Setup
Tonkin + Taylor is New Zealand’s leading environment and engineering consultancy with offices located globally. They shape interfaces between people and the environment which includes earth, water and air. They have won awards like Beaton Client Choice Award for Best Provider to Government and Community-2022 and IPWEA Award for Excellence in Water Projects for the Papakura Water Treatment Plan- 2021.
- https://www.tonkintaylor.co.nz/
- Location: New Zealand
Project Background
Scope & Requirement
- Setting up AWS environment for multi account, Multi environment setup
- Ensure the all AWS accounts follow same set of policies and adhering to all legal and regulatory compliance.
- Setup connectivity between different AWS accounts and on-prem network
- Setup AWS security hub to provide a comprehensive view of security state
- The Tonkin + Taylor On-Premise to Cloud Migration project aims to modernize and optimize the company’s IT infrastructure by migrating its existing on-premise systems to a cloud-based environment. The key objectives include reducing operational costs, enhancing scalability, improving system performance, and ensuring business continuity through a secure, reliable, and accessible cloud platform.
Implementation
Technology and Architecture
Technology/ Services used
- We used AWS services and helped them to setup below
- Cloud: AWS
- Organization setup: Control tower
- AWS SSO for authentication using existing AzureAD credentials
- Policies setup: Created AWS service control policies
- Templates created for using common AWS services
Security & Compliance:
- Tagging Policies
- AWS config for compliance checks
- NIST compliance
- Guardrails
- Security Hub
Network Architecture
- Site to Site VPN Architecture using Transit Gateway
- Distributed AWS Network Firewall
- Monitoring with Cloud Watch and VPC flow logs.
Backup and Recovery
Cost Optimization
Code Management, Deployment
Challenges of AWS Environment Setup
- It was a bit of a challenge to ensure the new environment meets all of the compliance criteria and still remain cost effective.
- As per best practices we need to have a set of Unique machines and each may need to have its own VPC but that may incur a cost to the client. So we discussed and agreed for a specific 75% to be achieved which would be deemed as acceptable
- We have some non compliance being generated by standard AWS services
- We got below feedback from AWS support . “It was reported that there are some policies and roles which are created by control tower and are created using best practices and there is no harm or security issue which arises it. Sometimes the conformance pack you use does shows the by default created policies to be non-compliant and you can ignore that for the time being. To make it compliant we have to either delete or the modify these roles/policies and since these roles and policies managed by the Control Tower, it is recommended not to manually update/delete the roles attached. Diving deeper into the issue, I could refer that the Control Tower managed artifacts feature heavily on the compliance failure list and there is no way to specifically exclude Control Tower managed artifacts from the compliance checking. Checking further with the internal team, I found that they are aware of the issue and there has been an ongoing feature request to exclude Control Tower managed artifacts from the compliance checking to avoid these such security findings. ” So based on this feedback we ensured to make a note that the non compliance being shown may not be an accurate one and these would be treated as exceptions.
Project Completion
Duration
- May 2024 to July 2024 ~ 3 months
Deliverables
- Implemented AWS Control tower with Org Unit structure and Service control policies
- Implemented AWS Config with NIST conformance pack
- Implemented AWS Security Hub
- Implemented Site to Site VPN using AWS Transit Gateway
- Handover and Support documents were provided
- Develop a detailed migration roadmap outlining timelines, key milestones, and risk mitigation strategies.
- Plan for data migration, including data validation, security protocols, and minimal downtime during the migration process.
- Establish contingency plans for legacy systems and ensure rollback strategies are in place specifically for multiple end user public facing apps they were planning to create.
Support
- 1 month extended support
- A template for Cloud formation stack to create more AWS resources using the available stacks
- Screen sharing sessions with demo of how the services and new workloads can be deployed.
- Offer support during the initial transition phase post-migration.
- Provide ongoing technical support, monitoring, and optimization services.
Testimonial
Santosh Dixit
Digitization delivery lead
Next Phase
We are now looking at the next phase of the project which involves:
- Launching new digital products with the help of AWS environments which have been setup
- Any adhoc change requests for managing the cloud environment
If You Are Looking For Similar Services? Please Get In Touch
Testimonial
Peritos is a team of highly skilled developers, technical experts, and delivery managers. We’ve been very impressed with their commitment. Their developers and delivery managers have conducted themselves with professionalism and diligence at all times, and the quality of the work they have performed has been excellent. Many times, they proposed better solutions, which resulted in better and faster products. Peritos is a reliable AWS Partner. You can trust and be satisfied.
“We are very happy with the results of this major IT project with Peritos Solutions. Their staff gained a good knowledge of our business and processes. They were able to work with many of our diverse team members and launch Business Central ERP on time and on budget according to our requirements. We will be using them again for future projects and additional development actions. Thank you!”
The project was completed on time and as per the agreed budget. The communication was excellent, as was the dedication to quickly turning around the required report development. The overall experience was good, and I would definitely work with the Peritos Team again.
Share us
Get A Quote
Proud projects that
make us stand out
Recent Home
Recent Home
Recent Home
Recent Home
Thinking about a project? Get in touch with us.
Connect with Us Today through the Details Below or Fill
Out the Form for a Prompt Response
Book Free Consultation
Guaranteed response within 8 business hours.
Trusted by Startups and Fortune 500 companies
We can handle projects of all complexities.
Startups to Fortune 500, we have worked with all.
Top 1% industry talent to ensure your digital success.
AWS Custom Application Development Product Landcheck For Tonkin + Taylor
AWS Custom Application Development Product Landcheck For Tonkin + Taylor
The project explores using AWS for custom app development using ESRI ArcGIS to read data and generate Hazard reports for a property that the user can search using Google Maps
Implementation
About Client
Project Background - AWS Custom Application Development using ESRI ArcGIS
Peritos and Landcheck got together to create a AWS Custom Application Development using ESRI ArcGIS integration to generate Hazard reports for specific properties. This was used for generating land based report which can be ordered specific to an address. client wanted to create an application which gives a comprehensive report to the user for their address indicating multiple hazards. It includes 10 hazards like Flooding, Winds, Liquefaction, Coastal Erosion, Active Fault etc. This report is created based on the latest data from authorised information provider, with expert Advice from Landcheck Engineers at a optimum cost which can help the end user get the information they need to make decisions regarding a specific property. This was all being done manually which the client now wanted to develop as a SAAS based offering.
Scope & Requirement
- A customized app which generates automatic reports of searched property address in Auckland Region
- Reports are generated from querying hazard data from ArcGIS server, where the information from Authorised council have been collated. Additional hazard risk calculation logic is applied on top of information returned from ArcGIS server to show the hazard risk in user friendly way. Based on the hazard risk level calculated for the property, Landcheck SMEs have also provided information to help understand the risk, which should also be added to report in a very user friendly way.
- Each hazard should have a property aerial image with hazard layers, showing how much area of the property is covered by different hazard levels.
- Reports should state the problem, hazard percentage and even the solution.
- User should be able to download the report in form of PDF files.
Implementation
Technology and Architecture
Technology
- The web app was deployed with the below technological component
- Backend Code: .NET Core, C#
- Web App code: ReactJS
- Database: PostgreSQL
- Cloud: AWS
Integrations
- Google APIs
- LINZ database
- ESRI ArcGIS
- Stripe
- Auth0
- SendGrid
Security:
- AWS WAF service is used for the firewall
- All API endpoints are token based
Responsive Design:
- All screens and UX was done keeping in mobile usage and are implemented with a responsive design in mind.
Scalability
Cost Optimization
Alerts and notifications are configured in the AWS to notify if the budget is being exceeded. Being deployed on serverless infrastructure, it desn’t imposes any additional cost if application is not being used a lot. Peritos being a cloud partner is managing the environment for the client keeping a close watch on the cost and finding ways to optimize the same
Backup and Recovery
- Automated backups are configured to backup the database and store multiple copies of the backup.
Code Management, Deployment
- CI/CD is implemented to automatically build and deploy any code changes.
Features of Application
- Search for an address, if the address is under supported regions then user will be able to select the address and application shows the outline of property in aerial view.
- User can get the report by creating an account on the application and making the payment
- Get the rating for the property for multiple hazards, like Winds, Flooding, Volcano, Earthquake etc. and expert advice from Landcheck Engineers on what are the remedial actions and next steps to take.
- This application, backend and front end are powered by AWS services.
Challenges
- We collated data from multiple council region and helped to get this stored on AWS layer. When a user buys the report, then the risk calculation logic goes through several datasets in ArcGIS server to calculate the risks for different hazards, then combine those results along with the expert advise from the Landcheck engineers and returns the result by generating a PDF. This was taking a huge amount of time when done at the go.
- Complex calculations are required for each hazard which involves data coming from different ArcGIS feature layers. In addition to this, an image for each hazard is also created combining multiple hazard layers from ArcGIS map server. All of these calculation were taking a lot of time in generating the report. In order to resolve this, we moved all the hazard calculation logic in a separate component, which gets triggered through an event. In this we optimized the code to perform each hazard calculation on separate thread. Also, we offloaded some of the GIS calculations to ArcGIS server, and access it with ArcGIS APIs. These changes reduced the time report creation time to just few minutes.
- Testing of the application with multiple addresses and users who were experts in their domain was a challenge.
- The data was quite complicated to understand and we relied on the Landcheck’s engineers to inform us what the expected result was. We did cover a lot of suburbs and did test close to 600 properties so we could be sure it is working as expected. However there were outliers and cases which did not work as expected and had to invest a fair bit of time to resolve those.
- ArcGIS integration was an issue as all the data from different Parcel and Linz layers had to be collated on the AWS ArcGIS server so we could get the information from a single source for multiple cities and suburb region
- This data was complicated to load and we had applied layers in terms of images and legends to display the data on the report side for an end user to easily interpret the results.
Project Completion
Duration
Deliverables
- Nov 2023 – Aug 2024 ~ Implementation and Support
- Sep 2024 – Present We are rolling out the changes to south Island in 2nd version
Deliverables
- Setting up the AWS services Architecture review and sign off by internal and existing vendors of Landcheck to ensure all best practices are followed and it is in alignment with best practices using AWS well Architected framework to ensure security , scalability and performance are upto the mark.
- Custom web application was developed by the Peritos team working closely with the client’s product owner and completing any changes, bugs and adding critical features prior to Go live to ensure we have a smooth release.
- We are still working on the handover documents and preparing for the final go Live
Support
As part of the project implementation we provide 2 months of Ongoing extended support. This also includes 20 hrs a month of development for minor bug fixes and a SLA to cover any system outages or high priority issues.
Testimonial
Edward Yarashev
Product manager Landcheck
Next Phase
We are now looking at the next phase of the project which involves:
- Ongoing Support and adding new features every Quarter with minor bug fixes
- Adding support for more NewZealand cities
If You Are Looking For Similar Services? Please Get In Touch
Testimonial
Peritos is a team of highly skilled developers, technical experts, and delivery managers. We’ve been very impressed with their commitment. Their developers and delivery managers have conducted themselves with professionalism and diligence at all times, and the quality of the work they have performed has been excellent. Many times, they proposed better solutions, which resulted in better and faster products. Peritos is a reliable AWS Partner. You can trust and be satisfied.
“We are very happy with the results of this major IT project with Peritos Solutions. Their staff gained a good knowledge of our business and processes. They were able to work with many of our diverse team members and launch Business Central ERP on time and on budget according to our requirements. We will be using them again for future projects and additional development actions. Thank you!”
The project was completed on time and as per the agreed budget. The communication was excellent, as was the dedication to quickly turning around the required report development. The overall experience was good, and I would definitely work with the Peritos Team again.
Share us
Get A Quote
Proud projects that
make us stand out
Recent Home
Recent Home
Recent Home
Recent Home
Thinking about a project? Get in touch with us.
Connect with Us Today through the Details Below or Fill
Out the Form for a Prompt Response
Book Free Consultation
Guaranteed response within 8 business hours.
Trusted by Startups and Fortune 500 companies
We can handle projects of all complexities.
Startups to Fortune 500, we have worked with all.
Top 1% industry talent to ensure your digital success.
