The Art of Infrastructure Elasticity using AWS

"The Art of Infrastructure Elasticity" was presented by Harish Ganesan on April 28th , Cloud Developer Conference 2012 , Bangalore.


Agenda :

Problem
Challenges
Requirements
Solution Architecture
Q&A


What is the problem scenario ?

• Big Sales Promotion  every quarter by the Enterprise
• Massive online Concurrent Visitors
• Limited processing capacity of the Booking Engine(~3k requests/sec)
• Unhappy Visitors 
• More Booking  opportunity lost

The art of infrastructure elasticity

View more presentations from Harish Ganesan

Solution (Step 1):

• Create a Queuing App before the Booking engine
• Efficiently Queue the concurrent visitors

Solution (Step 2) :
Moderate and move the visitors waiting in Queuing app to Booking engine 


What are the Challenges ?


Challenge 1: Concurrency
HTTP/AJAX/REST requests
Total : 500+ Million requests in 6 hours 
Average :23k+ requests/sec
Peak : 80K+ requests/sec 


Challenge 2:Queue efficiency

• Allot unique Queue Numbers for visitors
• Queue Number allotment on Fair Basis (As much possible)
• Reduce the wait time in Queue Number allotment process
• Reduce overall Queue wait time for the visitor

Challenge 3:Load Volatility
Massive utilization during promo and under utilization during other times 


Challenge 4:IP Whitelisting

• Booking engine and other 3rd party webservices needs EC2 IP Whitelisting for security
• Consecutive IP range needed for whitelisting

Challenge 5:Variety of OS / Software’s in the tech stack

• RedHat OS for Load Balancer , NoSQL and Queue Layer
• Apache Tomcat Java web/App Layer
• CentOS for  Processing Programs 
• MySQL for Result storage
• Hadoop for Analytics

What are the requirements from enterprise ?
Elastic Infrastructure 

• Create the Infrastructure 2 hrs before the promo
• Tear down infrastructure 2 hrs after the promo
• Elastically expand the infra during the promo

Highly Scalable and Available
Log Analytics
Complete Infrastructure Automation


Solution Architecture
Option 1: Single Queue ( Initial thought)
Option 2: Parallel Queue ( Recommended )


Request types

• Customer Visit is a HTTP request to the Queuing Application
• Current Visitor Queue position is a AJAX call every X seconds  to the Queuing Application. More Wait ~ More Calls

Solution Step 1 : The Cloud ?

• Amazon Web Services
• We had 4+ years Architecture experience in AWS
• It satisfied many customer requirements and challenges in this use case

Solution Step 2 : R53/NW

• Amazon VPC with Multi-AZ subnet configurations ( HA )
• Amazon Route 53 for Managed DNS
• DNS RR algorithm at Route53

Solution Step 3: Load Balancing

• HAProxy vs Amazon ELB
• Custom programs to Auto Scale HAProxy
• HAProxy Elastic -> Attach / Detach from Route53
• HAProxy IP whitelisting in 3rd party Gateway
• 16 HAProxy Instances , 2 AZ’s , 2 Subnets
• RR Load Balancing algorithm

Solution Step 4: Web/App Servers

•  Web/App instances under every HAProxy
• C1.Xlarge Instance Type for Web/App Instances
• Custom programs to Auto Scale C1.Xlarge 
• Automatic Attach / Detach from HAProxy
• Every web/App Instance with EIP for IP whitelisting
• 48 Web/App EC2 Instances spread across 2 AZ’s

Solution Step 5: Queue Servers

• RabbitMQ vs Amazon SQS
• FIFO/Concurrency/No Duplicate messages
• 1 RabbitMQ instance for queuing every sector
• M1. large Instance Type
• 16 RabbitMQ Instances overall

Solution Step 6: Redis

• Redis vs Amazon DynamoDB
• Redis : NoSQL KV Data store
• Visitors are shown their Current Queue position every X seconds from Redis
• 1 Redis Master-Slave instance for every sector
• M1. large Instance Type for Redis
• 32 Redis Instances overall

Solution Step 6: Processors

• BG Processors : Java Programs to 
• RabbitMq -> Redis : Allot Queue numbers to visitor requests  and insert to Redis
• Redis -> Booking Engine : Moderate the movement of queued visitors from Redis to Booking Engine
• Process the Response Status / Booking Status / Inactive Visitors /  Timeouts 
• 2 BG Processor node per sector
• CPU intensive : C1.Xlarge Instance Type
• 32 BG Processor Instances overall

Scalability

• New sectors containing LB, Web, Queue , NoSQL , BG stack will be created automatically depending upon the load
• Same AZ or multi-AZ can be specified for the creation
• CloudWatch Custom parameters used
• Automated Java Programs were used for the sector creation
• No Manual intervention needed

High Availability @ Instance Level

• HA built @ Web/App , Redis and BG processor instances
• Any Failure / Non responsive EC2 instances will be automatically detected/replaced by Java programs
• No Manual intervention needed

High Availability @ Sector level

• Any Failure / Non responsive instances inside Sectors will be automatically detected/replaced by Java programs
• If sector-3 fails , still other sectors will be active and can take requests

High Availability @ AZ level

• If entire AZ-2 fails then load will be balanced to instances in AZ-1
• Automated programs will create new sectors inside AZ-1 to handle the load

Log Analytics

• Redis , Web/App , HAProxy , RBQ logs synced to S3
• Elastic MapReduce Jobs to process / analyze the logs
• Processed result moved to RDS MySQL for reports/ Visualizations

Monitoring

• Nagios + Puppet (combined) for Auto scaled monitoring infra and deployment
• CloudWatch  Custom metrics / Tomcat Valve/ Automated Java Programs for EC2 

Backup

• No backups -> only Syncs to S3
• Golden AMI’s snapshot to S3
• Periodic Sync of data between EC2 and S3
• Periodic log Sync between Web/App to S3

Overall Infrastructure 

• Amazon Route53
• Amazon VPC – Public , Private subnet
• 150+ EC2 instances ,  2 AZ’s , 1 Region 
• 70+ Elastic IP’s 
• 200+ EBS 
• S3 buckets
• Suite of monitoring tools
• 1 Puppet Server
• Amazon CloudWatch 
• Amazon CloudFront

Infrastructure Elasticity

• Entire Infra created 2 hrs before promo
• Tear down infra 2 hrs after promo
• ~30 Mins to launch the infra in AWS
• ~45 Mins to tear down 
• Automated Failure detection/rectification
• Automated Programs for Infra creation

Infrastructure Cost 

• ~10K USD per promo
• Not inclusive of Data charges
• Unthinkable Savings 
• Visitor experience was good
• More Bookings per Promo
• Power of Elasticity is Simply priceless , AWS is “AWSome”

The Art of Infrastructure Elasticity

"The Art of Infrastructure Elasticity" was presented by Harish Ganesan on April 28th , Cloud Developer Conference 2012 , Bangalore.


Agenda :

Problem
Challenges
Requirements
Solution Architecture
Q&A


What is the problem scenario ?

• Big Sales Promotion  every quarter by the Enterprise
• Massive online Concurrent Visitors
• Limited processing capacity of the Booking Engine(~3k requests/sec)
• Unhappy Visitors 
• More Booking  opportunity lost

The art of infrastructure elasticity

View more presentations from Harish Ganesan

Solution (Step 1):

• Create a Queuing App before the Booking engine
• Efficiently Queue the concurrent visitors

Solution (Step 2) :
Moderate and move the visitors waiting in Queuing app to Booking engine 


What are the Challenges ?


Challenge 1: Concurrency
HTTP/AJAX/REST requests
Total : 500+ Million requests in 6 hours 
Average :23k+ requests/sec
Peak : 80K+ requests/sec 


Challenge 2:Queue efficiency

• Allot unique Queue Numbers for visitors
• Queue Number allotment on Fair Basis (As much possible)
• Reduce the wait time in Queue Number allotment process
• Reduce overall Queue wait time for the visitor

Challenge 3:Load Volatility
Massive utilization during promo and under utilization during other times 


Challenge 4:IP Whitelisting

• Booking engine and other 3rd party webservices needs EC2 IP Whitelisting for security
• Consecutive IP range needed for whitelisting

Challenge 5:Variety of OS / Software’s in the tech stack

• RedHat OS for Load Balancer , NoSQL and Queue Layer
• Apache Tomcat Java web/App Layer
• CentOS for  Processing Programs 
• MySQL for Result storage
• Hadoop for Analytics

What are the requirements from enterprise ?
Elastic Infrastructure 

• Create the Infrastructure 2 hrs before the promo
• Tear down infrastructure 2 hrs after the promo
• Elastically expand the infra during the promo

Highly Scalable and Available
Log Analytics
Complete Infrastructure Automation


Solution Architecture
Option 1: Single Queue ( Initial thought)
Option 2: Parallel Queue ( Recommended )


Request types

• Customer Visit is a HTTP request to the Queuing Application
• Current Visitor Queue position is a AJAX call every X seconds  to the Queuing Application. More Wait ~ More Calls

Solution Step 1 : The Cloud ?

• Amazon Web Services
• We had 4+ years Architecture experience in AWS
• It satisfied many customer requirements and challenges in this use case

Solution Step 2 : R53/NW

• Amazon VPC with Multi-AZ subnet configurations ( HA )
• Amazon Route 53 for Managed DNS
• DNS RR algorithm at Route53

Solution Step 3: Load Balancing

• HAProxy vs Amazon ELB
• Custom programs to Auto Scale HAProxy
• HAProxy Elastic -> Attach / Detach from Route53
• HAProxy IP whitelisting in 3rd party Gateway
• 16 HAProxy Instances , 2 AZ’s , 2 Subnets
• RR Load Balancing algorithm

Solution Step 4: Web/App Servers

•  Web/App instances under every HAProxy
• C1.Xlarge Instance Type for Web/App Instances
• Custom programs to Auto Scale C1.Xlarge 
• Automatic Attach / Detach from HAProxy
• Every web/App Instance with EIP for IP whitelisting
• 48 Web/App EC2 Instances spread across 2 AZ’s

Solution Step 5: Queue Servers

• RabbitMQ vs Amazon SQS
• FIFO/Concurrency/No Duplicate messages
• 1 RabbitMQ instance for queuing every sector
• M1. large Instance Type
• 16 RabbitMQ Instances overall

Solution Step 6: Redis

• Redis vs Amazon DynamoDB
• Redis : NoSQL KV Data store
• Visitors are shown their Current Queue position every X seconds from Redis
• 1 Redis Master-Slave instance for every sector
• M1. large Instance Type for Redis
• 32 Redis Instances overall

Solution Step 6: Processors

• BG Processors : Java Programs to 
• RabbitMq -> Redis : Allot Queue numbers to visitor requests  and insert to Redis
• Redis -> Booking Engine : Moderate the movement of queued visitors from Redis to Booking Engine
• Process the Response Status / Booking Status / Inactive Visitors /  Timeouts 
• 2 BG Processor node per sector
• CPU intensive : C1.Xlarge Instance Type
• 32 BG Processor Instances overall

Scalability

• New sectors containing LB, Web, Queue , NoSQL , BG stack will be created automatically depending upon the load
• Same AZ or multi-AZ can be specified for the creation
• CloudWatch Custom parameters used
• Automated Java Programs were used for the sector creation
• No Manual intervention needed

High Availability @ Instance Level

• HA built @ Web/App , Redis and BG processor instances
• Any Failure / Non responsive EC2 instances will be automatically detected/replaced by Java programs
• No Manual intervention needed

High Availability @ Sector level

• Any Failure / Non responsive instances inside Sectors will be automatically detected/replaced by Java programs
• If sector-3 fails , still other sectors will be active and can take requests

High Availability @ AZ level

• If entire AZ-2 fails then load will be balanced to instances in AZ-1
• Automated programs will create new sectors inside AZ-1 to handle the load

Log Analytics

• Redis , Web/App , HAProxy , RBQ logs synced to S3
• Elastic MapReduce Jobs to process / analyze the logs
• Processed result moved to RDS MySQL for reports/ Visualizations

Monitoring

• Nagios + Puppet (combined) for Auto scaled monitoring infra and deployment
• CloudWatch  Custom metrics / Tomcat Valve/ Automated Java Programs for EC2 

Backup

• No backups -> only Syncs to S3
• Golden AMI’s snapshot to S3
• Periodic Sync of data between EC2 and S3
• Periodic log Sync between Web/App to S3

Overall Infrastructure 

• Amazon Route53
• Amazon VPC – Public , Private subnet
• 150+ EC2 instances ,  2 AZ’s , 1 Region 
• 70+ Elastic IP’s 
• 200+ EBS 
• S3 buckets
• Suite of monitoring tools
• 1 Puppet Server
• Amazon CloudWatch 
• Amazon CloudFront

Infrastructure Elasticity

• Entire Infra created 2 hrs before promo
• Tear down infra 2 hrs after promo
• ~30 Mins to launch the infra in AWS
• ~45 Mins to tear down 
• Automated Failure detection/rectification
• Automated Programs for Infra creation

Infrastructure Cost 

• ~10K USD per promo
• Not inclusive of Data charges
• Unthinkable Savings 
• Visitor experience was good
• More Bookings per Promo
• Power of Elasticity is Simply priceless , AWS is “AWSome”

Introduction to Mobile Cloud Architectures using AWS

Mobile Cloud Computing Architectures

View more presentations from Harish Ganesan

Webinar : Mobile Cloud Computing Architectures


Agenda
Introduction to Amazon Web Services
Mobile landscape Overview
Importance of the mobile backend
3 Cases – Mobile Cloud Architectures
Cloud is your Storage
Cloud is your Processor
Cloud is your Extension
Q&A



What is Amazon Web Services?


Amazon Web Services is a cloud computing platform that provides flexible, scalable, and cost-effective technology infrastructure for businesses of all sizes around the world.…utilizing the knowledge, expertise, and tools used to run Amazon.com’s global web properties for well over a decade. 
Amazon Web Services is steadily expanding its global infrastructure to help customers achieve lower latency and higher throughput, and to ensure that their data resides only in the Region they specify. As customers grow their businesses, AWS will continue to provide infrastructure that meets their global requirements.
Amazon Web Services provides highly scalable computing infrastructure that enables organizations around the world to requisition compute power, storage, and other on-demand services in the cloud.  These services are available on demand so a customer doesn’t need to think about controlling them, maintaining them or even where they are located. 
AWS approach has always been to be a customer focused company.  They constantly look to develop services in line with the needs of our customers to make sure they get the flexibility and usability out of the service that they need to be successful.  


AWS Meets Enterprise Security Standards


Examining AWS, you’ll see that the same security isolations are employed as would be found in a traditional data center. These include physical data centre security, separation of the network, isolation of the server hardware, and isolation of storage. AWS customers have control over their data: they own the data, not us; they can encrypt their data at rest and in motion, just as they would in their own data center. 


Why AWS for Mobile cloud apps ?


Flexibility

• Wide range of services
• Pay as you go
• Align your capacity with your growth

Scale with AWS

• World-wide footprint
• Elastically increase your AWS  , DB , NoSQL , EC2

Wide Variety of Access Methods

• Protocols : JSON , BSON , SOAP , REST , HTTP/S , TCP, RTMP
• AWS Mobile SDKs : iOS and Android
• Server Side Languages : Java , Python , PHP ,Ruby 

Planet of the Apps – “Appnomy


Some interesting stats about Mobile Appnomy

• Tablets generate 21% of mobile traffic to retailer sites
• iTunes App Store 571,745 approved apps , Android Market 300,000 apps
• Smartphones used for food shopping by 43 percent of US owners
• By 2014, mobile internet should take over desktop internet usage
• iPhone owner has 50+ apps Android owners have 35+ apps
• 26% of the time Smart Phone users  will never use the same app again
• One in four mobile apps once downloaded is never used again
• Over 300,00 mobile apps have been developed in three years

We are observing in the last few years that Mobile Cloud computing is emerging as a new paradigm. Mobile applications are becoming highly intelligent and they are leveraging the user’s location, context , Data and network intelligence, for considerably improving user experience. Such complex Mobile applications require Solid Backend Architecture with Elastic processing and memory power. Let us see how we can harness AWS for developing and implementing such mobile applications.


I will be addressing this through 3 Architectural cases in the coming articles

AWS Cloud Tour 2011 Technical Track Presentation on Amazon AutoScaling

Embedded the Technical Track presentation given by me at AWS Cloud Tour 2011, India , New Delhi on Nov 24th , 2011




Presentation slides can be found here


Slide contents



Slide 2:
Load Patterns of Business 1 E-commerce 2 Online Ticketing Compute Compute Weekly Weekly 3 Online Travel 4 During Campaigns Compute Compute Yearly Yearly



Slide 3:

Load Burst Types 1 Un Predictable Bursts 2 Predictable BurstsCompute Compute Time Time



Slide 4:

Traditional Scaling Lost customers • Fixed capacity • Under utilization of resources • Lost CustomersAmazon Auto Scaling • Align no. of instances with demand • Flexible capacities and scaling sizes • Save costs



Slide 5:

Amazon AutoScaling• Handle all the variety of load bursts and patterns• Scale out Amazon EC2 instances seamlessly and automatically when demand increases• Scale down unwanted Amazon EC2 instances automatically and save money when demand subsides• Decide the scaling based on AWS CloudWatch metrics



Slide 6:

Amazon AutoScaling Architecture EC2 Instance Elastic Load Balancer … Amazon CloudWatch Scale UpKey Concepts : Scale Up Rule EC2 Instance• Launch Configuration Scale Down Scale Down• Auto Scaling Group Rule• Trigger Scale Up • Policy Predefined Scaling • Alarm Auto Scaling Scale Down Activity Group



Slide 7:

Auto Scaling : Unpredictable Burst 2 Un Predictable Bursts • Configure On demand Auto Scaling policy Compute • Create policies for both Scaling out and Scaling down scenarios Time • Auto Scaling will respond to changing conditions dynamically%as-create-auto-scaling-group as-group --availability-zones us-east-1c –launch-configuration my-lconfig --max-size 24 --min-size 4 --cooldown180 --load-balancers my-load-balancer



Slide 8:

Auto Scaling : Predictable Burst 1 Predictable Bursts Compute • Configure Time based Auto Scaling plan • PutScheduledUpdateGroup Time Action - specify the date and time of execution, minimum, maximum, and desired Instance size of capacity% as-put-scheduled-update-group-action my-group –name “My-scale-up” -- time “2011-11-12T02:00:00Z” -min 10 -max 50



Slide 9:

Candidates for Auto Scaling• Web Tier• Application Tier• Caching Tier• Load Balancing Tier• Other Stateless Tiers



Slide 10:

Auto Scaling Benefits : Cost Savings 1 35 % savings 2 30 % savings Compute Compute Time Time 30 % savings 3 35 % savings Compute Time



Slide 11:

Auto Scaling Benefits : No Lost customers We can closely align our Infrastructure with our load requirements and save costs Benefits of Auto Scaling – work in Happier DevOpsprogress !!!Source: AWS



Slide 12:

Key Points to Remember• Log Files , Data Files and Sessions• Multiple AZ’s inside a Region• EBS Backed AMI’s are faster to launch• Multiple Instances Types in an Auto Scaling group• Combining Custom CloudWatch Metrics opens huge possibilities• Evaluate Traditional Monitoring

Scale new business peaks with Amazon AutoScaling

Attached the Technical Track presentation given by me at AWS Cloud Tour 2011, India , New Delhi on Nov 24th , 2011

Topic: Scale New Business peaks with Amazon Auto Scaling

Scale new business peaks with Amazon auto scaling

View more presentations from Harish Ganesan

Presentation video can be found here

Some of the Key points presented were:

• ·         Server farm utilization and Load traffic patterns of various Online business
• ·         Load Burst Types: Unpredictable and Predictable
• ·         Traditional Scaling vs Amazon AutoScaling
• ·         Introduction to Amazon Auto Scaling
• ·         Amazon Auto Scaling Architecture and Key concepts
• ·         How to address Unpredictable bursts with Amazon AutoScaling
• ·         How to address predictable bursts with Amazon AutoScaling
• ·         Architecture Tiers - Candidates for Auto Scaling
• ·         Auto Scaling Benefits : Cost Savings
• ·         Auto Scaling Benefits : No Lost customers
• ·         Key Best practices are

o   Log Files , Data Files and Sessions

o   Multiple AZ’s inside a Region

o   EBS Backed AMI’s are faster to launch

o   Multiple Instances Types in an Auto Scaling group

o   Deployment using Puppet

o   Combining Custom Amazon CloudWatch Metrics opens huge possibilities

o   Evaluate Traditional Monitoring