Load Balancing in Amazon Web Services

Load balancing is one of the most important technique to be followed for architecting highly scalable and available applications on AWS cloud. Keeping this in mind, I have compiled my experience and articles i have written on this subject as "Load balancing on AWS" series.  

Dissecting Amazon Elastic Load Balancing : Amazon ELB is dissected into 18+ points and analysed based on the production implementation experience. This article is a must read for all Amazon ELB users who wants to understand in detail what ELB can do and what are some of the road blocks you can face sometimes when using Amazon ELB. Click here for the article.  

Amazon ELB Multi region migration checklist: Amazon Elastic Load Balancing has a Amazon EC2 Regional scope. It needs to be migrated to alternate Amazon EC2 region in event of DR or during new migration setup . I am sharing my experience in this post as few checklists/areas that needs to be taken care during this Amazon ELB migration to alternate EC2 region. Refer Article

Amazon ELB Implementation Architectures : Amazon ELB can be implemented in variety of architectures in your AWS cloud. Some of them are real bad cases and some of them follow best practices. If you want to understand the common implementation architectures and its pro's/con's refer this article  

Comparison Analysis between Amazon ELB and HAProxy:  HAProxy is a popular OSS software load balancer widely used in Amazon Cloud Infrastructure. This article provides a detailed comparison between the two and analyzes where both of them stand. This article is featured in the HAProxy web site as well. Click here for the article

Architecting High Availability @ HAProxy Load Balancing Tier: Not all the times Amazon ELB is used as the choice of load balancer in Amazon Web Services. But deploying an OSS load balancer in AWS has got is own set of considerations and high availability is one of them. In this post i explore how to design High Availability @ HAProxy load balancing layer. Refer Article

Architecting Highly Available Web App Layer using HAProxy Load Balancing : In this post we explore four options of Load balancing Web/App Tier using HAProxy in AWS. Refer Article

Configuring Citrix NetScaler Load Balancing on AWS: Citrix NetScaler is a popular Load balancer in the Enterprise world. Hardware and virtualized NetScaler has been serving many popular enterprise online assets for years. How to configure Citrix Netscaler based Load Balancing in AWS is explained in this post. Refer Article

Load Balancing Amazon RDS Read Replica's using HAProxy: In this post, let us explore how HAProxy can be used to solve load balancing between RDS Read Replica's in Amazon Cloud.Refer Article

Choosing right HAProxy- Amazon EC2 Instance Types and AMI Types for load balancing: Before choosing optimal Amazon EC2 instance Type for HAProxy Load balancing layer in AWS we need to minimum understand some important factors involved. Refer Article 

Configuring Amazon ELB and understanding the parameters in detail: Amazon provides a detailed documentation on how to configure ELB. In this article i have detailed the configuration steps with detailed understanding on the parameters and its implications. Must read of Amazon ELB newbies. Click here for the article 

Web Session Synchronization patterns in AWS: Architectures for synchronizing sessions of load balanced Web/App EC2 is detailed in this post. Refer Article

Geo Distributed Load Balancing using Route53 and Amazon ELB: Some customers would have Geo Distributed their architecture across multiple Amazon EC2 regions. In this article we explore why do we need Geo Distributed architecture, Cost of Latency and how to achieve it using Route 53 + Amazon ELB. Click here for the article

Deeper Health Checks and Problems in Load Balancing in AWS: Health Checks are one of the essential mechanisms that helps you to keep N-Tiered system highly available.This sounds simple and straight forward right, but some of the customers i have consulted follow a much deeper Health check diagnostic mechanism and it might have problems when migrated to AWS cloud. Let us explore this case in detail in this post. Refer Article

Configuring Amazon ELB With SSL offloading: How to configure SSL with Amazon ELB. Click here for the article What are the benefits of offloading SSL in Amazon ELB. Here

Monitoring Amazon ELB using Amazon CloudWatch and understanding the result and metrics

Why do we need Amazon Elastic Load Balancing and What are its benefits ?

Deeper Health Checks and Problems in Load Balancing in AWS

Health Checks are one of the essential mechanisms that helps you to keep N-Tiered system highly available. Usually a simple script or program is deployed on the Web/App Server. The Health check component of Load Balancer is configured to frequently call this script in Web/App Server in a light weight protocol. Based on the response from the script/program the Load balancer decides the status of the Web/App Servers and accordingly direct the requests to healthy Web/App servers. This is a usual mechanism that is followed in all popular load balancers like Amazon ELB, Netscaler, HAProxy and NGinx in AWS cloud. This sounds simple and straight forward right, but some of the customers i have consulted follow a much deeper Health check diagnostic mechanism and it might have problems when migrated to AWS cloud. 
Let us explore this case in detail :

What is the architecture ?
A simple multi-tiered architecture with : A  load balancer deployed at the front. The Web/App Server has the health script/program. The database is MySQL deployed Master+ Slave mode.  

What is deeper Health Check ?
The script/program deployed in the Web/App Server is little intelligent; when it is called by load balancer it performs simple operations and checks the status of the Database. So when you get a response back from the health check script / program you are verifying whether the health of DB and Web/App server is sound at the load balancer tier.

What is the problem scenario ?
Imagine when migrating this infrastructure to AWS you have adopted the standard architecture pattern consisting of :

• Amazon ELB is used as the Load balancer
• Web/App Server in auto scaling mode
• MySQL moved to Amazon RDS+Multi-AZ with RR

Now let us explore this problem in detail : 

• Imagine the any of the following condition in your production, network between database and Web/App is down intermittently for few minutes or RDS MySQL is elevating the Hot Standby as new Master. In such scenarios, the health check response actually timeouts at Database level, whereas the Load Balancer will mark the even the healthy App Servers as unhealthy because of the deeper health checks. This is not good especially for Amazon Auto Scaled scenario's where Amazon ELB marks Web/App EC2 as unhealthy because of deeper health check and Amazon Auto Scaling keeps restarting the Web/App EC2 auto automatically to maintain minimum healthy farm. This unwanted effect can cascade the overall availability and surely not good for the production in AWS. So in short Deeper Health checks are not surely recommended for complex N-Tier systems that follows Auto scaling/healing and Service oriented architecture patterns in AWS. 
• Usually the purpose of health check is to check the status of next tier or service consumed by a particular tier.  Deeper health checks is heavy weight and it usually takes much more time to respond because majority of your tiers are exercised in this process. If we set this frequency too aggressive, then health checks itself will eat lots of your CPU. So the frequency of the health checks and the response time out have to be set considerably large. Also during heavy traffic scenario, such heavy weight calls can be queued and you might not get faster response in deeper health checks. 
• Deeper health checks are usually suitable for simple and fixed infrastructures. When your infrastructure is non elastic , the decisions are taken manually by the ops team after analyzing the particular failing tier. For Elastic Auto scaled workloads in AWS it is better to isolate the health checks of load balancing tier separate from Deeper Health checks that can be used for assessing the availability of the infrastructure.

Amazon ELB Multi region migration checklist

Amazon Elastic Load Balancing has a Amazon EC2 Regional scope. It needs to be migrated to alternate Amazon EC2 region in event of DR or during new migration setup . I am sharing my experience in this post as few checklists/areas that needs to be taken care during this Amazon ELB migration to alternate EC2 region:

1. Amazon ELB automatically distributes incoming application traffic across multiple Amazon EC2 instances in Multiple Availability Zones. Amazon EC2 regions have different levels of AZ's, US-East EC2 region has 5 AZ's and Singapore has 2 AZ's. When you are migrating the ELB make sure your configuration matches the levels of target Amazon EC2 region Availability zones .
2. When Amazon ELB is created on target Amazon EC2 region, the ELB endpoint will be different from the source one. This has to be re configured on your Route 53 and other thick clients during migration.
3. Since the ELB end point is different, if you had raised any pre-warming request to AWS support , please raise it again for the target region with new ELB end point. This is valid in use cases where you are moving the traffic from primary region to target region in event of outage. 
4. SSL certificates associated on the old Elastic Load Balancing service (source regions) need to be re configured to the new Elastic Load Balancing service in the Target region.
5. Security Groups are region specific and has to be migrated/recreated in the target region. Make sure the security groups of ELB in target region matches the source region
6. Amazon ELB is not an isolated service and usually it will be used in production with Amazon EC2, Auto Scaling group and Cloud Watch Alarms. In addition there will be health checks, time outs, thresholds etc configured specifically for an application in Amazon ELB. You need to ensure that all these items associated with Amazon ELB are transferred target region for proper functioning of the application. Using Amazon CloudFormation templates for creating infrastructure properties will surely help and ease the efforts involved during such ELB migration.
7. To load balance your EC2 instances launched in default VPC, you have to create your load balancers within your default VPC. When you create a load balancer within default VPC, Elastic Load Balancing automatically creates a security group by defining the ports specified for the load balancer to be opened. Default VPC is currently not available in US-East. In case your Source region is US-West and you are planning your DR or migration with Target region as US-East, you need to be aware of this situation. 
8. In case Amazon ELB and Internal ELB are used inside Amazon VPC, you should ensure that the IP address ranges being used in source and target regions matches and aligns properly, so that routing and load balancing continues to work after migration. 

Other Multi Region Migration articles:

Security Groups - Multi Region Migration checklist

Elastic IP - Multi region Migration checklist, tips , best practices

EC2 AMI Copy - Multi Region Migration

MySQL Amazon EC2 Cross region Master-Slave Replication using SSL

Amazon EBS Snapshot copying- Performance, Tips and How to

Amazon EC2 AMI Copy Feature - Performance, Tips and How To

Configuring Citrix NetScaler Load Balancing on AWS

Citrix NetScaler is a popular Load balancer in the Enterprise world. Hardware and virtualized NetScaler has been serving many popular enterprise online assets for years. NetScaler supports many load balancing algorithms like (Round robin, Least connection, source IP Hash etc..) + comes with many more features. Citrix Netscaler with its powerful load balancing capabilities serves many websites under it inside an enterprise setup. Citrix NetScaler is usually preferred in AWS by :

• Enterprises which already have some partnership with Citrix and have some form of NetScaler licenses / agreements with Citrix
• Enterprises which want proper support for their load balancing tier which other OSS LB providers were not able to meet
• Enterprises which already have trained IT staff in Citrix products including NetScaler and do not want change
• Enterprises which wants Load balance tier to provide them with Security, Performance in addition to wide variety of LB options.

In this article i am going to share the experience on configuring Citrix NetScaler in AWS cloud. Citrix NetScaler will be used inside public subnet of Amazon Virtual Private Cloud (VPC) and will load balance requests between 2 web/App EC2 instances running in private subnet of the Amazon VPC. The Web/App amazon EC2 are deployed in multiple subnets in Multi-AZ architecture inside the Amazon VPC. 

Following is the step by step configuration details for the same:

Step 1: Login to AWS market place using the console using IAM user account and Click Amazon EC2 tab and Click the Launch Instance option.

Step 2: Find the Citrix NetScaler Instance and choose the right instance based on your application load and features needed. For example:  if your application load will not exceed 10 Mbps and you need round robin load balancing, you can choose NetScaler Standard Edition 10 Mbps as your instance type. If your application has heavy load and you are planning to leverage other features of Netscaler like performance, security etc, you can choose NetScaler VPX Platinum Edition - 1000 Mbps option as well. To know more about Citrix NetScaler on Amazon refer link https://aws.amazon.com/marketplace/seller-profile?id=fb9c6078-b60f-47f6-8622-49d5e1d5aca7

Step 3: Click “Launch with EC2 Console” option and launch the Citrix NetScaler in suitable+available Amazon EC2 region of your choice.

Step 4: Select Amazon VPC IP range example: 10.100.0.0/16 in our case, choose the public subnet for Citrix NetScaler EC2 instance and launch the EC2 Instance as shown in the below screen shots.

Step 5: Once the Instance status is “2/2 check passed”, attach one Amazon Elastic IP to Citrix NetScaler default ENI- Elastic Network Interface.

Step 6: Use the Amazon Elastic IP in your browser.Use the default user name and password “nsroot” and login into Citrix NetScaler console.

Step 7:Click configuration tab -> expand the Network and add the IP address  (The virtual IP address you have to configure for the load balancing)

Step 8: Expand the system option on the left panel and click the settings and click the “configure basic features” settings on the right panel. Citrix NetScaler offers many features like :

• Integrated Caching, HTTP compression SSL offloading for performance
• Content Filtering, Access Gateway, App Firewall, Authentication etc for Security

In this post we are going to explore only Load balancing feature, so select the “Load Balancing” and click OK.

Step 9: Select and expand the load balancing feature on the left panel and click the services.

Step 10: Click the “Add” button and give the web/app EC2 instance details to be load balanced. We have two Web/App EC2 with 10.100.5.15 and 10.100.4.15 in two subnets in Multi-AZ architecture. Both these web/app ec2 are attached to the NetScaler for Load Balancing. Illustrated in below screenshots.

Click “Create” , then check the service status is “UP”

Step 11: Click the “Virtual Servers” on the left panel.Type the Virtual Server name and the IP address, protocol and port number. Then select the service for the Web/App EC2 Servers.

Step 12: Select the “method assistance” tab and select the load balancing algorithms.   
The load balancing algorithm defines the criteria that the NetScaler appliance uses to select the service to which to redirect each client request. Different load balancing algorithms use different criteria. For example, the least connection algorithm selects the service with the fewest active connections, while the round robin algorithm maintains a running queue of active services, distributes each connection to the next service in the queue, and then sends that service to the end of the queue. Some load balancing algorithms are best suited to handling traffic on websites, others to managing traffic to DNS servers, and others to handling complex web applications used in e-commerce or on company LANs or WANs. The following table lists each load balancing algorithm that the NetScaler appliance supports, with a brief description of how each operates.

LEASTCONNECTION: Which service currently has the fewest client connections. This is the default load balancing algorithm.

ROUNDROBIN:Which service is at the top of a list of services. After that service is Selected for a connection, it moves to the bottom of the list.

LEASTRESPONSETIME: Which load balanced server currently has the quickest response time.

URLHASH:A hash of the destination URL.

DOMAINHASH:A hash of the destination domain.

DESTINATIONIPHASH:A hash of the destination IP address.

SOURCEIPHASH:A hash of the source IP address.

SRCIPDESTIPHASH:A hash of the client's IP address and port.

LEASTBANDWIDTH:Which service currently has the fewest bandwidth constraints.

LEASTPACKETS:Which service currently is receiving the fewest packets.

CUSTOMLOAD:Data from a load monitor.

TOKEN:The configured token.

LRTM:Fewest active connections and the lowest average response time.

The default is “least connection”.For simplicity purpose choose for “Round Robin” as your Load Balancing method for this post.

Then click create and check the status of the Virtual servers is “UP”. Note the method now is displayed as Round Robin.

Step 13: Now that the simple Load balancing configuration is completed in Amazon VPC, let us do a simple testing of our config till this stage. Go to the web browser in the Jump box provisioned in the VPC. Use the Virtual Server IP address of the Load balancer and check whether the web application is accessible

Step 14: In case you want to check the client request details, Login to NetScaler in CLI mode. Use the following command ”stat lb vserver NS-LB01”

Before user access the web application, load balancing service status is as follows:

After user accessed the web application, load balancing status is as follows:

Step 15: How to access the web application in public Internet ? Select the NetScaler Instance in EC2 console and select the “Managed Private IP Address” option.

Step 16: Click the “Assign a secondary private address” option and enter the IP Address and click the “Yes Update” option.

Step 17: Then go to Elastic IP window, select the Elastic IP Address (EIP) and right click the EIP and select the “Associate IP Address” 

Step 18:Select the NetScaler Instance on the “Instance” option, Select the IP Address from “Private IP Address” option and Click “Yes Associate”. The EIP will attach with the Secondary IP Address of NetScaler Instance.

Now Use the Elastic IP (Example: 54.244.36.218) in your browser and access your application from Public Internet.

Understanding Amazon Elastic Load Balancing in Detail

Amazon Elastic Load Balancing is one of the most popular load balancers of AWS Cloud infra. I have compiled the series of articles i have written on this subject here:

Dissecting Amazon Elastic Load Balancing : Amazon ELB is dissected into 18+ points and analysed based on the production implementation experience. This article is a must read for all Amazon ELB users who wants to understand in detail what ELB can do and what are some of the road blocks you can face sometimes when using Amazon ELB. Click here for the article.  

Amazon ELB Implementation Architectures : Amazon ELB can be implemented in variety of architectures in your AWS cloud. Some of them are real bad cases and some of them follow best practices. If you want to understand the common implementation architectures and its pro's/con's refer this article  

Comparison Analysis between Amazon ELB and HAProxy:  HAProxy is a popular OSS software load balancer widely used in Amazon Cloud Infrastructure. This article provides a detailed comparison between the two and analyzes where both of them stand. This article is featured in the HAProxy web site as well. Click here for the article

Configuring Amazon ELB and understanding the parameters in detail: Amazon provides a detailed documentation on how to configure ELB. In this article i have detailed the configuration steps with detailed understanding on the parameters and its implications. Must read of Amazon ELB newbies. Click here for the article 

Geo Distributed Load Balancing using Route53 and Amazon ELB: Some customers would have Geo Distributed their architecture across multiple Amazon EC2 regions. In this article we explore why do we need Geo Distributed architecture, Cost of Latency and how to achieve it using Route 53 + Amazon ELB. Click here for the article


Configuring Amazon ELB With SSL offloading: How to configure SSL with Amazon ELB. Click here for the article What are the benefits of offloading SSL in Amazon ELB. Here

Monitoring Amazon ELB using Amazon CloudWatch and understanding the result and metrics

Why do we need Amazon Elastic Load Balancing and What are its benefits ?

Comparison Analysis:Amazon ELB vs HAProxy EC2

In this article i have analysed Amazon Elastic Load Balancer (ELB) and HAProxy (popular LB in AWS infra) in the following production scenario aspects and fitment:

Algorithms: In terms of algorithms ELB provides Round Robin and Session Sticky algorithms based on EC2 instance health status. HAProxy provides variety of algorithms like Round Robin, Static-RR, Least connection, source, uri, url_param etc. For most of the production cases use Round Robin and Session Sticky is more than enough, But in case you require algorithms like least connection you might have to lean towards HAProxy currently. In future AWS might add this algorithm in their Load Balancer

Spikey or Flash Traffic: Amazon ELB is designed to handle unlimited concurrent requests per second with “gradually increasing” load pattern.  It is not designed to handle heavy sudden spike of load or flash traffic. For example: Imagine an e-commerce website whose traffic increases gradually to thousands of concurrent requests/sec in hours, Whereas imagine use cases like Mass Online Exam or GILT load pattern or 3-Hrs Sales/launch campaign sites expecting 20K+ concurrent requests/sec spike suddenly in few minutes, Amazon ELB will struggle to handle this load volatility pattern. If this sudden spike pattern is not a frequent occurrence then we can Pre-warm ELB else we need to look for alternative Load balancers like HAProxy in AWS infrastructure. If you expect a sudden surge of traffic you can provision X number of HAProxy EC2 instances in running state.

Gradually Increasing Traffic: Both Amazon ELB and HAProxy can handle gradually increasing traffic. But when your needs become elastic and traffic increases in a day, you either need to automate or manually add new HAProxy EC2 instances when the threshold is breached. Also when the load decreases you may need to manually remove the HAProxy EC2 instances from Load Balancing Tier. If you want to avoid these manual efforts you may need to engineer using automation scripts and programs.  Amazon has intelligently automated this elastic problem in their ELB Tier. We just need to configure and use this, that's all.


Protocols : Currently Amazon ELB only supports following protocols: HTTP, HTTPS (Secure HTTP), SSL (Secure TCP) and TCP protocols. ELB supports load balancing for the following TCP ports: 25, 80, 443, and 1024-65535. In case RTMP or HTTP Streaming protocol is needed, we need to use Amazon CloudFront CDN in your architecture. HAProxy can support both TCP and HTTP protocols. In case HAProxy EC2 instance is working in pure TCP mode. A full-duplex connection will be established between clients and servers, and no layer 7 examination will be performed. This is the default mode. It can be used for SSL, SSH, SMTP etc. Current 1.4 version of HAProxy does not support HTTPS protocol natively, you may need to use Stunnel or Stud or Nginx before HAProxy to do the SSL termination. HAProxy 1.5 dev-12 comes with SSL support, it will become production ready soon. 

Timeouts: Amazon ELB currently timeouts persistent socket connections @ 60 seconds if it is kept idle. This condition will be a problem for use cases which generates large files (PDF, reports etc) at backend EC2, sends them as response back and keeps connection idle during entire generation process. To avoid this you'll have to send something on the socket every 40 or so seconds to keep the connection active in Amazon ELB. In HAProxy you can configure very large socket timeout values to avoid this problem. 

White listing IP's :Some Enterprises might want to white list 3rd party Load Balancer IP range in their firewalls . If the 3rd party service is hosted using Amazon ELB it will become a problem. Currently Amazon ELB does not provide fixed or permanent IP address for the Load balancing instances that are launched in its tier. This will be a bottleneck for enterprises which have compulsion to white list the Load balancer IP’s in external firewalls/gateways. For such use cases, currently we can use HAProxy EC2 attached with Elastic IPs as load balancers in AWS infrastructure and white list the Elastic IP's.

Amazon VPC/ Non VPC : VPC- Virtual Private Cloud. Both Amazon ELB and HAProxy EC2 can work inside the VPC and Non VPC environments of AWS.

Internal Load Balancing: Both Amazon ELB and HAProxy can be used for internal load balancing inside VPC. You might provide a service that is consumed internally by the other applications which needs load balancing. ELB and HAProxy can fit in the same. In case internal Load balancing is required in Amazon Non-VPC environments, ELB is not capable currently and HAProxy can be deployed. 

URI/URL based Load balancing: Amazon ELB cannot Load Balance based on URL patterns like other Reverse proxies. Example Amazon ELB cannot direct and load balance between request URLs  www.xyz.com/URL1 and www.xyz.com/URL2. Currently for such use cases you can use HAProxy on EC2.

Sticky problem: This point comes as a surprise to many users using Amazon ELB. Amazon ELB behaves little strange when incoming traffic is originated from Single or  Specific IP ranges, it does not efficiently do round robin and sticks the request to some EC2's only. Since i do not know the ELB internals i assume ELB might be using "Source" algorithm as default for such conditions. No such cases were observed with HAProxy EC2 in AWS unless the balance algorithm is "Source". In HAProxy you can combine "Source" and "Round Robin" efficiently. In case the HTTP request does not have cookie it uses source algorithm, but if the HTTP request has a cookie HAProxy automatically shifts to RR or Weighted. (I will have to check this with AWS team)

Logging: Amazon ELB currently does not provide access to its log files for analysis. We can only monitor some essential metrics using CloudWatch for ELB. We cannot debug load balancing problems, analyze the traffic and access patterns; categorize bots / visitors etc currently because we do not have access to the ELB logs.This will also be a bottleneck for some organizations which has strong audit/compliance requirements to be met at all layers of their infrastructure. In case very strict/specific log requirements are needed, You might need to use HAProxy on EC2, in case it suffices the need. 

Monitoring: Amazon ELB can be monitored using Amazon CloudWatch. Refer this URL for ELB metrics that can be currently monitored: http://harish11g.blogspot.in/2012/02/cloudwatch-elastic-load-balancing.html. CloudWatch+ELB is detailed for most use cases and provides consolidated result of the entire ELB tier in console/API. On the other hand HAProxy provides user interface and stats for monitoring its instances. But if you have farms(20+) of HAProxy EC2 instances it becomes complex to manage this monitoring part efficiently. You can use tools like ServerDensity to monitor such HAProxy farms, but it has huge dependency on NAT instances availability for inside Amazon VPC deployments.

SSL Termination and Compliance requirements:

SSL Termination can be done at 2 levels using Amazon ELB in your application architecture .They are

• SSL termination can be done at Amazon ELB Tier, which means connection is encrypted between Client(browser etc) and Amazon ELB, but connection between ELB and Web/App EC2 is clear. This configuration may not be acceptable in strictly secure environments and will not pass through compliance requirements.
• SSL termination can be done at Backend with End to End encryption, which means connection is encrypted between Client and Amazon ELB, and connection between ELB and Web/App EC2 backed is also encrypted. This is the recommended ELB configuration for meeting the compliance requirements at LB level. 

HAProxy 1.4 does not support SSL termination directly and it has to be done in Stunnel or Stud or Nginx layer before HAProxy. HAProxy 1.5 dev-12 comes with SSL support, it will become production ready soon, i have not yet analyzed/tested the backend encryption support in this version.

Scalability and Elasticity : Most important architectural requirements of web scale systems are scalability and elasticity. Amazon ELB is designed for this and handle these requirements with ease.Elastic Load Balancer does not cap the number of connections that it can attempt to establish with the load balanced Amazon EC2 instances.Amazon ELB is designed to handle unlimited concurrent requests per second. ELB is inherently scalable and it can elastically increase /decrease its capacity depending upon the traffic. According to a benchmark done by RightScale, Amazon ELB was easily able to scale out and handle 20K+ or more concurrent requests /sec. Refer URL:http://blog.rightscale.com/2010/04/01/benchmarking-load-balancers-in-the-cloud/

Note: The load testing was stopped after 20K req/sec by RightScale because ELB kept expanding its capacity. Considerable of DevOps engineering is needed to automate this functionality with HAProxy.

High Availability: Amazon ELB is inherently fault tolerant and a Highly available service. Since it is a managed service, Unhealthy load balancer instances are automatically replaced in ELB tier. In case of HAProxy, you need to do this work yourself and build HA on your own. Refer URL http://harish11g.blogspot.in/2012/10/high-availability-haproxy-amazon-ec2.html to understand more about High Availability @ Load Balancing Layer using HAProxy.

Integration with Other services: Amazon ELB can be configured with work seamlessly with Amazon AutoScaling, Amazon CloudWatch and Route 53 DNS services. The new web EC2 instances launched by Amazon AutoScaling are added to the Amazon ELB for Load balancing automatically and whenever load drops; existing EC2 instances can be removed by Amazon Auto Scaling from ELB. Amazon AutoScaling and CloudWatch cannot be integrated seamlessly with HAProxy EC2 for this functionality. But HAProxy can be integrated with Route53 easily for DNS RR/Weighted algorithms.   

Cost: If you run a ELB in US-East Amazon EC2 region for a month (744 hrs) processing close to 1 TB of data, it will cost around ~26 USD (ELB usage+Data charge).  In case if you use HAProxy (2 X m1.large EC2 for HAProxy, S3 backed AMI, Linux instances, No EBS attached) as base capacity and add upto 4 or more m1.large EC2 depending upon traffic. It will minimum cost 387 USD for EC2 compute + Data Charges to start with. it is very clear and evident that larger deployments can save lots of cost and immensely benefit using Amazon ELB compared to HAProxy on EC2.  

Architecting Highly Available Web App Layer using HAProxy

Option 1: Architecting Web/App Availability-Using HAProxy + AWS Elastic IP 

HAProxy is attached with an AWS Elastic IP(example:50.19.82.183) since it is the entry point to your Web Application. The Elastic IP is assigned to your account and will not change when the Amazon EC2 instance is terminated. We can reuse the same AWS Elastic IP to any other EC2 instance or another HAProxy EC2 in future. The HAProxy Elastic IP is registered Route53 for DNS resolution.

The Web/App EC2’s (imagine Apache EC2 servers) are attached with another series of AWS Elastic IP’s (example:23.23.174.252-255). These Elastic IP’s are configured in the HAProxy configuration file.  In case anyone of the Web/App Apache EC2 goes down, we can manually or automatically spawn a new Web/App EC2 from the Base Golden Image and assign the same Web/App Elastic IP to it again. No Need to change the HAProxy configuration in event of any Web/App EC2 instance failure.

Frequent Sync of Apache Logs and User Uploaded files (if any) to S3 is highly recommended. This will improve the Recovery Point objective and fault tolerance levels of the overall system.

Disadvantages: 

• Re Mapping an AWS Elastic IP will take around 120 seconds (or more sometimes more depending upon the Region-AZ), some users who have been using the particular Web/App EC2 through that Elastic IP will face problems for few seconds.
• Additional cost and Data charges incurred for maintain Elastic IP’s
• This model not suggested for highly scalable systems running hundreds/thousands of Web/App EC2 because of the scarce nature of Elastic IP’s.
• Fractionally (negligible) slower because of Elastic IP -> Private IP translation

Advantages:  

• No change in HAProxy Configuration required during Web/App EC2 instance failure.

Sample HAProxy Config (Elastic IP):

Option 2: Architecting Web/App Availability-Using HAProxy + EC2 Private IP's

HAProxy is attached with an AWS Elastic IP(example:50.19.82.183) since it is the entry point to your Web Application. The Elastic IP is assigned to your account and will not change when the Amazon EC2 instance is terminated. We can reuse the same AWS Elastic IP to any other EC2 instance or HAProxy EC2 in future. The HAProxy Elastic IP is registered Route53 for DNS resolution.

The Web/App EC2's (imagine Apache EC2 servers) are accessed with private IP of the Amazon EC2. These private IP's(example:10.62.186.90-93) are configured in the HAProxy configuration file.  In case the Web/App Apache EC2 goes down, we can manually or automatically spawn a new Web/App EC2. Whenever a new EC2 instance is launched their public and private IP’s will change, so we need to hot reconfigure HAProxy file in event of failure with new Web/App Apache EC2 Private IP address assigned.

Frequent Sync of Apache Logs and User Uploaded files (if any) to S3 is highly recommended. This will improve the Recovery Point objective and fault tolerance levels of the overall system.

Disadvantages:

• Hot reconfigure of HAProxy has to be done whenever the private IP address of Web/App EC2 changes.

Advantages: 

• No additional cost incurred for accessing through Elastic IP.
• Fractionally faster because of access using Private IP

Sample HAProxy Config (Private IP):

Option 3: Architecting Web/App Availability-Using HAProxy with Backup Option

HAProxy is attached with an AWS Elastic IP since it is the entry point to your Web Application. The Elastic IP is assigned to your account and will not change when the Amazon EC2 instance is terminated. We can reuse the same AWS Elastic IP to any other EC2 instance or HAProxy EC2 in future. The HAProxy Elastic IP is registered Route53 for DNS resolution.

The Web/App EC2’s (imagine Apache EC2 servers) are grouped into two farms –Active and Backup. X web/app EC2’s are launched in the Active farms. Y more web/app EC2’s are launched in the backup farms. In event of Web/App EC2 failure in the active farms, requests are automatically diverted to the Web/App Apache EC2's in the Backup farms.  Backup farms can have lesser EC2 servers compared to the active farms in case manual/automated recovery can be done faster.  For website outages caused by heavy load traffic, it is better to have both Active and Backup farms in same size to avoid backup failing quickly. All the Web/App EC2 servers in Active and Backup are accessed with private IP's of the Amazon EC2. This private IP is configured in the HAProxy configuration file.  In case an Web/App Apache EC2 goes down, we can manually or automatically spawn a new Web/App EC2. Whenever a new EC2 instance is launched their public and private IP’s will change, so we need to hot reconfigure HAProxy file in event of failure with new Web/App Apache EC2 Private IP address.

Frequent Sync of Apache Logs and User Uploaded files (if any) to S3 is recommended. This will improve the Recovery Point objective.

Disadvantages:

• It is prohibitively costly to run passive Web/App EC2 in the HAProxy backup mode-You are running a backup farm,keeping it in idle state, paying by hour and using it only when needed( not advisable for clouds). Though this model does not fit the cloud architecture paradigm, i have seen some customers have deployed their Web/App EC2 farm in this fashion. Instead one can adopt Active-Active Web/App EC2 model itself discussed in the next section.

Advantages:  

• Architecture has High availability built in Web/App EC2 layer. Instant switch between EC2 in Active to Backup farms. Ops have sometime to breathe.
• RTO and RPO can be efficiently addressed.
• Only minor advantage i see is version rollouts can be propagated easily. Code rollout first to the Backup farms, Kill the EC2’s in Active Farms and Switch the Active-Backup farms through HAProxy Hot reconfigure. (Note: This scenario can be heavily negated if we start using Puppet /Chef for deployment)

Sample HAProxy Config (Backup Mode):

Option 4: Architecting Web/App Availability-Using HAProxy and AWS Multi-AZ

This is the Recommended architecture for deploying Web/App EC2 under HAProxy inside Amazon for High Availability.
We all know Amazon Web Services Global infrastructure consists of Amazon EC2 Regions. Every Region is designed to be completely isolated from the other Amazon EC2 Region. Every Amazon Region is further sub divided into Availability Zones. By launching EC2 instances in separate Availability Zones (AZ), we can protect our applications from the failure of a single location. To know more about AWS Regions and Availability Zones click here 
In this architecture HAProxy is attached with an AWS Elastic IP and launched in AZ-1.The HAProxy Elastic IP is registered Route53 for DNS resolution.
The Web/App EC2’s (imagine Apache EC2 servers) are launched in 2 or more Availability Zones in the AWS EC2 region. The Private IP addresses of these Web/App EC2 instances are configured in the HAProxy Configuration file. Bringing the Multiple Availability zones into the architecture does not add any extra configuration effort in the HAProxy. 

In event of an Web/App EC2 failure, requests are automatically diverted to the healthy Web/App Apache EC2's in the same or other AZ's (based on LB algorithm). We can manually or automatically spawn a new Web/App EC2 and hot configure their new private IP address into HAProxy.

In event of an entire AZ failure, requests are still automatically diverted to the healthy Web/App Apache EC2's in the other available AZ's (based on LB algorithm). This helps us protect our application from the failure of a single location (Data center) itself and design much more robust HA architecture. 

Frequent Sync of Apache Logs and User Uploaded files (if any) to S3 is recommended. This will improve the Recovery Point objective.

Disadvantages:

• Data charges between AZ
• Little complex to maintain because of Multiple EC2 running in active mode
• No constant NW latency between AZ's makes Web/App EC2 distribution little painful

Advantages:  

• Architecture has recommended levels of High availability built in Web/App EC2 layer.
• RTO and RPO can be efficiently addressed.

Sample HAProxy Config (Private IP + Multi-AZ):

HAProxy- Amazon EC2 Instance Types and AMI Types

Before choosing optimal Amazon EC2 instance Type for HAProxy layer in AWS we need to minimum understand the parameters involved.  They are:

On HAProxy Side

• HAProxy follows an event-driven, single-process model. Event-driven models can address the memory limits, System scheduler limits, and lock contention efficiently than Apache like models because all the tasks are in user-space which allows finer resource and time management.
• Processing several hundreds of tasks in a millisecond using HAProxy is possible, and usually the memory usage will be in the order of a few kilobytes per session compared to Apache-like models (megabytes per process).
• HAProxy is also optimized to perform using very low CPU usage on moderate loads and High loads (Event driven program models generally don't scale well on multi-processor systems)
• From the above it is evident that by design HAProxy, is not memory intensive and does not utilize additional cores.

On Amazon EC2 Side

• Amazon Web Services offers various EC2 instance types.  In addition to this AWS has 32-bit and 64-bit platform. Because of such variety provided by AWS in EC2’s, new users often get confused on which instance type is suitable for the HAProxy tier.
• The virtual interface of Amazon EC2’s peaks at 100K PPS (packets per second) in total (input+output). Earlier load tests have pointed that that a maximum of about 125K PPS were achieved, with an average of ~120K being more typical for M1.Large. During such scenarios, other people running instances on the same physical hardware can be affected.
• Scale out strategy is better than Scale up for many uses cases using HAProxy as Load balancer in Amazon EC2. Even in reference to above point, Scale out minimizes the risk of getting affected by noisy neighbors  
• EC2 instances can be launched from EBS Backed AMI’s or S3 Backed AMI’s – Which is ideal for a HAProxy tier?

On Application Side

• Whether the application supports HTTP or HTTPS protocol?
• What is the concurrency expected in the application LB tier?

Choosing an EC2 Instance Type

Scenario 1: If the application expects around ~4-10K concurrency on HTTP protocol, then have Multiple (2-3) M1.Large EC2 instance Type with S3 Backed AMI’s are ideal. Sync logs to S3. 64 bit platform is preferred. M1.large comes with High IO compared to m1.small or c1.medium. 

Scenario 2: If the application expects around ~100 requests/sec concurrency on HTTP protocol, then a single c1.medium EC2 instance Type with S3 Backed AMI’s are ideal.Sync logs to S3. 32 bit or 64 bit platform both ok. Since the traffic is not heavy c1.medium might be sufficient. T1.micro is suitable only for not so frequent spikey patterns and surely a NO for such scenarios. 

Scenario 3: If the application expects around ~4-10K requests/sec concurrency on HTTPS protocol, then have Multiple M1.XLarge EC2 instance Type with S3 Backed AMI’s. SSL termination needs more memory and CPU. Using “taskset” assign CPU Core affinity for HAProxy and Stud processes. 64 bit platform is mandatory for such use cases. M1.Xlarge comes with High IO compared to lower peers. 

Choosing an EC2 AMI Type

Amazon Web Services has two types of AMI’s – S3 backed AMI and EBS backed AMI.
EBS backed AMI comes with an EBS volume and are faster to launch (~ 20 secs), whereas S3 backed AMI’s are little slower to launch (~150 secs) compared to EBS AMI’s.  EC2 Instances launched from S3 backed AMI’s comes only with ephemeral disks and do not have EBS volume; we need to attach one depending upon the needs.  EBS backed AMI’s cost extra (for the inbuilt EBS volume) than the S3 backed AMI which usually comes only with Ephemeral disks.
When we are constructing a Load Balancing tier using HAProxy in AWS, there are multiple approaches we can adopt, they are:

Option 1: Install HAProxy on a S3 Backed AMI and re bundle -> launch the same. The HAProxy and its logs will reside in the Ephemeral disk of EC2. The logs have to be periodically synced to S3. Log analysis can be done using AWS EMR or other programs. Since there is no dependency on EBS for HAProxy, we can totally avoid performance issues and EBS dependent outages. 

Option 2: Install HAProxy on a S3 Backed AMI and attach an EBS volume for storing the logs. The logs can be kept in EBS as long as we want. Logs can also be periodically synced to S3 or Splunk server. Log analysis can be done using AWS EMR from S3 or Splunk or other programs. 

Option 3: Some customers will prefer a single Linux variant (like RedHat) across all tiers of their technical stack for easy maintenance. Note: Some of the Linux OS variants are currently available only on EBS backed AMI in AWS. Also some customers do not have admin resources to manage their systems and logs efficiently; their HAProxy logs will lie in the EBS disks for months and are analyzed only on need basis. For both the cases, it is preferable to use EBS backed AMI for HAProxy.

Other General recommendations:

• Since HAProxy cannot be clustered, for achieving High Availability, Add Multiple HAProxy EC2 under Route53 in Weighted or DNS RR.
• HAProxy can be scaled out automatically using Custom scripts inside Amazon VPC and Non VPC.
• Depending upon the Load Volatility scenarios (Example: Seasonal loads), optimum EC2 instance capacity for HAProxy should be revisited for cost savings in AWS.