Load Testing tool comparison - JMeter on it's own vs JMeter & BlazeMeter together

Load testing is an important aspect of web applications life cycle on Amazon Cloud. Some of our customers ask us to generate 50000+ RPS to load test the scalability of their application deployed on Amazon cloud. Whenever we used to help such customers and migrate their applications on Amazon cloud for achieving scalability, load testing phase itself becomes a pain. Setting up the Load Testing infrastructure, writing automation around it, Managing, Maintaining and monitoring the load test infrastructure is an headache. Our Load testers and Infrastructure teams were spending considerable time and efforts on the above , instead of focusing only on load testing. We usually work with variety of tools from Grinder, JMeter, HP Load Runner to custom engineered load testing tools during the load testing phase. Some time back , our team started playing around with a SAAS based load testing tool called BlazeMeter. In this article i am going to share our experience in the form comparison between BlazeMeter and JMeter and why BlazeMeter has a bright future.
Blazemeter is a Saas based high scalable load testing tool that handle up to 300,000+ concurrent users. Their load test infrastructure is spread across major AWS regions. Since most of us have been using JMeter for years , the 100 % compatibility  it provides to existing JMeter scripts is a good feature. Blazemeter also provides a Chrome Extension which can record browser actions & convert it to .jmx file.

10 Things i like about BlazeMeter

Point 1) Load Test becomes effective only when the load comes from different IP Addresses similar to real world scenario and not from a single source IP. When multiple virtual user load is generated from the same IP, the router as well as the server tries to cache information and optimize the throughput many times. Hence by using multiple IP addresses for the host, the EC2 server will get an illusion of receiving requests from multiple source IP's.  Also it is better that load is generated from multiple IP's for Amazon ELB to evenly distribute load. Refer URL. BlazeMeter has capability to generate load from IP's which is very important on load testing the cloud applications.

Point 2) Customizing the Network Emulation: Usually online applications will be accessed from multiple devices like PC's, Laptop and mobiles. These devices have multiple network types such as 3G, broad band etc.Also at times times our online application will be accessed from locations which has poor network bandwidth , Both these parameters play an important role in capacity planning and load testing. We can chose the Bandwidth and network type emulation while doing the load test using BlazeMeter. Example we can configure the network type such as Unlimited Internet, 3G, Cable, Wifi etc and. Bandwidth download limit per device can also be set.

Point 3) Controlling the Throughput: Target throughput is a parameter of Apache JMeter that can be used to achieve a required throughput value of the application. A server’s performance need not always satisfy the target throughput value mentioned in JMeter. It could provide more throughput or lesser.The target throughput parameter can be controlled in run time in Blazemeter. Live server monitoring can help us identify if our servers are performing well for say 5000 Hits/sec & change the throughput value in run-time to a higher or lower value based on the server’s performance.

Point 4) Controlling the Agents: Apache JMeter works based on Master-Agent based architecture where the Master controls multiple agents generating the load. The number of agents parameter has to be usually decided before the starting of the test while using JMeter based load testing on Amazon Cloud. Option to dynamically change the throughput value is a very good feature to have while load testing a cloud application requiring thousands of Requests per second. BlazeMeter enables us to add or remove agent instances when a test is running. Any instance can be marked as Master or Slave(Agent) while the test is running.

Point 5) Controlling No. of Simulated Users on Slaves (Agents) : A load test strategy is mainly determined by following parameters like number of concurrent users, ramp up time, no. of test engines and test iterations and the test duration. Apache JMeter allows us to manually configure these values before the test is started. New EC2 instances have to provisioned for the Agents, the IP addresses (Usually Elastic IP) of the slaves/agents has to be manually added to the master. The entire setup has to be maintained, managed and monitored during the test cycles.  This is ok for an load testing environment with few load test agents and low RPS, imagine an environment where you have generate thousands of RPS and having 50+ agents running. This process of managing the EC2 load test infrastructure will become tedious process overall for the load testing teams. In BlazeMeter, once the number of concurrent users is given, the number of test engines, number of threads and engine capacity is chosen automatically. This can be made semi-automatic, where the number of engines & number of threads as well can be selected by user and only engine capacity is chosen by BlazeMeter. Since it is a managed Load Test infrastructure, the Load Testers can concentrate the testing and not managing 100's of EC2 load agents.

Point 6) Integrated Monitoring: 

• BlazeMeter offers live monitoring of essential parameters of test servers when the test is running which enables us to decide on the number & instance type for the test. In the conventional Apache JMeter load test setup in Amazon EC2 we have to observe the Key parameters using AWS Cloudwatch.
• Blaze Meter provides AWS Cloud watch integration.An account with IAM access has to be created and
• AWS Access Key & Secret Key values have to be configured so that the metrics are available in the Blazemeter’s dashboard. This features helps us to understand how the assets in the cloud are reacting to our load tests and help us accordingly tune the infrastructure.
• While performing load testing, it is important not only to monitor your Web Servers & Databases but also the agents from where the load is generated . The New Relic plugin gives us the front end KPIs and back end KPIs. 
• BlazeMeters’s frontend KPIs provide insight on how many users are actually trying to access your website, mobile site or mobile apps.
• BlazeMeters’s backend KPIs show how many users are getting through to your applications.

Point 7) Blazemeter allows us to have a different csv file per load test engine. Though this possible in Apache JMeter, it had to be done manually by copying the files onto the JMeter Agent EC2 instances and have the same filename since the agents refer to the Master’s properties. Blazemeter allows us to parameterize the values of even filenames and have different csv files in each engine without giving us to the trouble of copying files into specific EC2 instances & holds the files in a common repository so that it can be referred from there to each agent.

Point 8) Run the load test using older version of JMeter scripts: Old scripts can be reusable with this feature of BlazeMeter which lets us run the test using any version of Apache JMeter right from version 2.3.2 to 2.10. Some complex scripts prepared some months/years ago can be still be made usable and need not be redone. Saves efforts and costs.

Point 9) Schedule the Test & Stay Relaxed: BlazeMeter as well as JMeter lets you schedule your test duration & test time so that we can run longevity test at any time of the day. Even weekly scheduling is possible in BlazeMeter it is an added advantage, though it is not widely used.

Point 10) Interesting Plug-ins provided by Blazemeter :

• Integration with Google Analytics: At the time of scripting, it is enough if we select the Google Analytics Option & provide account details of Google Analytics. BlazeMeter obtains the last 12 months of data and creates a test with 5 most visited pages and sets up the number of concurrent users based on that record.
• Integration with WordPress: BlazeMeter provides integration with WordPress where WordPress users can test their App by using the BlazeMeter plug-in without any scripting.
• Integration with Drupal & Jenkins: Plugins are available to load test Drupal & Jenkins servers as well. 

Post Co Authored with Harine 8KMiles.

AWS Cost Saving Tip 16: Periodically remove your unwanted AWS resources and save costs

Thanks "@HighScalability.com"  for referencing this post "Stuff The Internet Says On Scalability For October 11th, 2013"

Following are some of the AWS assets that needs to be periodically reviewed for under utilization and should be removed as part of your IT process to avoid cost leakage in AWS. 

Remove unwanted AMI's: Periodically audit for unwanted AMI's in your dev/test/production environment and remove them. Tagging the AMI's with proper identifiers can help you during the cleansing process.

EBS snapshots are incremental in nature and only the changes are moved to them. The unwanted snapshots should be periodically identified (Tagging snapshots will help you) and deleted as part of your IT Infrastructure operations. EBS snapshots are priced $0.095 per GB-month of data stored and you can save few hundred $$$ depending upon the volume of unwanted snapshots stored.To know more about EBS Snapshots

EBS Volumes: In development and testing environments of large enterprises we can often find lots of EBS volumes provisioned and lying unattached or unused. EBS Volumes are priced at $0.10 per GB-month of provisioned storage. Usually 10-30 GB are the most used volume ranges for the dev/test. Imagine a enterprise having few hundred EC2's for their Dev/Test infrastructure and 10-25% of their storage are unused, Periodically identifying , tagging, consolidating and cleansing them will save few hundred to thousands of $$$ for the enterprises.To know more about EBS Volumes

S3 Buckets and Objects: Unused objects in S3 buckets have to be frequently deleted. Objects with a limited life period can be configured to expire dynamically using S3 object expiration policies. Objects that are rarely used can be archived to Amazon Glacier from S3. This exercise is an important candidate for Media and online companies using AWS. To know more about other S3 Cost saving tips 1 | 2.

Elastic IP is a scarce resource in AWS cloud, in event the EIP's attached to your account are not associated with any running EC2 instance, you will be charged $0.005 per hour on a prorata basis.
EIP's can usually be leaked in: 

• Dev/Test environments, where the developer disassociates the EIP and keeps it unused.
• In production, where sometimes EIP are assigned to Auto Scaled EC2 instances using scripts and they are not properly released back after EC2 instances are terminated by AWS.

Though if the EIP is kept unused can lead to around only ~$3.5 leakage per month, if no proper tracking procedures are in place many such EIP's can be residing in your account unused and lead to more such leakages.

ELB: Unused and Unwanted Elastic Load Balancers should be removed periodically to avoid cost leakage. A single ELB can cost around ~216 USD if kept unused a year. Usually ELB leakage is prone to happen in Dev/Test environments compared to production. To know more about Amazon ELB

Monitoring : Detailed monitoring option of CloudWatch is generally not required for Development EC2 instances. On the other hand, they are required on Load Testing and Production environments for monitoring the health. Custom Metrics are also good candidates for Production and Load Test environments. Constant review of the infrastructure in AWS has to be done to identify whether such resources(over provisioning) are associated with Dev/Test env and related cost leakages can be avoided.Detailed monitoring costs $3.50 per instance per month, provided at 1-minute frequency and Custom Metrics costs $0.50 per metric per month.

Glacier archives : Amazon Glacier is designed for use cases where data is retained for months, years, or decades. But there are times where you may find some data life has expired and they can be removed. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. Please ensure it is part of you IT process. To know more about other Cost savings in Glacier 1 | 2 | 3

Other small stuff :

• AWS charges $1.00 per month for inactive data pipelines. They should be periodically reviewed for leakages.  

Note:

• There are times where an AWS asset can be created in a non frequently used AWS Region mistakenly by a developer and it can be lying idle for months. Have an automated or manual IT process to periodically cleanse the above leakages points in "ALL" AWS Regions and not just your most frequently used regions.
• Many times on large AWS deployments/Dev/Test environments, it becomes difficult to track who created and who is currently using the AWS resources. AMI's, EBS snapshots, Volumes and S3 buckets should be tagged for proper identification. Identifiable names should be kept for other assets (wherever applicable) , so that it helps your IT team during the cleansing process. 
• Employ automation using scripts/programs or use governance tools ( which i will be discussing later) to monitor under utilization of resources and there by remove them.

Other Cost Saving Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

Load Balancing in Amazon Web Services

Load balancing is one of the most important technique to be followed for architecting highly scalable and available applications on AWS cloud. Keeping this in mind, I have compiled my experience and articles i have written on this subject as "Load balancing on AWS" series.  

Dissecting Amazon Elastic Load Balancing : Amazon ELB is dissected into 18+ points and analysed based on the production implementation experience. This article is a must read for all Amazon ELB users who wants to understand in detail what ELB can do and what are some of the road blocks you can face sometimes when using Amazon ELB. Click here for the article.  

Amazon ELB Multi region migration checklist: Amazon Elastic Load Balancing has a Amazon EC2 Regional scope. It needs to be migrated to alternate Amazon EC2 region in event of DR or during new migration setup . I am sharing my experience in this post as few checklists/areas that needs to be taken care during this Amazon ELB migration to alternate EC2 region. Refer Article

Amazon ELB Implementation Architectures : Amazon ELB can be implemented in variety of architectures in your AWS cloud. Some of them are real bad cases and some of them follow best practices. If you want to understand the common implementation architectures and its pro's/con's refer this article  

Comparison Analysis between Amazon ELB and HAProxy:  HAProxy is a popular OSS software load balancer widely used in Amazon Cloud Infrastructure. This article provides a detailed comparison between the two and analyzes where both of them stand. This article is featured in the HAProxy web site as well. Click here for the article

Architecting High Availability @ HAProxy Load Balancing Tier: Not all the times Amazon ELB is used as the choice of load balancer in Amazon Web Services. But deploying an OSS load balancer in AWS has got is own set of considerations and high availability is one of them. In this post i explore how to design High Availability @ HAProxy load balancing layer. Refer Article

Architecting Highly Available Web App Layer using HAProxy Load Balancing : In this post we explore four options of Load balancing Web/App Tier using HAProxy in AWS. Refer Article

Configuring Citrix NetScaler Load Balancing on AWS: Citrix NetScaler is a popular Load balancer in the Enterprise world. Hardware and virtualized NetScaler has been serving many popular enterprise online assets for years. How to configure Citrix Netscaler based Load Balancing in AWS is explained in this post. Refer Article

Load Balancing Amazon RDS Read Replica's using HAProxy: In this post, let us explore how HAProxy can be used to solve load balancing between RDS Read Replica's in Amazon Cloud.Refer Article

Choosing right HAProxy- Amazon EC2 Instance Types and AMI Types for load balancing: Before choosing optimal Amazon EC2 instance Type for HAProxy Load balancing layer in AWS we need to minimum understand some important factors involved. Refer Article 

Configuring Amazon ELB and understanding the parameters in detail: Amazon provides a detailed documentation on how to configure ELB. In this article i have detailed the configuration steps with detailed understanding on the parameters and its implications. Must read of Amazon ELB newbies. Click here for the article 

Web Session Synchronization patterns in AWS: Architectures for synchronizing sessions of load balanced Web/App EC2 is detailed in this post. Refer Article

Geo Distributed Load Balancing using Route53 and Amazon ELB: Some customers would have Geo Distributed their architecture across multiple Amazon EC2 regions. In this article we explore why do we need Geo Distributed architecture, Cost of Latency and how to achieve it using Route 53 + Amazon ELB. Click here for the article

Deeper Health Checks and Problems in Load Balancing in AWS: Health Checks are one of the essential mechanisms that helps you to keep N-Tiered system highly available.This sounds simple and straight forward right, but some of the customers i have consulted follow a much deeper Health check diagnostic mechanism and it might have problems when migrated to AWS cloud. Let us explore this case in detail in this post. Refer Article

Configuring Amazon ELB With SSL offloading: How to configure SSL with Amazon ELB. Click here for the article What are the benefits of offloading SSL in Amazon ELB. Here

Monitoring Amazon ELB using Amazon CloudWatch and understanding the result and metrics

Why do we need Amazon Elastic Load Balancing and What are its benefits ?

Deeper Health Checks and Problems in Load Balancing in AWS

Health Checks are one of the essential mechanisms that helps you to keep N-Tiered system highly available. Usually a simple script or program is deployed on the Web/App Server. The Health check component of Load Balancer is configured to frequently call this script in Web/App Server in a light weight protocol. Based on the response from the script/program the Load balancer decides the status of the Web/App Servers and accordingly direct the requests to healthy Web/App servers. This is a usual mechanism that is followed in all popular load balancers like Amazon ELB, Netscaler, HAProxy and NGinx in AWS cloud. This sounds simple and straight forward right, but some of the customers i have consulted follow a much deeper Health check diagnostic mechanism and it might have problems when migrated to AWS cloud. 
Let us explore this case in detail :

What is the architecture ?
A simple multi-tiered architecture with : A  load balancer deployed at the front. The Web/App Server has the health script/program. The database is MySQL deployed Master+ Slave mode.  

What is deeper Health Check ?
The script/program deployed in the Web/App Server is little intelligent; when it is called by load balancer it performs simple operations and checks the status of the Database. So when you get a response back from the health check script / program you are verifying whether the health of DB and Web/App server is sound at the load balancer tier.

What is the problem scenario ?
Imagine when migrating this infrastructure to AWS you have adopted the standard architecture pattern consisting of :

• Amazon ELB is used as the Load balancer
• Web/App Server in auto scaling mode
• MySQL moved to Amazon RDS+Multi-AZ with RR

Now let us explore this problem in detail : 

• Imagine the any of the following condition in your production, network between database and Web/App is down intermittently for few minutes or RDS MySQL is elevating the Hot Standby as new Master. In such scenarios, the health check response actually timeouts at Database level, whereas the Load Balancer will mark the even the healthy App Servers as unhealthy because of the deeper health checks. This is not good especially for Amazon Auto Scaled scenario's where Amazon ELB marks Web/App EC2 as unhealthy because of deeper health check and Amazon Auto Scaling keeps restarting the Web/App EC2 auto automatically to maintain minimum healthy farm. This unwanted effect can cascade the overall availability and surely not good for the production in AWS. So in short Deeper Health checks are not surely recommended for complex N-Tier systems that follows Auto scaling/healing and Service oriented architecture patterns in AWS. 
• Usually the purpose of health check is to check the status of next tier or service consumed by a particular tier.  Deeper health checks is heavy weight and it usually takes much more time to respond because majority of your tiers are exercised in this process. If we set this frequency too aggressive, then health checks itself will eat lots of your CPU. So the frequency of the health checks and the response time out have to be set considerably large. Also during heavy traffic scenario, such heavy weight calls can be queued and you might not get faster response in deeper health checks. 
• Deeper health checks are usually suitable for simple and fixed infrastructures. When your infrastructure is non elastic , the decisions are taken manually by the ops team after analyzing the particular failing tier. For Elastic Auto scaled workloads in AWS it is better to isolate the health checks of load balancing tier separate from Deeper Health checks that can be used for assessing the availability of the infrastructure.

Amazon ELB Multi region migration checklist

Amazon Elastic Load Balancing has a Amazon EC2 Regional scope. It needs to be migrated to alternate Amazon EC2 region in event of DR or during new migration setup . I am sharing my experience in this post as few checklists/areas that needs to be taken care during this Amazon ELB migration to alternate EC2 region:

1. Amazon ELB automatically distributes incoming application traffic across multiple Amazon EC2 instances in Multiple Availability Zones. Amazon EC2 regions have different levels of AZ's, US-East EC2 region has 5 AZ's and Singapore has 2 AZ's. When you are migrating the ELB make sure your configuration matches the levels of target Amazon EC2 region Availability zones .
2. When Amazon ELB is created on target Amazon EC2 region, the ELB endpoint will be different from the source one. This has to be re configured on your Route 53 and other thick clients during migration.
3. Since the ELB end point is different, if you had raised any pre-warming request to AWS support , please raise it again for the target region with new ELB end point. This is valid in use cases where you are moving the traffic from primary region to target region in event of outage. 
4. SSL certificates associated on the old Elastic Load Balancing service (source regions) need to be re configured to the new Elastic Load Balancing service in the Target region.
5. Security Groups are region specific and has to be migrated/recreated in the target region. Make sure the security groups of ELB in target region matches the source region
6. Amazon ELB is not an isolated service and usually it will be used in production with Amazon EC2, Auto Scaling group and Cloud Watch Alarms. In addition there will be health checks, time outs, thresholds etc configured specifically for an application in Amazon ELB. You need to ensure that all these items associated with Amazon ELB are transferred target region for proper functioning of the application. Using Amazon CloudFormation templates for creating infrastructure properties will surely help and ease the efforts involved during such ELB migration.
7. To load balance your EC2 instances launched in default VPC, you have to create your load balancers within your default VPC. When you create a load balancer within default VPC, Elastic Load Balancing automatically creates a security group by defining the ports specified for the load balancer to be opened. Default VPC is currently not available in US-East. In case your Source region is US-West and you are planning your DR or migration with Target region as US-East, you need to be aware of this situation. 
8. In case Amazon ELB and Internal ELB are used inside Amazon VPC, you should ensure that the IP address ranges being used in source and target regions matches and aligns properly, so that routing and load balancing continues to work after migration. 

Other Multi Region Migration articles:

Security Groups - Multi Region Migration checklist

Elastic IP - Multi region Migration checklist, tips , best practices

EC2 AMI Copy - Multi Region Migration

MySQL Amazon EC2 Cross region Master-Slave Replication using SSL

Amazon EBS Snapshot copying- Performance, Tips and How to

Amazon EC2 AMI Copy Feature - Performance, Tips and How To

Understanding Amazon Elastic Load Balancing in Detail

Amazon Elastic Load Balancing is one of the most popular load balancers of AWS Cloud infra. I have compiled the series of articles i have written on this subject here:

Dissecting Amazon Elastic Load Balancing : Amazon ELB is dissected into 18+ points and analysed based on the production implementation experience. This article is a must read for all Amazon ELB users who wants to understand in detail what ELB can do and what are some of the road blocks you can face sometimes when using Amazon ELB. Click here for the article.  

Amazon ELB Implementation Architectures : Amazon ELB can be implemented in variety of architectures in your AWS cloud. Some of them are real bad cases and some of them follow best practices. If you want to understand the common implementation architectures and its pro's/con's refer this article  

Comparison Analysis between Amazon ELB and HAProxy:  HAProxy is a popular OSS software load balancer widely used in Amazon Cloud Infrastructure. This article provides a detailed comparison between the two and analyzes where both of them stand. This article is featured in the HAProxy web site as well. Click here for the article

Configuring Amazon ELB and understanding the parameters in detail: Amazon provides a detailed documentation on how to configure ELB. In this article i have detailed the configuration steps with detailed understanding on the parameters and its implications. Must read of Amazon ELB newbies. Click here for the article 

Geo Distributed Load Balancing using Route53 and Amazon ELB: Some customers would have Geo Distributed their architecture across multiple Amazon EC2 regions. In this article we explore why do we need Geo Distributed architecture, Cost of Latency and how to achieve it using Route 53 + Amazon ELB. Click here for the article


Configuring Amazon ELB With SSL offloading: How to configure SSL with Amazon ELB. Click here for the article What are the benefits of offloading SSL in Amazon ELB. Here

Monitoring Amazon ELB using Amazon CloudWatch and understanding the result and metrics

Why do we need Amazon Elastic Load Balancing and What are its benefits ?

Comparison Analysis:Amazon ELB vs HAProxy EC2

In this article i have analysed Amazon Elastic Load Balancer (ELB) and HAProxy (popular LB in AWS infra) in the following production scenario aspects and fitment:

Algorithms: In terms of algorithms ELB provides Round Robin and Session Sticky algorithms based on EC2 instance health status. HAProxy provides variety of algorithms like Round Robin, Static-RR, Least connection, source, uri, url_param etc. For most of the production cases use Round Robin and Session Sticky is more than enough, But in case you require algorithms like least connection you might have to lean towards HAProxy currently. In future AWS might add this algorithm in their Load Balancer

Spikey or Flash Traffic: Amazon ELB is designed to handle unlimited concurrent requests per second with “gradually increasing” load pattern.  It is not designed to handle heavy sudden spike of load or flash traffic. For example: Imagine an e-commerce website whose traffic increases gradually to thousands of concurrent requests/sec in hours, Whereas imagine use cases like Mass Online Exam or GILT load pattern or 3-Hrs Sales/launch campaign sites expecting 20K+ concurrent requests/sec spike suddenly in few minutes, Amazon ELB will struggle to handle this load volatility pattern. If this sudden spike pattern is not a frequent occurrence then we can Pre-warm ELB else we need to look for alternative Load balancers like HAProxy in AWS infrastructure. If you expect a sudden surge of traffic you can provision X number of HAProxy EC2 instances in running state.

Gradually Increasing Traffic: Both Amazon ELB and HAProxy can handle gradually increasing traffic. But when your needs become elastic and traffic increases in a day, you either need to automate or manually add new HAProxy EC2 instances when the threshold is breached. Also when the load decreases you may need to manually remove the HAProxy EC2 instances from Load Balancing Tier. If you want to avoid these manual efforts you may need to engineer using automation scripts and programs.  Amazon has intelligently automated this elastic problem in their ELB Tier. We just need to configure and use this, that's all.


Protocols : Currently Amazon ELB only supports following protocols: HTTP, HTTPS (Secure HTTP), SSL (Secure TCP) and TCP protocols. ELB supports load balancing for the following TCP ports: 25, 80, 443, and 1024-65535. In case RTMP or HTTP Streaming protocol is needed, we need to use Amazon CloudFront CDN in your architecture. HAProxy can support both TCP and HTTP protocols. In case HAProxy EC2 instance is working in pure TCP mode. A full-duplex connection will be established between clients and servers, and no layer 7 examination will be performed. This is the default mode. It can be used for SSL, SSH, SMTP etc. Current 1.4 version of HAProxy does not support HTTPS protocol natively, you may need to use Stunnel or Stud or Nginx before HAProxy to do the SSL termination. HAProxy 1.5 dev-12 comes with SSL support, it will become production ready soon. 

Timeouts: Amazon ELB currently timeouts persistent socket connections @ 60 seconds if it is kept idle. This condition will be a problem for use cases which generates large files (PDF, reports etc) at backend EC2, sends them as response back and keeps connection idle during entire generation process. To avoid this you'll have to send something on the socket every 40 or so seconds to keep the connection active in Amazon ELB. In HAProxy you can configure very large socket timeout values to avoid this problem. 

White listing IP's :Some Enterprises might want to white list 3rd party Load Balancer IP range in their firewalls . If the 3rd party service is hosted using Amazon ELB it will become a problem. Currently Amazon ELB does not provide fixed or permanent IP address for the Load balancing instances that are launched in its tier. This will be a bottleneck for enterprises which have compulsion to white list the Load balancer IP’s in external firewalls/gateways. For such use cases, currently we can use HAProxy EC2 attached with Elastic IPs as load balancers in AWS infrastructure and white list the Elastic IP's.

Amazon VPC/ Non VPC : VPC- Virtual Private Cloud. Both Amazon ELB and HAProxy EC2 can work inside the VPC and Non VPC environments of AWS.

Internal Load Balancing: Both Amazon ELB and HAProxy can be used for internal load balancing inside VPC. You might provide a service that is consumed internally by the other applications which needs load balancing. ELB and HAProxy can fit in the same. In case internal Load balancing is required in Amazon Non-VPC environments, ELB is not capable currently and HAProxy can be deployed. 

URI/URL based Load balancing: Amazon ELB cannot Load Balance based on URL patterns like other Reverse proxies. Example Amazon ELB cannot direct and load balance between request URLs  www.xyz.com/URL1 and www.xyz.com/URL2. Currently for such use cases you can use HAProxy on EC2.

Sticky problem: This point comes as a surprise to many users using Amazon ELB. Amazon ELB behaves little strange when incoming traffic is originated from Single or  Specific IP ranges, it does not efficiently do round robin and sticks the request to some EC2's only. Since i do not know the ELB internals i assume ELB might be using "Source" algorithm as default for such conditions. No such cases were observed with HAProxy EC2 in AWS unless the balance algorithm is "Source". In HAProxy you can combine "Source" and "Round Robin" efficiently. In case the HTTP request does not have cookie it uses source algorithm, but if the HTTP request has a cookie HAProxy automatically shifts to RR or Weighted. (I will have to check this with AWS team)

Logging: Amazon ELB currently does not provide access to its log files for analysis. We can only monitor some essential metrics using CloudWatch for ELB. We cannot debug load balancing problems, analyze the traffic and access patterns; categorize bots / visitors etc currently because we do not have access to the ELB logs.This will also be a bottleneck for some organizations which has strong audit/compliance requirements to be met at all layers of their infrastructure. In case very strict/specific log requirements are needed, You might need to use HAProxy on EC2, in case it suffices the need. 

Monitoring: Amazon ELB can be monitored using Amazon CloudWatch. Refer this URL for ELB metrics that can be currently monitored: http://harish11g.blogspot.in/2012/02/cloudwatch-elastic-load-balancing.html. CloudWatch+ELB is detailed for most use cases and provides consolidated result of the entire ELB tier in console/API. On the other hand HAProxy provides user interface and stats for monitoring its instances. But if you have farms(20+) of HAProxy EC2 instances it becomes complex to manage this monitoring part efficiently. You can use tools like ServerDensity to monitor such HAProxy farms, but it has huge dependency on NAT instances availability for inside Amazon VPC deployments.

SSL Termination and Compliance requirements:

SSL Termination can be done at 2 levels using Amazon ELB in your application architecture .They are

• SSL termination can be done at Amazon ELB Tier, which means connection is encrypted between Client(browser etc) and Amazon ELB, but connection between ELB and Web/App EC2 is clear. This configuration may not be acceptable in strictly secure environments and will not pass through compliance requirements.
• SSL termination can be done at Backend with End to End encryption, which means connection is encrypted between Client and Amazon ELB, and connection between ELB and Web/App EC2 backed is also encrypted. This is the recommended ELB configuration for meeting the compliance requirements at LB level. 

HAProxy 1.4 does not support SSL termination directly and it has to be done in Stunnel or Stud or Nginx layer before HAProxy. HAProxy 1.5 dev-12 comes with SSL support, it will become production ready soon, i have not yet analyzed/tested the backend encryption support in this version.

Scalability and Elasticity : Most important architectural requirements of web scale systems are scalability and elasticity. Amazon ELB is designed for this and handle these requirements with ease.Elastic Load Balancer does not cap the number of connections that it can attempt to establish with the load balanced Amazon EC2 instances.Amazon ELB is designed to handle unlimited concurrent requests per second. ELB is inherently scalable and it can elastically increase /decrease its capacity depending upon the traffic. According to a benchmark done by RightScale, Amazon ELB was easily able to scale out and handle 20K+ or more concurrent requests /sec. Refer URL:http://blog.rightscale.com/2010/04/01/benchmarking-load-balancers-in-the-cloud/

Note: The load testing was stopped after 20K req/sec by RightScale because ELB kept expanding its capacity. Considerable of DevOps engineering is needed to automate this functionality with HAProxy.

High Availability: Amazon ELB is inherently fault tolerant and a Highly available service. Since it is a managed service, Unhealthy load balancer instances are automatically replaced in ELB tier. In case of HAProxy, you need to do this work yourself and build HA on your own. Refer URL http://harish11g.blogspot.in/2012/10/high-availability-haproxy-amazon-ec2.html to understand more about High Availability @ Load Balancing Layer using HAProxy.

Integration with Other services: Amazon ELB can be configured with work seamlessly with Amazon AutoScaling, Amazon CloudWatch and Route 53 DNS services. The new web EC2 instances launched by Amazon AutoScaling are added to the Amazon ELB for Load balancing automatically and whenever load drops; existing EC2 instances can be removed by Amazon Auto Scaling from ELB. Amazon AutoScaling and CloudWatch cannot be integrated seamlessly with HAProxy EC2 for this functionality. But HAProxy can be integrated with Route53 easily for DNS RR/Weighted algorithms.   

Cost: If you run a ELB in US-East Amazon EC2 region for a month (744 hrs) processing close to 1 TB of data, it will cost around ~26 USD (ELB usage+Data charge).  In case if you use HAProxy (2 X m1.large EC2 for HAProxy, S3 backed AMI, Linux instances, No EBS attached) as base capacity and add upto 4 or more m1.large EC2 depending upon traffic. It will minimum cost 387 USD for EC2 compute + Data Charges to start with. it is very clear and evident that larger deployments can save lots of cost and immensely benefit using Amazon ELB compared to HAProxy on EC2.  

High Availability @ Load Balancing Layer-HAProxy / ELB

Architecting High Availability at the Load Balancing layer is one of the important aspects in the web scale systems in AWS. We can follow multiple strategies for achieving the same. I am listing some of the designs for achieving the same.

Pattern 1: Route 53 DNS RR + HAProxy

Route53 is a Managed DNS service provided by Amazon Web Services. Route 53 supports Round robin and weighted algorithms. If the Route53 DNS server has several entries for a given hostname, it will return all of them in a rotating order. This way, various users will see different addresses for the same name and will be able to reach different EC2 instances in LB Tier.

$ host -t a HAProxyTestXYZ.com

HAProxyTestXYZ.com. has address 50.19.82.183 (Primary EIP)

HAProxyTestXYZ.com. has address 23.23.174.254 (Secondary EIP)

Example, if we attach the Elastic IP’s of 2 HAProxy EC2 instances under the Route 53, both the IP’s are sent to the user browsers by the Route 53 DNS. In case, the algorithm configured is Round Robin at the Route 53 level, then browser- 1 will get EIP-1(50.19.82.183) of HAProxy-1 as the primary IP and browser -2 will get EIP-2 (23.23.174.254) of HAProxy-2 as the primary IP in rotation basis.  The browser- 1 will contact the HAProxy-1 and in case HAProxy-1 is not reachable it will contact the secondary EIP which is HAproxy-2 and so forth. This is an age old technique generally used by search engines, content servers (or) web scale systems for achieving scalability in LB layer. But this method does not provide any means of High availability @ LB layer. It requires additional measures to permanently check the HAProxy EC2 LB instances status and switch a failed EC2 instance EIP to another HAProxy EC2 LB. For this reason, this pattern is generally used as a complementary solution in High Availability, not as a primary one.  For achieving better stability at this layer in AWS, I usually recommend having 2 or more HAProxies distributed on multiple AZ’s inside the Amazon EC2 region. This way if one of the HAProxy is down, the website still functions with the help of other HAProxies and even if the entire Amazon EC2 AZ is down still the HAProxies in the other AZ can handle the requests and keep the website active. Some load tests have proven that HAProxy on m1.large EC2 instance can handle close to ~4500+ HTTP requests/second. So depending upon the number of concurrent requests/sec needed on your application you can go ahead and attach multiple HAProxy EC2 instances to the Route 53. Now that we achieved availability horizontally using the Route53 DNS Round Robin in HAProxy layer let us try to understand the intricacies behind this architecture.  Since we now have 2 or more HAProxies what will happen to the contextual web sessions data that resides in the application servers. HAProxies need to know in which application server the session data of the user resides else the requests will have authorization failures.

There are 2 architecture designs we can follow for solving this contextual problem they are:

Stateless Application design:  This is the recommended and widely used design. The web session data is separated out from the Web/App server memory and they are kept in common cache stores like MemCacheD, TerraCotta etc. Refer this URL to understand common patterns followed in AWS: http://harish11g.blogspot.in/2012/10/web-session-synchronization-aws-ec2.html

Since the session data is now kept in a common store like MemCacheD, HAProxies can direct their requests to any of the web/app servers attached under it without knowing where the session state is mapped. Whenever any web/app server receives the request from any of the HAProxies, it will validate and authorize the session data from the common store.  In event of any HAProxy or Web/App EC2 failure still the website functions without problems because other HAProxies and Web/App servers are still able to handle the subsequent requests. Thus we achieve availability and scalability on the HAProxy/Load Balancing layer following this model.

Sticky Application design: Things are usually not ideal and the way we assume to be in real world. Some applications are still designed with stateful nature and they store the session data, cache data etc. in their web/app server memory.  We can always recommend the application teams to re-architect this model to stateless, but not always this suggestion works for short term migrations, inter dependencies etc.  So as architects we need to find way to live and cope up with this design and still try to achieve availability on the load balancing layer. HAProxy follows a technique called as “Cookie Learning” and “Cookie Insertion” to help state full applications. HAProxy can be configured to learn the application cookie ("JSESSIONID"), when HAProxy receives the user's request, it checks if it contains this particular cookie and a known value. If this is not the case, it will direct the request to any Web/App EC2 server, according to the load balancing algorithm configured. HAProxy will then extract the cookie value from the response and add it along with the server's identifier to a local table. When the user request comes back again, the load balancer sees the cookie, lookups the table and finds the Web/App EC2 server to which it forwards the request. Let me detail this important flow a little bit;

HAProxy-1 EC2 instance will receive client’s requests from the browser. If a request does not contain a cookie, it will be forwarded to a valid Web/App EC2 Instance Apache-A. In return, if a JESSIONID cookie is seen, the Web/App EC2 Instance name (Example “A”)will be prefixed into it, followed by a delimiter ('~') like "JSESSIONID=A~xxx".When the browser client requests again with the cookie  JSESSIONID=A~xxx", HAProxy-1 will know that it must be forwarded to Web/App Instance Apache-A. The EC2 Instance name ”A” will then be extracted from cookie before it is sent to the Web/App EC2 Instance Apache-A.

If Web/App EC2 Instance Apache-A dies, then requests will be sent to another valid server Web/App EC2 Instance Apache-B by LB and the cookie will be reassigned.

If HAProxy-1 itself dies, then requests will be sent to HAProxy-2 which will identify the Web/App EC2 instance to forward the request. This way even if the subsequent requests moves from HAProxy-1 to HAProxy-2 in event of HAProxy-1 failure, still the requests are sent to the same Web/App instance Apache-A by the cookie learning/insertion mechanism of HAProxy.

Sample HAProxy Settings to achieve this

listen webfarm 192.168.1.1:80

       mode http

       balance roundrobin

       cookie JSESSIONID prefix

       option httpclose

       option forwardfor

       option httpchk HEAD /index.html HTTP/1.0

       server Apache-A 192.168.1.11:80 cookie A check

       server Apache-B 192.168.1.12:80 cookie B check

Note: You can use more sophisticated DNS services like UltraDNS , DNSMadeEasy etc also in this architecture to better control the Load balancing and traffic direction at the DNS level. 

Pattern 2: Route 53 DNS RR + HAProxy in Active-Passive mode

This is an extension of the Route53 DNS RR pattern and everything discussed in the previous pattern still applies to this context. In addition to associating HAProxies horizontally under Route53, we will build availability for every HAProxy vertically as well in this pattern. High Availability is built taking into consideration HAProxy process failure and HAProxy EC2 instance failure.

2 or more HAProxies from multiple AZ’s are taken and they are attached with Amazon Elastic IP’s. These Elastic IP’s are then associated in Route 53 with DNS RR. These HAProxies are now “Active” and are ready to handle the user requests. For HA, another equivalent set of HAProxies are launched in the respective AZ’s as “Standby”. In event of the “Active” HAProxy failure, the Standby HAProxy remaps to the same Amazon Elastic IP takes over the subsequent requests from the client.

In the above diagram, there are 2 HAProxies in “Active” state with Elastic IP’s 50.19.82.183 & 23.23.174.254. They are deployed across Multiple Availability Zones inside an Amazon EC2 region. Another 2 HAProxies are launched in respective AZ’s, but they are kept idle in “Standby” state. In event of HAProxy-1 (EIP: 50.19.82.183) failure the Elastic IP is remapped to Standby HAProxy-3 in the same AZ. The remapping takes ~60 seconds and the HAPorxy-3 will be handling the subsequent requests directed by the browsers to the 50.19.82.183 IP.

Broadly there are 2 levels of failure in this pattern;

Failure @ HAProxy Process level

Failure @ HAProxy EC2 instance level

Failure @HAProxy Process level:  When HAProxy Process at the “Active” server fails; we can detect this using KeepAliveD and switch the Elastic IP from Active -> Standby. We have observed it takes ~60-120 seconds for the standby to takeover. During this time the particular HAProxy alone will be unreachable. KeepAliveD script is configured in both the Active and standby HAProxy EC2 instance. KeepAliveD implements a set of checkers to dynamically and adaptively maintain and manage load balanced server pool according to their health. High availability is achieved by Virtual Router Redundancy Protocol VRRP protocol of the KeepAliveD. Since Amazon EC2 currently does not support Multicast protocol we need to configure KeepAliveD with Unicast TCP in this scenario.  For more details refer http://www.keepalived.org/. Mean time manually we can bring the failed HAProxy Process up and make this as the new standby.

Script Name: “/etc/keepalived/keepalived.conf”

vrrp_script chk_haproxy {           # Requires keepalived-1.1.13 script "killall -0 haproxy"     # cheaper than pidof interval 20                      # check every 2 seconds weight 20                        # add 2 points of prio if OK } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 ipriority 101                    # 101 on master, 100 on backup vrrp_unicast_bind 10.215.31.4       #internal IP address of EC2 instance 01 vrrp_unicast_peer 10.85.110.252   #internal IP address of EC2 instance 02 notify_master "/etc/keepalived/vrrp.sh" track_script { chk_haproxy weight 20 } }

Script Name: /etc/keepalived/vrrp.sh

#vrrp.sh #!/bin/bash cd /opt/aws/apitools/ec2/bin #DisAssociate EIP from this instance. ./ec2-disassociate-address --aws-access-key XXXXXXX –aws-secret-key XXXXXXX  [EIP] #Mapping EIP to secondary server ./ec2-associate-address --aws-access-key XXXXXXX  –aws-secret-key XXXXXXX  [EIP] -i [ec2_instance_id_of_primary_or_secondary]

Failure @ HAProxy EC2 instance level: When the Active HAProxy EC2 instance itself fails; we can detect this using Heartbeat and switch the Elastic IP from Active -> Standby. "Heartbeat" tool connects two servers and checks the regular "pulse" or "heartbeat" between them. The standby server takes over the work of the “Active” as soon as it detects an alteration in the "heartbeat" of the former. We have observed it takes ~120+ seconds for the standby to takeover. During this time the particular HAProxy alone will be unreachable. Heartbeat has to be configured in both the Active and standby HAProxy EC2 instance. Since Amazon EC2 currently does not support Multicast protocol we need to configure Heartbeat with Unicast UDP in this scenario. Mean time manually we can bring the failed HAProxy EC2 instance up and make this as the new standby.

Script Name: /etc/ha.d/ha.cf

logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 30 initdead 120 udpport 694 ucast eth0 xx.xxx.xxx.xxa #Internal IP of EC2 instance 01 ucast eth0 xx.xxx.xxx.xxb #Internal IP of EC2 instance 02 auto_failback off

Script Name: Create a script named “elastic_ip” in both the servers.

#!/bin/bash I_ID="[ec2_instance_id" # different for each EC2 servers. ELASTIC_IP="X.X.X.X" case $1 in     start) ec2-associate-address --aws-access-key XXXXX --aws-secret-key XXXXX "$ELASTIC_IP" -i "$I_ID" > /dev/null        echo $0 started        ;;     stop) ec2-disassociate-address --aws-access-key XXXXX --aws-secret-key XXXXX "$ELASTIC_IP" > /dev/null     echo $0 stopped        ;;     status) ec2-describe-addresses --aws-access-key XXXXX --aws-secret-key XXXXX | grep "$ELASTIC_IP" | grep "$I_ID" > /dev/null     # grep will return true if this ip is mapped to this instance     [ $? -eq 0 ] && echo $0 OK || echo $0 FAIL     ;; esac

Why do we need this redundancy in the HAProxy layer?

Not all the times the DNS RR with LB Cookie Insertion alone is enough for ensuring availability;

Case 1: Imagine you have not automated the scalability @ Load Balancing Layer and one of your Load balancer is down. You do not want to be waked up in the middle of the night rather it is better to have a standby Load Balancer automatically replacing the failed one. Manually you can replace the faulty LB next day.

Case 2: You have a gaming site where long running TCP sockets are established from flash gaming clients to the LB layer. You have planned the capacity of Front end Load Balancers with concurrent connections/sec. Now couple of your Load balancers are down, the new connections will be established to other running LB, but overall your site will now start performing poorly and chances are new connections are exhausted after few hours of heavy traffic. It is better to automatically detect and replace the faulty LB EC2 instance with the standby.

Case 3: Some clients cache the IP address of the Load Balancer, Some of them have long running sticky sessions with web/app, Some hardware devices can take only IP address to push data into the Server infrastructure. Though it is suggested to resolve the IP using DNS, still in reality some use cases does not work the same way.

Pattern 3: Use ELB

Do not worry about all the above patterns, just go and configure Amazon Elastic Load Balancing (ELB). For most of the use cases ELB is more than sufficient.

Amazon Elastic Load Balancer can distribute incoming traffic across your Amazon EC2 instances in a single Availability Zone or multiple Availability Zones. Amazon Elastic Load Balancing automatically scales its request handling capacity in response to incoming application traffic. It can handle 20k+ concurrent requests/sec with ease. It enables you to achieve even greater fault tolerance in your applications, seamlessly providing the amount of load balancing capacity needed in response to incoming application traffic. Elastic Load Balancing detects unhealthy instances within a pool and automatically reroutes traffic to healthy instances until the unhealthy instances have been restored. Any faulty Load balancers in the ELB tier are automatically replaced.

Though for most of the common use cases ELB is more than sufficient in AWS. There are some unique cases which demand the use of Load balancers like HAProxy, Nginx and NetScaler in our architecture in the AWS infrastructure. I will be writing some articles in coming months comparing ELB and others in use case perspectives and context. 

Why do we need Elastic Load Balancing ?

Dissecting Amazon ELB: 18 things you should know

Monitoring ELB with Amazon CloudWatch

Configuring Amazon ELB using AWS Management Console

Amazon ELB Implementation Architectures

Amazon Elastic Load balancing benefits

Offloading SSL in Amazon ELB

Amazon ELB and Flash Traffic

Amazon ELB Supported Algorithms and Protocols

I would like to thank Ramprasad for co-authoring some portions of this article with me.