Billion Messages - Art of Architecting scalable ElastiCache Redis tier

Whenever we are designing a highly scalable architectures on AWS running thousands of application servers and supporting millions of requests, usage of NoSQL solutions have become inevitable part. One such solution we often been using for years on AWS is Redis . We love Redis. 
AWS introduced ElastiCache Redis on 2013 and we started using the same since it eased the management and operational efforts.  In this article i am going to share my experience on designing large scale Redis tiers supporting billions of messages per day on AWS, step by step guide on how to deploy the same, what are the Implications you face at scale ? Best Practices to be adopted while designing sharded+replicated Redis Tiers etc.

Since we need to support billions of message requests per day and it was growing:

• the ElastiCache Redis tier was designed with Partitions( shards) to scale out as the customer grows
• the ElastiCache Redis tier was designed with Replica Slaves for HA and read scaling as the read volumes grow

When your application is growing at Rapid pace and lots of data are created every day, you cannot keep increasing (scaling up) the size of the ElastiCache Node. At one point you will hit the maximum memory capacity of your EC2 instance and you will be forced to partition.  Partitioning is the process of splitting your Key Value data into multiple ElastiCache Redis instances, so that every instance will only contain a subset of your Key Value pair. It allows for much larger ElastiCache Redis data stores, using the sum of the memory of many ElastiCache Redis Nodes. It also allows to scale the computational power to multiple cores and multiple EC2, and the network bandwidth to multiple EC2 network adapters. There are two widely used partition/shard implementation techniques that are available for ElastiCache Redis Tier :
Technique 1) Client side partitioning means that the Redis clients directly select the right ElastiCache Redis node where to write or read a given key. Many Redis clients implement client side partitioning, chose the right one wisely.
Technique 2) Proxy assisted partitioning means that your clients send requests to a proxy that is able to speak the Redis protocol, which in turn sends requests directly to the right ElastiCache Redis instance. The proxy will make sure to forward our request to the right Redis instance accordingly to the configured partitioning schema. Currently the most widely used Proxy assisted partitioning tool is Twemproxy , written by Manju Raj of twitter. Git hub link https://github.com/twitter/twemproxy . Twemproxy is a proxy developed at Twitter for the Memcached ASCII and the Redis protocol. Twemproxy supports automatic partitioning among multiple Redis instances and  currently it is the suggested way to handle partitioning with Redis.

In this article we are going to explore in detail about Proxy assisted partitioning technique for highly scalable and available Redis tier.

Welcome to Twemproxy
Twemproxy( nutcracker) is a fast single-threaded proxy supporting the Memcached ASCII protocol and more recently the Redis protocol.

Installing Twemproxy:
Download the Twemproxy package.
wget http://twemproxy.googlecode.com/files/nutcracker-0.3.0.tar.gz
tar -xf nutcracker-0.3.0.tar.gz
cd nutcracker-0.3.0
./configure
make
make install

Configuration:
Twemproxy (Nutcracker) can be configured through a YAML file specified by the -c or --conf-file command-line argument on process start. The configuration file is used to specify the server pools and the servers within each pool that nutcracker manages. The configuration files parses and understands the following keys:

• listen: The listening address and port (name:port or ip:port) for this server pool.
• hash: The name of the hash function.
• hash_tag: A two character string that specifies the part of the key used for hashing. Eg "{}" or "$$". Hash tag enable mapping different keys to the same server as long as the part of the key within the tag is the same.
• distribution: The key distribution mode.
• timeout: The timeout value in msec that we wait for to establish a connection to the server or receive a response from a server. By default, we wait indefinitely.
• backlog: The TCP backlog argument. Defaults to 512.
• preconnect: A boolean value that controls if nutcracker should preconnect to all the servers in this pool on process start. Defaults to false.
• redis: A boolean value that controls if a server pool speaks redis or memcached protocol. Defaults to false.
• server_connections: The maximum number of connections that can be opened to each server. By default, we open at most 1 server connection.
• auto_eject_hosts: A boolean value that controls if server should be ejected temporarily when it fails consecutively server_failure_limit times. See liveness recommendations for information. Defaults to false.
• server_retry_timeout: The timeout value in msec to wait for before retrying on a temporarily ejected server, when auto_eject_host is set to true. Defaults to 30000 msec.
• server_failure_limit: The number of consecutive failures on a server that would lead to it being temporarily ejected when auto_eject_host is set to true. Defaults to 2.
• servers: A list of server address, port and weight (name:port:weight or ip:port:weight) for this server pool.

For More details Refer: https://github.com/twitter/twemproxy

Running and Accessing Twemproxy 
To start the proxy just use the command “nutcracker” with the configuration file path specified or in its default path(conf/nutcracker.yml) .
Based on the configuration the twemproxy will be running and listening. Configure your application to point to the port and address instead of the Redis cluster.

Twemproxy Deployment models:

We usually deploy Twemproxy one one of the following models in AWS :
Model 1: Twemproxy as a separate Proxy Tier: In this model Twemproxies are deployed in separate EC2 instances, The application tier is configured to point to Twemproxies . The Twemproxy tier in turn maintains the mappings to the ElastiCache redis nodes. It is better to use instances with very good IO bandwidth for twemproxy tier in AWS. In case you feel the instance CPU is underutilized, you can launch multiple Twemproxy instances inside the same single EC2 instance as well.

Though the above model looks clean and efficient there are optimizations that can be applied to this architecture :
What happens when the twemproxy01 fails, how will the Application server instances know about it ?
Why should i pay additional for twemproxy EC2 instances, Can it be minimized ?

Model 2 : Twemproxy bundled with application tier EC2's: In this model twemproxies are bundled in the same box of the application server EC2 itself. Since two twemproxies are not aware of each others existence, it is easy to architect this model even in App->Auto Scaling mode. Every application server talks to the local twemproxy deployed in the same box this saves cost and avoids managing additional tier complexity as well.

Reference ElastiCache Redis + Twemproxy  deployment:
(This is a Reference deployment, the same can be scaled out to hundreds depending upon the need. It is a Redis Partitioned + replicated setup )
1. Two ElastiCache Redis nodes in AWS (twem01 and twem02)
2. Replication group for each ElastiCache redis nodes (twem01-rg and twem02-rg with one Read Replica each)
3. Two twemproxy servers running in separate EC2. (twemproxy01 and twemproxy02)
Once the above setup is done please note down the endpoints. We will be using the Replication group endpoint as the ElastiCache Redis endpoint for the twemproxy.

ElastiCache Redis Endpoints:
twem01-twem01.qcdze2.0001.usw2.cache.amazonaws.com:6379
twem02-twem02.qcdze2.0001.usw2.cache.amazonaws.com:6379
ElastiCache Redis Replication endpoints:
twem01-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379
twem02-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379

To test the Twemproxy we pumped following keys:
Pump KV data through the Twemproxy01 (1-2000 keys)
Pump KV data through the Twemproxy02(2001-4000 keys).

Configuration:
beta:
  listen: 127.0.0.1:22122
  hash: fnv1a_64
  hash_tag: "{}"
  distribution: ketama   #Consistent Hashing
  auto_eject_hosts: false
  timeout: 5000
  redis: true
  servers:
- twem01-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379:1 server1
- twem02-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379:1 server2

Test 1: Testing Key accessibility . Testing "GET" operation across both the Twemproxy Instances for few sample keys. 

Fetch 4 Keys spread across 4000 KV data from Twemproxy01  EC2 instance:
[root@twemproxy01 redish]# src/redis-cli -h 127.0.0.1 -p 22122
redis 127.0.0.1:22122> get 1000
"1000-data"
redis 127.0.0.1:22122> get 2000
"2000-data"
redis 127.0.0.1:22122> get 3000
"3000-data"
redis 127.0.0.1:22122> get 4000
"4000-data"
Fetch 4 Keys spread across 4000 KV data from Twemproxy02  EC2 instance:
[root@twemproxy02 redish]# src/redis-cli -h 127.0.0.1 -p 22122
redis 127.0.0.1:22122> get 1000
"1000-data"
redis 127.0.0.1:22122> get 2000
"2000-data"
redis 127.0.0.1:22122> get 3000
"3000-data"
redis 127.0.0.1:22122> get 4000
"4000-data"

From the above test it is evident that all 4000 KV data inserted using both Twemproxies are accessible from both Twemproxies( testing the sample) even though they are not aware among themselves. This is because of the same hashing and Key mapping translation done at Twemproxy level.

Test 2: Testing the ElastiCache Redis Availability and Fail over mechanism:

We are going to promote the twem01-rg replication group read replica to be the Primary Redis Node. After promotion we are going to test:

1. Whether the Twemproxy is able to recognize the newly promoted master
2. Whether the sample KV data is safely replicated and still accessible , to ensure failover is successful.

To promote ElastiCache Redis slave just click the promote Action and confirm or automate using API. During the promotion of Read Replica to master we observed that the transition happens very quickly and there is no timeout but the response time for the query is about 4-5 secs for about 3-4 minutes during the switch over. In the Twemproxy configuration we can set the timeout configuration, this value needs to be set accordingly so that during switch over there will be no connection refused. For the sample test we have set it as 5000

Repeat Test 1:

[root@twemproxy01 redish]# src/redis-cli -h 127.0.0.1 -p 22122
redis 127.0.0.1:22122> get 1000
"1000-data"
redis 127.0.0.1:22122> get 2000
"2000-data"
redis 127.0.0.1:22122> get 3000
"3000-data"
redis 127.0.0.1:22122> get 4000
"4000-data"
Fetch 4 Keys spread across 4000 KV data from Twemproxy02  EC2 instance:
[root@twemproxy02 redish]# src/redis-cli -h 127.0.0.1 -p 22122
redis 127.0.0.1:22122> get 1000
"1000-data"
redis 127.0.0.1:22122> get 2000
"2000-data"
redis 127.0.0.1:22122> get 3000
"3000-data"
redis 127.0.0.1:22122> get 4000
"4000-data"

From the above test it is evident that all 4000 KV data are replicated properly between master and slaves nodes and the transition between slave to master happened successfully with all the data.

Reporting

Nutcracker exposes stats at the granularity of server pool and servers per pool through the stats monitoring port. The stats are essentially JSON formatted key-value pairs, with the keys corresponding to counter names. By default stats are exposed on port 22222 and aggregated every 30 seconds.

Some best practices while designing highly scalable+available ElastiCache Redis Tier :

Practice 1 : Reduce the Number of Connections and pipeline messages:
Whenever the application instance gets a request to get/put value to the ElastiCache redis node, the client makes a connection to the Redis Tier. Imagine it is a heavy traffic site, then thousands of requests hitting translates to thousands of connections from the application instance to Redis Tier. Now when you add Auto- scaling to your application tier and you have few hundred servers scaled out , then imagine the connection complexity and overhead this architecture brings to the ElastiCache Redis Tier.

Best practice is minimize the number of connections made from your application instance to your ElastiCache redis node. Use Twemproxy in bundled mode with Application EC2 instance, this keeps the process in close proximity and reduces the connection overhead.  Secondly, Twemproxy internally uses minimal connections to ElastiCache Redis Instance by proxying multiple client connections onto one or few server connections.
Redis also supports pipelines, where multiple requests can be pipelined and sent on a single connection. In a simple test using large Application & ElastiCache node we were able to process 125K message/sec in pipeline mode, now imagine what you could achieve on bigger instance types on AWS. The connection minimization architectural setup of twemproxy makes it ideal for pipelining requests and responses and hence saving on the round trip time.  For example, if twemproxy is proxying three client connections onto a single server and we get requests - 'get key\r\n', 'set key 0 0 3\r\nval\r\n' and 'delete key\r\n' on these three connections respectively, twemproxy would try to batch these requests and send them as a single message onto the server connection.

Note : It is important to note that "read my last write" constraint doesn't necessarily hold true when twemproxy is configured with server_connections: > 1. Let us consider a scenario where twemproxy is configured with server_connections: 2. If a client makes pipelined requests with the first request in pipeline being set foo 0 0 3\r\nbar\r\n (write) and the second request being get foo\r\n (read), the expectation is that the read of key foo would return the value bar. However, with configuration of two server connections it is possible that write and read request are sent on different server connections which would mean that their completion could race with one another. In summary, if the client expects "read my last write" constraint, you either configure twemproxy to use server_connections:1 or use clients that only make synchronous requests to twemproxy.

Practice 2:  Configure Auto Ejection and Hashing combination properly
Design for failure is the mantra of cloud architecture. Failures are commons when things are distributed on scale. Though partitioning when using ElastiCache Redis as a data store or cache is conceptually the same on broad lines, there is a huge difference operationally on large scale systems. When you are using ElastiCache Redis as a data store you need to be sure that a given key always maps to the same instance, Whereas if you are using  ElastiCache Redis as cache if a given node is not available, then you can always start afresh using a different node in the hash ring with consistent hashing implementations.
To be resilient against failures, it is recommended that you configure Auto eject hosts false when you treat redis as a Data Store and true in when you treat redis as a cache.
resilient_pool:
  auto_eject_hosts: true
  server_retry_timeout: 30000
  server_failure_limit: 3
Enabling auto_eject_hosts: This property ensures that a dead ElastiCache redis Node can be ejected out of the hash ring after server_failure_limit: consecutive failures have been encountered on that node. A non-zero server_retry_timeout: ensures that we don't incorrectly mark a node as dead forever especially when the failures were really transient. The combination of server_retry_timeout: and server_failure_limit: controls the tradeoff between resiliency to permanent and transient failures.
Note that an ejected node will not be included in the hash ring for any requests until the retry timeout passes. This will lead to data partitioning as keys originally on the ejected node will now be written to another node still in the pool. If ElastiCache Redis is used as a cache (in memory) then in event of a Redis Node going down, the cache data will be lost. This cache miss can cascade performance problems to other tiers and altogether bring down your system on the cloud. To minimize KV cache miss,  you can design your hash ring with Ketama hashing on the Redis Proxy. This will minimize the Cache miss in event of cache node failure, also it decreases the overall re-balancing needed in your Redis tier.  In addition to helping hand on availability problems, Redis Proxy+Ketama can also help your Redis farm to Scale out and Scale down easily with minimal cache miss. To know more about Ketama on ElastiCache refer http://harish11g.blogspot.com/2013/01/amazon-elasticache-memcached-internals_8.html  . The below diagram illustrates a ElastiCache Redis Cache Farm with Consistent Hash Ring.

In short to minimize the cache miss when using auto eject with true it is recommended to use "Ketama Hashing ( Consistent Hashing Algorithm)" on your Twemproxy configuration. 

ElastiCache Redis as a Data Store:
What if the data stored in your Cache is important and needs to persisted across node failures and launch ? What if the date stored in your Cache cannot be lost and it needs to be replicated and promoted during failures?
Welcome to ElastiCache Redis as Data store. ElastiCache Redis offers features to persist the in memory cache data to disk and also replicate it to a slave for high availability. If ElastiCache Redis is used as a store (persistent), you need to keep the map between keys and nodes fixed, and a fixed number of nodes. Since the data stored is important when you treat ElastiCache Redis as a data store, in event one Redis node goes down, you should have immediate standby up and running in minutes.  You can architect ElastiCache Redis master with one or more replication Slave launched on different AZ from Master for High Availability in AWS. In event master node failure or master AZ failure, the slave Redis node can be promoted in minutes to act as master. This whole High availability design keeps the number of nodes on the hash ring stable and simple, Otherwise, you will end up building a system to re balance the keys (which is not easy) between nodes whenever there is a addition or removal of nodes during outages. In addition to above the ElastiCache Redis supports Partial Resynchronization with Slaves - If the connection between a master node and a slave node is momentarily broken, the master now accumulates data that is destined for the slave in a backlog buffer. If the connection is restored before the buffer becomes full, a quick partial resync will be done instead of a potentially longer full resync. This really saves network bottleneck during momentary failures.
In large scale systems you will often find some partitions are heavily used than others , in event the usage is read heavy in nature you can add upto 5 Read replicas for the ElastiCache Redis Master partition. Since these replicas are used only for read they do not affect the Hash ring structure. But Twemproxy lacks the support for read scaling with Redis Replica's. So in event when you face this problem, you will have to Scale up the capacity(instance/node type) of the Master and Slave of that partition alone.

If you are using ElastiCache redis as a Data store in the TwemProxy it is recommended to keep "auto_eject_hosts" property false so that in event of redis node failure it is not ejected from the hash ring. The hash ring can be built with both ketama or modula hash algorithms , since in event of Primary node failure, the Slave is going to be promoted and ring structure is going to be always maintained. But if you feel there is immense possibility for the number of primary node partitions to grow, or major failures to occu, it is better to choose ketama hash ring itself from beginning. The below diagram illustrates the architecture.

Practice 3: Configure the Buffer properly:
All memory for incoming requests and outgoing responses is allocated in mbuf in Twemproxy. Mbuf enables zero copy for requests and responses flowing through the proxy. By default an mbuf is 16K bytes in size and this value can be tuned between 512 and 16M bytes using -m or --mbuf-size=N argument. Every connection has at least one mbuf allocated to it. This means that the number of concurrent connections twemproxy can support is dependent on the mbuf size. A small mbuf allows us to handle more connections, while a large mbuf allows us to read and write more data to and from kernel socket buffers. Large Scale web/mobile applications involving millions of hits might have small size request/response and lots of concurrent connections to handle in their backend. So at such scenarios, when Twemproxy is meant to handle a large number of concurrent client connections, you should set chunk size to a small value like 512 bytes to 1K bytes using the -m or --mbuf-size=N argument.

Practice 4: Configure proper Timeouts
It is always a good idea to configure Twemproxy timeout: for every server pool, rather than purely relying on client-side timeouts. Eg:

resilient_pool_with_timeout:
  auto_eject_hosts: true
  server_retry_timeout: 30000
  server_failure_limit: 3
  timeout: 400
Relying only on client-side timeouts has the adverse effect of the original request having timed out on the client to proxy connection, but still pending and outstanding on the proxy to server connection. This further gets exacerbated when client retries the original request.


Benefits of using Twemproxy for Redis Scaling

• Avoids re inventing the wheel. Thanks Manju Raj (twitter).
• reduce the number of connections to your cache server by acting as a proxy
• shard data automatically between multiple cache servers
• support consistent hashing with different strategies and hashing functions
• be configured to disable nodes on failure
• run in multiple instances, allowing client to connect to the first available proxy server
• Pipelining and batching of requests and hence saving of round-trips

Disadvantages of Partitioning Model:

Point 1) Operations involving multiple keys are usually not supported. For instance you can't perform the intersection between two sets if they are stored in keys that are mapped to different Redis instances (actually there are ways to do this, but not directly).Redis transactions involving multiple keys can not be used.
Point 2) The partitioning granularity is the key, so it is not possible to shard a dataset with a single huge key like a very big sorted set. Ideally in such cases you should Scale UP the particular Redis Master-Slave to larger EC2 instance or pro grammatically stitch up the sorted set.
Point 3)When partitioning is used, data handling is more complex, for instance you have to handle multiple RDB / AOF files, and to make a backup of your data you need to aggregate the persistence files/snapshots from multiple EC2 Redis slaves.
Point 4) Architecting a partitioned + replicated ElastiCache Redis tier not complex. What is more complex is ? supporting transparent rebalancing of data with the ability to add and remove nodes at runtime. Systems like client side partitioning and proxies don't support this feature. However a technique called Presharding helps in this regard with limitations. Presharding technique ->Since Redis is lightweight, you can start with a lot of EC2 instances since the beginning itself. For example if you start with 32 or 64 EC2 instances (micro or small Cache Node instance type)  as your node capacity , it will provide enough room to keep scaling up the capacity when your data storage needs increase. It is not a highly recommended technique. But still can be used in production if your growth pattern is very predictable.

Future of highly scalable + available Redis tiers -> Redis Cluster

Redis Cluster is the preferred way to get automatic sharding and high availability. It is currently not production ready. Once Redis Cluster / Client  is available on Amazon ElastiCache, it will be the de facto standard for Redis partitioning. It uses a mix between query routing and client side partitioning.

References:
http://redis.io/documentation
https://github.com/twitter/twemproxy

This article was co-authored with Senthil

Other Amazon ElastiCache Articles :
Caching architectures using Memcached & Amazon ElastiCache
Highly Available ElastiCache Redis replication cluster in AWS VPC
Deep dive into Amazon Elasticache
Distributed Cache On Steroids: Amazon ElastiCache
Launching Amazon ElastiCache in 3 Easy Steps
AWS Performance Tip 3: Use Amazon ElastiCache to improve application performance

Load Balancing in Amazon Web Services

Load balancing is one of the most important technique to be followed for architecting highly scalable and available applications on AWS cloud. Keeping this in mind, I have compiled my experience and articles i have written on this subject as "Load balancing on AWS" series.  

Dissecting Amazon Elastic Load Balancing : Amazon ELB is dissected into 18+ points and analysed based on the production implementation experience. This article is a must read for all Amazon ELB users who wants to understand in detail what ELB can do and what are some of the road blocks you can face sometimes when using Amazon ELB. Click here for the article.  

Amazon ELB Multi region migration checklist: Amazon Elastic Load Balancing has a Amazon EC2 Regional scope. It needs to be migrated to alternate Amazon EC2 region in event of DR or during new migration setup . I am sharing my experience in this post as few checklists/areas that needs to be taken care during this Amazon ELB migration to alternate EC2 region. Refer Article

Amazon ELB Implementation Architectures : Amazon ELB can be implemented in variety of architectures in your AWS cloud. Some of them are real bad cases and some of them follow best practices. If you want to understand the common implementation architectures and its pro's/con's refer this article  

Comparison Analysis between Amazon ELB and HAProxy:  HAProxy is a popular OSS software load balancer widely used in Amazon Cloud Infrastructure. This article provides a detailed comparison between the two and analyzes where both of them stand. This article is featured in the HAProxy web site as well. Click here for the article

Architecting High Availability @ HAProxy Load Balancing Tier: Not all the times Amazon ELB is used as the choice of load balancer in Amazon Web Services. But deploying an OSS load balancer in AWS has got is own set of considerations and high availability is one of them. In this post i explore how to design High Availability @ HAProxy load balancing layer. Refer Article

Architecting Highly Available Web App Layer using HAProxy Load Balancing : In this post we explore four options of Load balancing Web/App Tier using HAProxy in AWS. Refer Article

Configuring Citrix NetScaler Load Balancing on AWS: Citrix NetScaler is a popular Load balancer in the Enterprise world. Hardware and virtualized NetScaler has been serving many popular enterprise online assets for years. How to configure Citrix Netscaler based Load Balancing in AWS is explained in this post. Refer Article

Load Balancing Amazon RDS Read Replica's using HAProxy: In this post, let us explore how HAProxy can be used to solve load balancing between RDS Read Replica's in Amazon Cloud.Refer Article

Choosing right HAProxy- Amazon EC2 Instance Types and AMI Types for load balancing: Before choosing optimal Amazon EC2 instance Type for HAProxy Load balancing layer in AWS we need to minimum understand some important factors involved. Refer Article 

Configuring Amazon ELB and understanding the parameters in detail: Amazon provides a detailed documentation on how to configure ELB. In this article i have detailed the configuration steps with detailed understanding on the parameters and its implications. Must read of Amazon ELB newbies. Click here for the article 

Web Session Synchronization patterns in AWS: Architectures for synchronizing sessions of load balanced Web/App EC2 is detailed in this post. Refer Article

Geo Distributed Load Balancing using Route53 and Amazon ELB: Some customers would have Geo Distributed their architecture across multiple Amazon EC2 regions. In this article we explore why do we need Geo Distributed architecture, Cost of Latency and how to achieve it using Route 53 + Amazon ELB. Click here for the article

Deeper Health Checks and Problems in Load Balancing in AWS: Health Checks are one of the essential mechanisms that helps you to keep N-Tiered system highly available.This sounds simple and straight forward right, but some of the customers i have consulted follow a much deeper Health check diagnostic mechanism and it might have problems when migrated to AWS cloud. Let us explore this case in detail in this post. Refer Article

Configuring Amazon ELB With SSL offloading: How to configure SSL with Amazon ELB. Click here for the article What are the benefits of offloading SSL in Amazon ELB. Here

Monitoring Amazon ELB using Amazon CloudWatch and understanding the result and metrics

Why do we need Amazon Elastic Load Balancing and What are its benefits ?

Deeper Health Checks and Problems in Load Balancing in AWS

Health Checks are one of the essential mechanisms that helps you to keep N-Tiered system highly available. Usually a simple script or program is deployed on the Web/App Server. The Health check component of Load Balancer is configured to frequently call this script in Web/App Server in a light weight protocol. Based on the response from the script/program the Load balancer decides the status of the Web/App Servers and accordingly direct the requests to healthy Web/App servers. This is a usual mechanism that is followed in all popular load balancers like Amazon ELB, Netscaler, HAProxy and NGinx in AWS cloud. This sounds simple and straight forward right, but some of the customers i have consulted follow a much deeper Health check diagnostic mechanism and it might have problems when migrated to AWS cloud. 
Let us explore this case in detail :

What is the architecture ?
A simple multi-tiered architecture with : A  load balancer deployed at the front. The Web/App Server has the health script/program. The database is MySQL deployed Master+ Slave mode.  

What is deeper Health Check ?
The script/program deployed in the Web/App Server is little intelligent; when it is called by load balancer it performs simple operations and checks the status of the Database. So when you get a response back from the health check script / program you are verifying whether the health of DB and Web/App server is sound at the load balancer tier.

What is the problem scenario ?
Imagine when migrating this infrastructure to AWS you have adopted the standard architecture pattern consisting of :

• Amazon ELB is used as the Load balancer
• Web/App Server in auto scaling mode
• MySQL moved to Amazon RDS+Multi-AZ with RR

Now let us explore this problem in detail : 

• Imagine the any of the following condition in your production, network between database and Web/App is down intermittently for few minutes or RDS MySQL is elevating the Hot Standby as new Master. In such scenarios, the health check response actually timeouts at Database level, whereas the Load Balancer will mark the even the healthy App Servers as unhealthy because of the deeper health checks. This is not good especially for Amazon Auto Scaled scenario's where Amazon ELB marks Web/App EC2 as unhealthy because of deeper health check and Amazon Auto Scaling keeps restarting the Web/App EC2 auto automatically to maintain minimum healthy farm. This unwanted effect can cascade the overall availability and surely not good for the production in AWS. So in short Deeper Health checks are not surely recommended for complex N-Tier systems that follows Auto scaling/healing and Service oriented architecture patterns in AWS. 
• Usually the purpose of health check is to check the status of next tier or service consumed by a particular tier.  Deeper health checks is heavy weight and it usually takes much more time to respond because majority of your tiers are exercised in this process. If we set this frequency too aggressive, then health checks itself will eat lots of your CPU. So the frequency of the health checks and the response time out have to be set considerably large. Also during heavy traffic scenario, such heavy weight calls can be queued and you might not get faster response in deeper health checks. 
• Deeper health checks are usually suitable for simple and fixed infrastructures. When your infrastructure is non elastic , the decisions are taken manually by the ops team after analyzing the particular failing tier. For Elastic Auto scaled workloads in AWS it is better to isolate the health checks of load balancing tier separate from Deeper Health checks that can be used for assessing the availability of the infrastructure.

Load Balancing Amazon RDS Read Replica's using HAProxy

When you are architecting a read intensive online application in AWS cloud you can employ techniques like CDN, Caching etc to improve the overall concurrency and performance of the application. One of the age old techniques applied is also scaling out the Database Read Slaves. To solve this need, Amazon RDS MySQL has a concept of Master and Read Replica Slaves. Depending upon the read intensity and concurrency needed, you can scale out and add more read replica slaves to the Master RDS MySQL. Ideally 1 to 5 Read replica slaves can be placed with RDS MySQL master for performance. If more than 5 Read Replica Slaves are required, it surely sounds like a bad design because of the load it puts on the master, replication lag and overall manageability of this tier itself. I would suggest you need to functionally partition your database or use some other high performance/scale out data stores like caching, MongoDB, DynamoDB, Redis etc in your architecture to over come this. 
Now let us define a common architecture deployment pattern for read intensive site:

• Entire setup is inside Amazon VPC
• Your Web App is deployed in Amazon Auto Scaling Mode.
• You have 2 or more RDS Read replica slaves with your RDS Master. This is good, now the question is how do you load balance requests between your Read replica's, What happens when you elastically scale out new RDS Read replica's ? 

There are multiple architecture techniques that can be followed to solve this problem from embedding Load balancing plugins in PHP, to introducing HAProxy in between etc.
In this post, let us explore how HAProxy can be used to solve this problem in Amazon Cloud.

Architecture 1:  HAProxy as a Separate Tier

• Web/App EC2 instances are deployed in the public subnet of Amazon VPC in Auto Scaling mode.
• RDS MySQL Master and 2 Read replica's are deployed in Multiple Subnet - Multi - AZ mode.
• Programmatic changes, plugins or some ideal mechanism is engineered in the Web/App to separate the Writes and Reads DB.All writes go to Master and reads goes to RDS Read Replica's.
• Web/App EC2 instances are pointed to HAProxy EC2 address. HAProxy Load balancer is provisioned in a separate tier to load balance internal requests from the Auto Scaled Web/App EC2 instances to the RDS Read Replica EC2's. 
• Two or more HAProxies are deployed to avoid Single Point of Failure in this tier. 

Below diagram illustrates this architecture technique:

  
Now let us analyse this architecture technique:

• This is a widely used technique in AWS cloud environment for such problems. You can use this RDS MySQL or MySQL on EC2 as well.
• New RDS Read replica's can be added and removed elastically depending upon the traffic without modifying the app configuration files. HAProxy configuration entries can be hot deployed
• Minimum of 2 HAProxy EC2 instances are needed to avoid Single Point of failure in this tier
• HAProxy can be deployed in Multiple-AZ and Multiple Subnet architecture for better HA 
• It is recommended to start with m1.large for HAProxy EC2 instances and scale up instance type depending upon traffic/concurrency. Note: m1.small/medium etc have moderate IO bandwidth and may degrade performance between Read replica's and App Tier.
• Frequent Scale up of HAProxy EC2 to higher instance type might be needed in case hundreds of Web/App EC2 's are auto scaled every day
• Logic has to be built in App tier to redirect traffic to secondary HAProxy in event of primary HAProxy failure
• In case both HAProxies are used actively, then logic has to be built in App tier to use both them 
• Additional price to be paid for 2 or more m1.large HAProxy EC2 instances
• Additional cost of monitoring and managing this HAProxy Tier

Architecture 2: HAProxy is embedded 

• Web/App EC2 instances are deployed in the public subnet of Amazon VPC in Auto Scaling mode.
• RDS MySQL Master and 2 Read replica's are deployed in Multiple Subnet - Multi - AZ mode. 
• HAProxy is installed/bundled with every Auto Scaled Web/App EC2.
• Every Web/App EC2 instance is pointed to the local HAProxy itself.  HAProxy will load balance requests from that Auto Scaled Web/App EC2 instances to any of the RDS Read Replica EC2's. 

   

Now let us analyse this architecture technique:

• This is not widely used as the previous one by many users, probably because not many would have thought/implemented on these lines. But i found this simple and manageable in larger AWS production deployments.
• New RDS Read replica's can be added and removed elastically depending upon the traffic without modifying the app configuration files. Read Replica endpoints can be propagated to HAProxy using Chef and then hot deployed in HAProxy
• No single point of failure, if your web/app EC2 instance fails your HAProxy also fails. HAProxy rarely fails individually and is very stable.
• No additional HAProxy EC2 instances are needed - hence lower cost and ease of manageability.
• Found this embedded technique really useful in larger, auto scaled AWS production deployments
• This technique gives more performance when you use larger Web/App EC2 instances like m1.xlarge/ C1.Xlarge etc. HAProxy uses very less CPU and memory and utilizes large IO band with coming with larger instance types. When you have designed your Web/App EC2 with smaller ec2 like medium/small, this is not suggested because of the resource contention
• Lesser response latency because lesser NW trip
• No Scale up of HAProxy required. HAProxy is very light weight and super stable process. It can easily scale the requests with your applications need in the embedded model.
• No complex logic has to be built in web/App tier. they simply contact the HAproxy and it does the rest.

Sample Configuration Steps for Architecture Technique -2:

Setup Details:

• Web/App EC2 (Amazon Linux) : 2 . (Can be running in amazon auto scaling as well in production) 
• RDS MySQL Master DB Instance :1
• RDS MySQL Read Replicas: 2 - 5. (Use Larger EC2 instance types for production purpose)
• HAProxy will be running on each Web/App EC2
• Versions,Instance type and configurations used below are strictly for illustrative purposes only. Note: For production use some modifications might be needed. 

Step 1: Creating Read Replicas:
Create two Read Replicas from the RDS MySQL Master DB instance. To create MySQL Read replica navigate to the dashboard of Amazon RDS, select the Amazon RDS MySQL Master and use the option of “Create Read Replica”. On successful creation, you will get endpoint for each of the Read Replica slaves. The below screenshots illustrates the same.

Step 2: Installing HAProxy on Web/App EC2
Installing HAProxy can be done from the source or from the repository. We have installed it from the repository. To install HAProxy from the repository and start it use the following commands,

#yum install haproxy. 
#service haproxy start.

Step 3: HAProxy Configuration on Single Web/App EC2
The configuration file for HAProxy will be available in the following location
 /etc/haproxy/haproxy.cfg
In the configuration file there are many sections like global, default, listen. In each section you may need to specify some parameters and values.
In listen section ,specify port for the RDS MySQL(3306) and user for the mysql-check. "mysql-check" is used to check the health status of the back end read replica nodes. In order for health check to work create an user on RDS MySQL master with no password and use it for the mysql-check user option.This detail will be automatically propogated to read replica's as well. The Load balance algorithm used here is Round Robin. 

Sample configuration file:
##/etc/haproxy/haproxy.cfg##

global

log         127.0.0.1 local2 debug
chroot      /var/lib/haproxy
pidfile      /var/run/haproxy.pid
maxconn     4000
daemon

defaults
mode        tcp
log         global
option tcplog
timeout connect 10000 # default 10 second time out if a backend is not found
timeout client 300000
timeout server 300000
maxconn     20000

# For Admin GUI
listen stats
bind :8080
mode http
stats enable
stats uri /stats

listen mysql *:3306
mode tcp
balance roundrobin
option mysql-check user check
option log-health-checks
server db01 sample-r1.XXXX.amzonaws.com:3306 check port 3306 inter 1000
server db02 sample-r2.XXXX.amazonaws.com:3306 check port 3306 inter 1000

Use the following to create user on master for health check.

use mysql;

create user check;

insert into user (Host,User) values ('<IP/RANGE_OF_HAPROXIES>','check');

FLUSH PRIVILEGES;

flush hosts;

For Production use in Amazon Web Services the HAProxy configuration file and setup can be propagated using Chef.

The Web/App process must be configured to use HAProxy for the MySQL read connections.Once the setup is running, In the admin page of HAProxy you can see the distribution of sessions equally in round robin fashion.
Admin URL Page:

AWS Cost Saving Tip 5: How Amazon Auto Scaling can save costs

One of the biggest technical challenges of running an online business is how well they are able to handle the scalability requirements.  The Load traffic pattern keeps varying for online businesses and accordingly they will have to scale and maintain the acceptable performance levels.  Since the Traffic patterns are fluctuating in online business, they either tend to under provision and loose customers (or) over provision and waste hardware + costs. This problem is well illustrated in the below diagrams.

Business usually makes detailed capacity planning and large upfront investment in their hardware and software. This HW/SW’s are usually provisioned with fixed capacity.

Many times because of variations in capacity planning and usage predictions, you can observe that HW/SW capacity is underutilized. This is because in reality the actual demand may not be uniform with your predictions. The systems are utilized well during peak period but lie idle most of the times when the peak period is over.  This leakage is illustrated in the below diagram in Grey shades.

On the other hand, sometimes online businesses totally get their predictions wrong and under provision their HW/SW. This is a usual occurrence during holiday seasons and campaigns where one cannot predict the user traffic accurately. Though the business provision 4X more capacity during these period as buffer, still there are lots of chances that they get traffic more than expected. In case the traffic exceeds, users can experience degraded performance or altogether cannot access the site itself sometimes. Companies can lose customers and potential business opportunities because of this mismatch. This scenario is illustrated as grey shade in the below diagram

If businesses can closely align their load requirements and capacities then above scenarios can be countered efficiently. To solve this, we need an elastic and auto scalable infrastructure, which can be automated to expand and collapse depending upon the load traffic. Let us explore in AWS Cloud computing context how this problem can be solved.

Auto scaling service was introduced by Amazon Web services especially to balance this problem by increasing/decreasing capacities automatically depending upon the traffic.

Let us explore few Load Volatility patterns and how Amazon Auto Scaling can be used to reduce leakages and save cost.

Scenario 1: Daily Spikes-Valley

Daily Spikes and Valley pattern can be best illustrated by the below diagram.

This pattern is usually observed by ecommerce companies which have peak usage for 12 hours between 8:00 am to 8:00 pm in a day and rest of the day the capacities are usually under-utilized. Imagine you are running 20 X m1.large for you web/app tier and they are fully utilized during peak hours.  During the non-peak period the load decreases gradually and ~25 % utilization overall is observed in the nights.  Since only 25% utilization is observed in nights if you can reduce your capacity to 5 EC2 web/app instances in an automated way it will save infra + labor costs.  Let us calculate the savings one can achieve by introducing Amazon Auto Scaling into this scenario:

Formula:  Number of EC2 X current cost of m1.large (0.24 USD) X Hours

Without Cost Optimization
20 X m1.Large X 744	~3572 USD per month

Cost Optimization with Amazon Auto Scaling
20 X m1.large X 372 (hrs)	~1785 USD per month
15 X m1.large X 124	~446
10 X m1.Large X 124	~298
5 X m1.large X 124	~149
	~2678 USD per month

Savings: Using Amazon Auto Scaling you can save ~25 % leakage a month in this scenario

Image source: AWS

Scenario 2: Weekly Fluctuation

Weekly Spikes and Valley pattern can be best illustrated by the below diagram.

This pattern is usually observed by online ticketing companies which have peak usage for 4 days week and normal -> under-utilization other days of the week. Since the load is fluctuating in a week, these companies cannot afford to lease capacity weekly. They usually plan their capacity based on the peak load traffic and provision their infrastructure. Since the entire infrastructure is running all days in a week, they will be wasting resources during the normal days.

These dynamic workloads are very good use case for Amazon Web Services and Amazon Auto Scaling. Let us explore this scenario in detail below. Imagine the ticketing company needs 20 m1.large ec2 instances to handle their peak periods. Instead of running 20 X m1.large EC2 instances satisfying your peak capacities all the time, you can run 5 EC2 instances for the normal days and Schedule an auto scale out to 20 EC2 instances before the peak days of the week. During the normal or peak days of the week, if the load traffic exceeds your capacity planning you can still automatically add more EC2 instances into your fleet.  Since 3 of the 7 days your instances are under-utilized (+) 8-12 hours under-utilization (late nights) during the peak days as well, using amazon auto scaling and dynamically expanding/collapsing the web-app tier will save lots of cost to the company. Let us calculate the savings one can achieve by introducing Amazon Auto Scaling into this scenario:

Without Cost Optimization
20 X m1.Large X 168	~806 USD per week

Cost Optimization with Amazon Auto Scaling
20 X m1.large X 48 (hrs)	~230 USD per week
15 X m1.large X 24	~87
5 X m1.large X 96	~115
	~432 USD per week

Savings: From the above calculation you can observe that using Amazon Auto Scaling ~25 % leakage can be saved in a week.

Scenario 3: Seasonal Spike scenario

Seasonal Spikes pattern can be best illustrated by the below diagram.

This pattern is usually observed by online travel/ecommerce companies which have peak holiday period for 3-5 months with heavy utilization and rest of the year with normal/ under-utilized capacities. Companies traditionally used to buy or lease capacities well before the season and spend months provisioning them before the holiday season. Whereas Amazon Cloud world, though you can launch instances on demand, still some companies have this traditional mind set and run their infrastructure with peak capacity throughout the year. Reasons: sometimes it is because of the bad application/infra architecture and at times it because of IT team’s inadequate knowledge in AWS infra. This is a complete waste of resources and will eventually lead to leakage in cost.

Imagine for serving your peak traffic during holiday season you need 20 m1.large web/app instances, instead of having them running for whole year, you can automatically scale them down post the season gradually in Amazon infrastructure.

Using Amazon Auto Scaling you can schedule gradual increase of EC2 instance capacity 2 months before the holiday season and gradually decrease capacity 1-2 months post the peak season.  During the peak season if there is unpredictable increase in traffic auto scaling can still increase the EC2 fleet dynamically to handle the dynamic load. This approach avoids both cost leakage and lost opportunity cost for business. Let us calculate the savings one can achieve by introducing Amazon Auto Scaling into this scenario:

Formula:  Number of EC2 X current cost of m1.large (0.24 USD) X Hours X months

Without Cost Optimization
20 EC2 X m1.Large X 744 hrs X 12 months	~42,854  USD per year

Cost Optimization with Amazon Auto Scaling
20 X m1.large X 744 X 4	~14,284 USD per year
10 X m1.large X 744 X 2	~3571
5 X m1.large X 124 X 7	~6249
	~24,105 USD per year

Savings: Using Amazon Auto Scaling you can save ~43 % leakage over a year in this scenario

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage