Exploring Amazon RDS MySQL Second Tier Read Replica

AWS recently introduced Second Tier Replica for RDS MySQL.You can use this feature to shift the load from primary master DB to the replica in first tier and prepare your application for handling extremely high level of read traffic.  In this post we are going to explore this feature in following aspects :

• Steps to configure Multi-Tiered Amazon RDS MySQL Read replicas
• Second Tier Read Replica Deployment architectures
• Important Points to note

Amazon RDS MySQL version 5.6.12 on AWS WEST region was used. 

Steps to configure Multi- Tiered Amazon RDS MySQL Read Replicas

Configuration steps for the following architecture is given in this post :

Step 1: Creating Read Replica from Master and Place at Tier 1:

To create RDS MySQL Read replica navigate to the dashboard of Amazon RDS, select the Amazon RDS MySQL Master(named "sampledb") and use the option of “Create Read Replica”. 

Name the newly created Amazon RDS Read replica as "sampledb-level1" and place it in Tier 1. The Tier 1 Amazon RDS MySQL read Replica can be created in same AZ of Master or in a different AZ for High Availability. When the Tier replica is placed in Different AZ , you should factor few extra milliseconds of latency during replication.

Explore the status from Master DB: Post successful creation of the Tier 1 Read Replica, You can see the Read Replica Id's from exploring the details of the Master DB. Illustrated in Below Screen shot :

Explore the status from Tier 1 Read Replica : When you explore the Tier 1 Read Replica details, you will find it is pointing to the to Master DB. Illustrated in Below Screen Shot :

Step 2: Creating Second Tier Read Replica from Tier 1 Read replica :
To create Second Tier Read replica navigate to the dashboard of Amazon RDS, select the Amazon RDS Tier 1 Read Replica (sampledb-leve1) as the source and use the option of “Create Read Replica”.

Name the newly created Amazon RDS Read replica as "sampledb-level2" and place it in Second Tier 2. The Second Tier -2 Amazon RDS MySQL read Replica can be created in same AZ of Tier 1/Master or in a different AZ for High Availability. 

Explore the status from Tier 2 Read Replica : When you explore the Tier 2 Read Replica details, you will find it is pointing to the to Tier 1 - sampledb-level1 as replication source. Illustrated in Below Screen Shot :

To Load Balance AWS RDS Read Replica's Refer this article : http://harish11g.blogspot.com/2013/08/Load-balancing-Amazon-RDS-MySQL-read-replica-slaves-using-HAProxy.html

Second Tier Read Replica Deployment Architectures

One of the main complexity behind Multi-Level replication is that, if Tier 1 Read Replica if not properly architected/placed, it can be a single point of failure. Imagine a case where you need 4 Read Replicas for your Master DB. You can take following approaches as illustrated below while designing your infrastructure for this requirement.

In Architecture-1, Tier-1 Read replica is a Single Point of Failure. Instead if you split Tier-1 itself into two separate fleets it offers better availability than architecture-1. Since both Tier-1 RR put replication load on Master DB , you can by pass this using Hot Standby instance. Benefits of this approach is explained below in the best practice architecture. 

Best Practice High Availability Architecture for Second Tier Read Replica:

Condition 1 - Master DB Failure : Hot Standby becomes new master and Tier-1 Read Replicas points automatically to new source
Condition 2- Tier-1 Read Replica Failure : Entire Tier-1 and Associated Tier-2 has to be recreated. The Alternate Active Fleet of Tier-1+Tier-2 will serve the requests for high availability.
Condition 3 - Tier-2 Read Replica Failure : Only the non performing Tier-2 Read Replica instance has to be recreated
Condition 4- AZ NW problem: In event AZ-3 is failed, Requests are served by alternate fleet of Tier-1+Tier-2 in another AZ(AZ-4).

Multi-AZ Hot Standby Implementation is a recommended best practice when it comes to Multi-tiered Read replica implementation

• It is recommended to create Tier 1 Read Replicas from Multi-AZ DB instance to offload read queries from the source master DB instance for high traffic sites. If the source instance of a Multi-AZ deployment fails over to the secondary, any associated read replicas will be switched to use the secondary as their replication source automatically. This model guarantees high availability.
• Also when you initiate the creation of a Tier 1 read replica, Amazon RDS takes the DB Snapshot of your Standby DB instance (instead of Source DB) and begins replication. This model saves I/O suspension on your source DB during the snapshot process 

Other Points to Note:

Note : 
To Load Balance AWS RDS Read Replica's Refer this article : http://harish11g.blogspot.com/2013/08/Load-balancing-Amazon-RDS-MySQL-read-replica-slaves-using-HAProxy.html

P1) Circular replication are not allowed in this tiered replica creation process. 
P2)Third Tier cannot be created. Actually in production, very rarely we need third tier and it is not important feature
P3) When the "X" Tier replica is placed in Different AZ , you should factor few extra milliseconds of latency during replication.
P4) Before a DB instance (Master or Tier 1) can serve as a replication source, you must enable automatic backups on the source DB instance by setting the backup retention period to a value other than 0. This requirement does not apply to second tier replica as they are not source DB instance for another read replica.So create “Read Replica” option will appear only when the Backup Retention is set to minimum of 1 day for any level Read Replica creation. When we "create Read Replica" by default RR is created with Backup Retention set to 0. Use the modify option Illustrated in below screenshot and set the  Backup Retention Period.  :

P5) We have used t1.micro as RDS DB for article explanatory purpose. For production use cases please use proper instance types after proper capacity planning.
P6) AWS has released Parallel replica creation process, where you can create multiple Tier 1 and Tier 2 replicas in parallel. Since you no longer need to wait for one replica creation before starting the next one, it becomes easy to create multiple RR quickly. Without this feature, it would take hours to create a large Multi-tiered Replica setup.
P7) If a replica lags too far behind for your environment, the normal practice is to scale up or consider deleting and recreating the read replica. Imagine you have architecture where there are two Tier 1 and three Second Tier RR as illustrated in below architecture:

Now since Tier -1 RR is lagging , you are planning to delete and recreate the same. When you delete the Tier-1 RR, All the Second Tier Read Replica's will now become Standalone , Single AZ DB's. Once you have recreated the Tier -1 RR again and managed to retain the original end point as well, you cannot re-point these Second Tier RR to Tier -1 again. You need to recreate all 3 second tier from the Tier -1 again. For a application with heavy DB dependency and read traffic, it means 1/2 of the fleet is now down. This can lead to uncomfortable performance situation. This is not a ideal condition and AWS RDS team can take this in their Road Map. For such cases, it is better to create a new Tier -1 and Second tier fleet first , update your LB/App configs and then delete the old lagging fleet 
P8)Why it is better to have Multi-tiered Read Replica as the last resort in your architecture ?
8.a)Currently 30 Read Replica can be created overall for a master in two tiers. 30 Read replica is more than sufficient and usually turns out to be costly architecture approach. Use this approach only for cases, which demand heavy read and when application code cannot accommodate changes and are highly DB dependent.
8.b)It is recommended in DB world to stay away from Multi-level replication as much possible. Your architecture will be much simpler with one master and "X" replica slaves, rather than having tiered replica's. As we observed in above deployment architectures, the second tier replica slave will be a trouble to manage in event of replication delay, crashes and network problems affecting the Tier-1 RR or the Master DB. 
8.c)In case your application code can be redesigned, it is recommended to take following approaches before resorting to Second Tier replicas architecture

• Functional partition the RDS MySQL with Hot Stand By and Read Replica's
• Re-balance the DB load by using alternate data stores provided by AWS like DynamoDB, ElastiCache, CloudSearch etc.

If the above methods does not work for you, take the Second tier Read replica approach.

Thanks Senthil and Ram of 8KMiles for taking part in this analysis.

Amazon ELB Multi region migration checklist

Amazon Elastic Load Balancing has a Amazon EC2 Regional scope. It needs to be migrated to alternate Amazon EC2 region in event of DR or during new migration setup . I am sharing my experience in this post as few checklists/areas that needs to be taken care during this Amazon ELB migration to alternate EC2 region:

1. Amazon ELB automatically distributes incoming application traffic across multiple Amazon EC2 instances in Multiple Availability Zones. Amazon EC2 regions have different levels of AZ's, US-East EC2 region has 5 AZ's and Singapore has 2 AZ's. When you are migrating the ELB make sure your configuration matches the levels of target Amazon EC2 region Availability zones .
2. When Amazon ELB is created on target Amazon EC2 region, the ELB endpoint will be different from the source one. This has to be re configured on your Route 53 and other thick clients during migration.
3. Since the ELB end point is different, if you had raised any pre-warming request to AWS support , please raise it again for the target region with new ELB end point. This is valid in use cases where you are moving the traffic from primary region to target region in event of outage. 
4. SSL certificates associated on the old Elastic Load Balancing service (source regions) need to be re configured to the new Elastic Load Balancing service in the Target region.
5. Security Groups are region specific and has to be migrated/recreated in the target region. Make sure the security groups of ELB in target region matches the source region
6. Amazon ELB is not an isolated service and usually it will be used in production with Amazon EC2, Auto Scaling group and Cloud Watch Alarms. In addition there will be health checks, time outs, thresholds etc configured specifically for an application in Amazon ELB. You need to ensure that all these items associated with Amazon ELB are transferred target region for proper functioning of the application. Using Amazon CloudFormation templates for creating infrastructure properties will surely help and ease the efforts involved during such ELB migration.
7. To load balance your EC2 instances launched in default VPC, you have to create your load balancers within your default VPC. When you create a load balancer within default VPC, Elastic Load Balancing automatically creates a security group by defining the ports specified for the load balancer to be opened. Default VPC is currently not available in US-East. In case your Source region is US-West and you are planning your DR or migration with Target region as US-East, you need to be aware of this situation. 
8. In case Amazon ELB and Internal ELB are used inside Amazon VPC, you should ensure that the IP address ranges being used in source and target regions matches and aligns properly, so that routing and load balancing continues to work after migration. 

Other Multi Region Migration articles:

Security Groups - Multi Region Migration checklist

Elastic IP - Multi region Migration checklist, tips , best practices

EC2 AMI Copy - Multi Region Migration

MySQL Amazon EC2 Cross region Master-Slave Replication using SSL

Amazon EBS Snapshot copying- Performance, Tips and How to

Amazon EC2 AMI Copy Feature - Performance, Tips and How To

Caching architectures using Memcached & Amazon ElastiCache

Applications can be often made to perform better and run faster by caching critical pieces of data in memory.  Frequently accessed data, layers of HTML fragments, results of time-consuming/expensive database queries, search results, sessions, results of complex calculations and processes are usually very good candidates for cache storage. In general not all application architectures will benefit from having a caching solution in their system, example applications that are read intensive, will usually have better performance gains using cache whereas write intensive applications may not get much benefit.

There are various ways in which caching layer can be designed in AWS infra. The most popular model is distributed caching using Memcached.

Memcached is a high-performance, in-memory key-value store with distributed memory object caching system.

Anatomy of a Memcached system

• A memcached client which is given a list of available memcached servers in the farm
• A memcached client-based hashing algorithm which decides (GET/PUT) a server based on the "key" input.
• Memcached server instances which stores your values with their keys into an internal hash table

Memcached uses memory and NW heavily followed by CPU. Memcached supports TCP and UDP protocols in binary and text format for communication. Memcached supports client libraries for popular programming languages like Java, .Net, PHP, Python, Ruby etc.

In this article let us explore and analyze some popular memcached/ElastiCache deployment architectures in AWS.

Architecture 1: Apache + Memcached shared in same EC2 (Distributed Cache)

Memcached is shared with Apache in the same Amazon EC2 instance. Imagine m1.large EC2 instance where 7 GB RAM and 2 CPU Cores are shared between OS, Apache and memcached. Since Amazon ElastiCache runs on a separate tier it will not fit this shared approach. The Apache-A can contact any memcached-A/memcached-B or memcached-X node depending upon the Key/Value hash. Since the unused memory is shared with memcached and no dedicated EC2 instances are launched for caching tier this model is usually cost effective. We have seen some implementations using this approach in production but in my opinion it is not suitable for applications which demands heavy scaling and clear service separation. This model can be scaled up and but not scaled out optimally, ie based on your traffic demand you can Scale up the shared EC2 instance to bigger capacities like Xlarge, Quadruple, High IO ,M3 class etc. but not easily add new instances of this type. Some negatives that this approach brings to the table are:

Maintenance: Since the Apache and memcached are shared in the same EC2 it is strictly advised not to Auto Scale this layer. Only Manual Scaling is possible in this layer which might add heavy configuration burden to IT team during traffic peaks and valleys.

Auto Scaling: Using Amazon Auto Scaling we can add new web/app EC2 instances dynamically depending upon the traffic demands. But when web/app Server instance also contains memcached running in it, it brings in cascading complexity into the architecture. Imagine there are 2 m1.large Apache + memcached EC2 instances running and a third one is launched by Amazon Auto Scaling based on the traffic. Now the load balancer splits the 1/3rd of the web traffic to this Apache EC2 instance. Since the cache is empty 1/3^rd of the requests will now hit the backend database heavily. Now imagine instead of 1 EC2 you are auto scaling out by 2 Apache EC2 during peak, this will increase Database load to 50% more because of un warmed memcached. Secondly, the new memcached endpoint has to be propagated and configured on other memcached clients, which adds another complexity and devops engineering into the architecture. Finally, Amazon Auto Scaling will pull out an Apache EC2 instance when the load decreases, now if you are pulling out an Apache + memcached that is properly warmed it will again increase the DB load because of the cache miss.  

Note:  We can still try to address this problem by adding more complexity of designing/engineering with progressive weighted EC2 balancing + Scaling out and internal Cache Warming techniques etc, but if you deeply ask a question is it worth it, it is not many times. Alternatively, we can altogether avoid this complexity by simplifying the overall architecture of the system, which we will see as the article progresses.

Sharing: We observed earlier that sharing Apache+ memcached on same EC2 saves cost. On the other hand, this sharing also causes problems if one is not aware of the environment. In our case Apache + memcached are shared and Apache-A can talk to memcached in same EC2 or other Apache EC2 instance as well depending upon KV hash. Based on this flow let us explore some problems in sharing approach.

• ·         Apache is usually heavy on Memory and CPU. Memcached is low on CPU, high on memory and network depending upon the average size of your items.
• ·         If the memcached is not configured with Memory limits it can crash your Apache and OS. If the Website is heavily loaded and built to be cache dependent there will be heavy CPU contention between Apache and memcached.
• ·         If the request/response of Apache and memcached are bigger in size there will be bigger contention on the shared network layer. Overall request throughput can reduce because of heavy buffering and NW contention.
• ·         Apache EC2 instance will now have bigger headache of handling all the TCP sockets flowing between Internet, Database, Internal NW and memcached.  Some of them address the last point marginally by using UDP protocol for memcached communication and reduce the TCP socket temporary exhaustion. Overall this is a stitch and not a proper solution.  

Architecture 2: Apache + Memcached shared in same EC2 (like Local Cache)

This approach has a slight difference from the above one.  Apache/NginX and memcached are shared in the same EC2 instance, but the Web (Apache or NginX) process will strictly and only call the local memcached and will not call the remote memcached. Basically, memcached is used here as a local instance cache and not as a distributed cache. Every Apache/NginX will cache items in memcached and use it as extended memory. Since the items are coming from the same EC2 instance, the throughput and latency are better for cached entries. Though this approach has lesser configuration headache than the previous approach, it still inherits lots of problems from the previous one. Session sticky algorithm is preferred on the Load balancing tier to optimally reuse the cache items and reduce the DB load because RR algorithm can heavily exercise the DB during initial cache warming phase.   Rapid scaling out and scaling down should be avoided on smaller deployments because it transfers the load on the DB immediately. If already there is a large fleet (100’s) of NginX + memcached running then rapidly adding few (5-10) EC2’s of this kind will not have huge problems on the DB. Proper architecture guidance is recommended before fitting the above architecture into the use case.

As we observed  in detail, the above approaches might be cost effective for smaller deployments , but as the site gets popular, traffic increases and it demands scalability it will become complex to handle. Usually in architecture if complexity arises because of improper designs, it will be followed by heavy maintenance and management cost.

Now that we have understood the impacts sharing memcached with web/app server, a simple solution is to split the memcached into separate EC2 instance. Recently introduced M3 class instance types are good candidates for designing separate memcached tier. But the question is, whether we really need to manage and maintain a separate additional memcached layer. The answer is NO, USE AMAZON ELASTICACHE.

Amazon ElastiCache is a web service that is protocol-compliant with Memcached, a widely adopted memory object caching system, so code, applications, and popular tools that you use today with existing Memcached environments will work seamlessly with the service.

Architecture 3: Apache + Amazon ElastiCache in separate tier

Apache and Caching runs on clearly separated tiers in this approach.  Since the tiers are separated Apache EC2 can be easily scaled out using Amazon Auto Scaling or custom scaling. Dynamically launching/terminating Apache instances will not swamp database because the warmed cache is separated and still accessible by all the Apache EC2 instances. It is also easy to roll out configuration changes, add new nodes in caching layer and propagate the changes to the cache clients. The clear separation also enables us to isolate and address issues creeping up in Apache and Caching layer individually.

ElastiCache nodes are grouped inside an ElastiCache cluster.  An ElastiCache cluster is a collection of one or more cache nodes, each running an instance of the memcached service. The word cluster in this context should be related to “grouping” and not “data synchronization” because ElastiCache nodes will not talk to each other or exchange information between them inside the cluster. Most of the operations like configuration, security and parameter changes will be performed at the cache cluster level and not at individual cache node level. This enables easy maintenance and management of the caching tier on whole. Since ElastiCache is also protocol compliant with memcached, programs written in Java, PHP, and Python on Apache can still use their respective memcached clients and perform SET/GET operations seamlessly. The ElastiCache Node end points (like“ecache1a.sqjbuo.0001.use1.cache.amazonaws.com:11211") needs to be configured on the memcached clients of the Apache EC2. “ecache1a” is the cluster name, “0001” is the node number and 11211 is the port in the above mentioned URL endpoint. Whenever a new node is added into this “ecache1a” cluster, a sequence of numbers like “0002, 0003” will be assigned in end point URL to the nodes. This predictive pattern helps us to automate the detection of cache node endpoints in client side of scalable environments. Since a single ElastiCache cluster can currently span only in a single Amazon Availability zone, it is advised to keep both Apache EC2 and ElastiCache Instances in same Availability zone for improved latencies. Inside Single AZ a single SET/GET operation between Apache and ElastiCache will take around ~1-5 milliseconds using AWS High Memory Quadruple Instance types. This latency measurement also depends upon parameters like Apache EC2 instance type, ElastiCache Instance type, size of the SET/GET requests, Single or Bulk operations etc. Imagine you use m1.large for Apache and ElastiCache instance and every SET/GET is around 1 MB size. Then if the available NW bandwidth between Apache EC2 to ElastiCache is only 15 MB at that instant of time, only 15-20 requests can be performed concurrently at that instant. You may find the CPU under-utilized and max connection well set in ElastiCache, but still the throughput is less because of the above reason. This is not the problem of ElastiCache performance, but rather a bad understanding of the architecture components behaviors. If the web app is cache dependent, it is advised to spread the items in multiple cache nodes. Imagine you have close to 20 GB Cache size requirement. You can distribute it in either 2 m1.xlarge ElastiCache nodes or 4 m1.large ElastiCache nodes. The cache data will be distributed by the memcached client to multiple nodes based on the KV hash. In case one cache node goes down then 50% of the cache load will now hit the backend data stores in m1.xlarge approach whereas 25% of the cache load will only hit the data stores in m1.large approach.  Also since it is currently not possible to have multiple cache node instance types inside a single ElastiCache cluster, I advise you to do proper capacity planning taking into consideration the cache dependency and capacities of backend DB to take direct requests before planning the cache node numbers, size and consolidation levels.

Amazon ElastiCache as the name suggests you can automatically/manually add or remove cache nodes from the existing ElastiCache cluster making the whole tier elastic and flexible for customers. This is one of the important features of Amazon ElastiCache and this feature eventually falls in line on any growing websites roadmap.  Now let us try to understand the remapping implications while adding or removing cache nodes from the cache cluster.

With a normal hashing algorithm, changing the number of servers can cause many keys to be remapped to different servers resulting in huge sets of cache misses. Imagine you have 10 ElastiCache Nodes in your cache Cluster, adding an eleventh server may cause 40%+ of your keys to suddenly point to different servers than normal. This activity is undesirable, may cause cache misses and eventually swamping your backend DB with requests. To minimize this remapping it is recommended to follow consistent Hashing model in your cache clients. Consistent Hashing is a model that allows for more stable distribution of keys given addition or removal of servers. Consistent Hashing describes methods for mapping keys to a list of servers, where adding or removing servers causes a very minimal shift in where keys map to. Using this approach, adding an eleventh server should cause less than 10% of your keys to be reassigned. This % may vary in production but it is far more efficient in such elastic scenarios compared to normal hash algorithms.  It is also advised to keep memcached server ordering and number of servers same in all the client configurations while using consistent Hashing. Java Applications can use “Ketama library” through spymemcached to integrate this algorithm into their systems. More information on consistent hashing can be found at http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consistent_hashing_algo_for_memcache_clients

Deep dive into Amazon ElastiCache and understand the internals like connection overheads, memory allocations, Elasticity implications in this article:  http://harish11g.blogspot.in/2013/01/amazon-elasticache-memcached-internals_8.html

Architecture 4: Apache + Amazon ElastiCache in Multiple Availability Zones

This is an extension of the previous approach, for better availability the cache nodes are distributed among multiple Availability zones of an Amazon EC2 region. Most of the points discussed on the above approach will be applicable on this architecture as well. Since the ElastiCache cluster currently cannot span across multiple AZ’s you can create multiple ElastiCache clusters in Multiple AZ’s. Example: you can create ElastiCache cluster “ecache1a” in Amazon AZ - 1A and have a node launched with endpoint “ecache1a.sqjbuo.0001.use1.cache.amazonaws.com:11211”. In the same way you can create another ElastiCache cluster “ecache1b” in Amazon AZ – 1B and have a node launched with endpoint “ecache1b.sqjbuo.0001.use1.cache.amazonaws.com:11211”. Both the cache nodes endpoint should be configured in memcached clients.  Since the AZ concept is built transparently by AWS, the memcached clients in Apache EC2 can distribute data seamlessly and easily to both the cache nodes distributed across AZ’s.  You can manage the cache clusters separately as well you can distribute the data across AZ in this approach. In case an entire AZ is affected still the cache nodes in the other alternate AZ will be still accessible and functional.  Instead of DB getting swamped by 100% cache misses now you are reducing it to ~50% with AZ distribution in this approach. This % can be reduced much more if data is distributed among 2 or more AZ’s with more cache nodes inside them.

ElastiCache Maintenance Window allows you to specify the time range (UTC) during which any scheduled maintenance activities such as software patching or pending cache cluster modifications you requested would occur. Scheduled maintenance activities occur infrequently (generally once every few months) and will be announced on the AWS forum two weeks prior to being scheduled. After maintenance window our cache nodes may lose all the data stored in it memory and needs to be warmed again. Imagine having a single ElastiCache cluster with 10 cache nodes and all of them needing the cache warming phase after maintenance period, It puts heavy burden on your DB and other backend data stores during this refresh phase and sometimes even brings down your system to knees on heavy cache dependent architectures. Since AWS is very elastic and flexible, either you can plan to increase your backend capacity on demand for few hours to few days till the cache layer is adequately warmed or leverage the multi-AZ ElastiCache approach. Imagine you have 4 ElastiCache clusters distributed in 4 Availability zones inside an Amazon EC2 region. You can configure maintenance windows spanning multiple days for multiple cache clusters. Example ecache1a can have maintenance on Monday, ecache1b on Tuesday so forth. This distribution of ElastiCache Maintenance windows may give you enough time to warm cache nodes in phases and also helps you avoid cache swamping your backend with requests simultaneously.

This architecture approach is not suitable for smaller deployments running in single AZ’s. I suggest this for only larger deployments where Apache EC2’s are auto scaled, Apache and ElastiCache clusters are well distributed across multiple AZ’s so that overall cache item SET/GET latencies are in acceptable levels.  

Launch Amazon ElastiCache in 3 Easy Steps:  http://harish11g.blogspot.in/2012/11/configuring-amazon-elasticache-launch.html

Architecture 5: Apache + Amazon ElastiCache + Redundancy

This is a slightly different approach built with availability and redundancy. Apache and ElastiCache are deployed in separate tiers. Apache EC2 can be individually auto scaled across multiple AZ’s.  Multiple ElastiCache clusters are created spreading across multiple availability zones inside Amazon EC2 region (till now very similar to previous approach).  Certain items are redundantly cached in two cache nodes in multiple AZ- ElastiCache clusters for better availability in this approach. Results of time consuming and expensive data base queries, results of complex calculations etc are good candidates for this approach. Imagine an expensive query that pounds the database for ~250 or more milliseconds, if the data does not change quite frequently in this case it can be redundantly stored in 2 ElastiCache nodes.  If cache node 1 is down or throws connection error or if cache item miss occurs then the redundant cache node 2 can be requested for the same item. If the item is not present in cache node 2 also then as last resort the DB is queried and latest result is stored in both the cache nodes redundantly. Imagine it takes around 2-5ms for single ElastiCache node to return a value hitting 2 cache nodes redundantly still gives the results in ~10ms, which is far better compared to pounding and getting the result from DB. This approach is not suitable for frequently changing data flows because it may result in fetching stale data from the cache, for such scenarios ElastiCache-> DB fallback approach is better. Also it is not necessary to have redundancy built for all the cache nodes and cache clusters totally, you should build redundancy only for specific cache nodes in the system. This feature is not pre built on the memcached API’s currently and it has to be manually implemented in the application code by crudely making multiple calls to multiple sets of cache nodes. Though it reduces the overall GET time for complex requests, your SET times will marginally increase because of multiple requests made to cache nodes.

 It is costlier and complex compared to other architecture approaches mentioned above. But for some use cases it can save on your database HW capacity cost heavily and provide immense infra cost savings overall. It is suggested to carefully analyze the fitment of this approach based on your use case, cost and maintenance needs.


Related Articles
Part 1: Understanding Amazon ElastiCache Internals : Connection overhead
Part 2: Understanding Amazon ElastiCache Internals : Elasticity Implication and Solutions
Part 3: Understanding Amazon ElastiCache Internals : Auto Discovery
Part 4: Understanding Amazon ElastiCache Internals : Economics of Choosing Cache Node Type
Launching Amazon ElastiCache in 3 Easy Steps
Caching architectures using Memcached & Amazon ElastiCache
Web Session Synchronization patterns in AWS

Dissecting Amazon ELB : 18 things you should know

While designing highly scalable systems load balancing tier becomes an integral part of any architecture. We have captured some of our prior experiences working with Amazon ELB in this article as points detailed below. Some of the points mentioned here will be encountered only by advanced users in complex use cases. But surely if you/your team have noted some of these points, I feel it might shorten your efforts while debugging a problem or designing a solution and not go through the same effort cycle and pain as our team.

In AWS, there are wide variety of solution choices for the Load balancing layer like Amazon Elastic Load Balancing (ELB) , EC2 AMI’s like HAProxy , Nginx , Zeus , Citrix NetScaler. In this article we are going to dissect our experience with Amazon ELB layer as X points which you will not frequently encounter in Amazon documents or blogosphere. 

To know more about Configuring Amazon ELB in 4 Easy Steps, Refer article:   http://harish11g.blogspot.in/2012/02/configuring-aws-elb-console.html

Currently there are 18 points in this article and i am having plans to add some more in coming days . So if you are an advanced user of Amazon ELB , please watch this article closely. 

Some of the points are:

Point 1) Algorithms supported by Amazon ELB

Currently Amazon ELB only supports Round Robin(RR) and Session Sticky Algorithms.

Round Robin algorithm can be used for load balancing traffic between

•  Web/App EC2 instances which are designed stateless
•  Web/App EC2 instances which synchronizes the state between them
•  Web/App EC2 instances which synchronizes the state using common data stores like MemCached , ElastiCache , Database etc.

Session Sticky algorithm can be used for load balancing traffic between

• Web/App EC2 instances which are designed to be statefull

Current Version of ELB does not support Weighted or Least Connection algorithms like other Reverse proxies. We can probably expect these algorithms to be supported in future.

Point 2) Amazon ELB is not a PAGE CACHE

Amazon ELB is just a load balancer and not to be confused with Page Cache Server or Web Accelerator.  Web Accelerators like Varnish can cache pages, Static assets etc and also do RR load balancing to backend EC2 servers. Amazon ELB is designed to do just Load balancing efficiently and elastically. If you need page accelerators + LB you can use Varnish or NetScaler in your LB Tier. Refer Article Varnish or NetScaler. Amazon ELB can be used with Amazon CloudFront to deliver the static assets and dynamic assets that can be page cached at edge location itself to reduce latency for above use cases.

Point 3) Amazon ELB can be pre warmed on request basis

Amazon ELB can be pre warmed by raising a request to Amazon Web Service Support Team. Amazon team will pre warm the Load Balancers in the ELB tier to handle the sudden load/flash traffic. This is advisable for scenarios like Quarterly sales/launch campaigns, promotions etc which follow flash traffic pattern. AWS team would require details like estimated Request per second, average request size in bytes, average response size in bytes, what percentage of traffic is SSL/ Non SSL, whether HTTP/1.1 keep alive is enabled ? etc from your team. Once provided, it will be activated by them. Amazon ELB pre warm cannot be done on hourly/daily basis (i think). It will be a cool feature if Amazon team can get these details and offer ELB Pre warming as a configurable feature into the AWS console (like Amazon DynamoDB console)

Point 4) Amazon ELB is not designed for sudden load spikes /Flash traffic

Amazon ELB is designed to handle unlimited concurrent requests per second with “gradually increasing” load pattern.  It is not designed to handle heavy sudden spike of load or flash traffic. For example: Imagine an e-commerce website whose traffic increases gradually to thousands of concurrent requests/sec in hours, Amazon ELB can easily handle this traffic pattern. According to RightScale benchmark, Amazon ELB was easily able to handle 20K+ requests/sec and more in such patterns. Whereas imagine use cases like Mass Online Exam or GILT load pattern or 3-Hrs Sales/launch campaign sites expecting 20K+ concurrent requests/sec spike suddenly in few minutes, Amazon ELB will struggle to handle this load pattern. If this sudden spike pattern is not a frequent occurrence then we can pre warm ELB, else we need to look for alternative Load balancers in AWS infrastructure. 

Comparison analysis of HAProxy vs Amazon ELB, Refer article:   http://harish11g.blogspot.in/2012/11/amazon-elb-vs-haproxy-ec2-analysis.html

Point 5) Protocols supported by Amazon ELB

Currently Amazon ELB only supports following protocols: HTTP, HTTPS (Secure HTTP), SSL (Secure TCP) and TCP protocols. ELB supports load balancing for the following TCP ports: 25, 80, 443, and 1024-65535. In case RTMP or HTTP Streaming protocol is needed, we need to use Amazon CloudFront CDN in your architecture.

Point 6) Amazon ELB timeouts at 60 seconds (kept idle)

Amazon ELB currently timeouts persistent socket connections @ 60 seconds if it is kept idle. This condition will be a problem for use cases which generates large files (PDF, reports etc) at backend EC2, sends them as response back and keeps connection idle during entire generation process. To avoid this you'll have to send something on the socket every 40 or so seconds to keep the connection active in Amazon ELB. Note: I heard we can extend this value after explaining the case to AWS support team.

Point 7) Amazon ELB does not provide Permanent or Fixed IP for its load Balancers

Currently Amazon ELB does not provide fixed or permanent IP address for the Load balancing instances that are launched in its tier. This will be a bottleneck for enterprises which have compulsion to whitelist their Load balancer IP’s in external firewalls/gateways. For such use cases, currently we can use HAProxy, NginX, NetScaler over EC2 attached with Elastic IPs as load balancers in AWS infrastructure.

Designing High Availability @ HAProxy / ELB Layer http://harish11g.blogspot.in/2012/10/high-availability-haproxy-amazon-ec2.html

Point 8) Amazon ELB cannot do Multi AWS Region Load Balancing

Amazon ELB can be used to Load balance

• Multiple EC2 instances launched inside a Single Amazon Availability Zone
• Multiple EC2 instances launched inside Multiple Availability Zones inside a Single Region

Amazon ELB cannot load Balance between EC2 instances launched on Multiple AWS regions . Use Route53 DNS RR / LBR / Failover configurations for Load balancing at DNS level between ELB, EC2 etc launched at multiple AWS Regions

To know more about DNS Load Balancing : http://harish11g.blogspot.in/2012/06/aws-high-availability-dns-load.html To know more about Geo Distributed Load Balancing using Amazon Route 53 : http://harish11g.blogspot.in/2012/09/geo-distributed-route53-lbr-latency.html

Point 9) Amazon ELB sticks request when traffic is generated from Single IP

This point comes as a surprise to many users using Amazon ELB. Amazon ELB behaves little strange when incoming traffic is originated from Single or  Specific IP ranges, it does not efficiently do round robin and sticks the request.  Amazon ELB starts favoring a single EC2 or EC2’s in Single Availability zones alone in Multi-AZ deployments during such conditions. For example: If you have application A(customer company) and Application B, and Application B is deployed inside AWS infrastructure with ELB front end. All the traffic generated from Application A(single host) is sent to Application B in AWS, in this case ELB of Application B will not efficiently Round Robin the traffic to Web/App EC2 instances deployed under it. This is because the entire incoming traffic from application A will be from a Single Firewall/ NAT or Specific IP range servers and ELB will start unevenly sticking the requests to Single EC2 or EC2’s in Single AZ.

Note: Users encounter this usually during load test, so it is ideal to load test AWS Infra from multiple distributed agents. 

Point 10)Too long Load Balancer CNAMES causes issues in some firewalls /ISP

Some ISP's do not allow Amazon ELB CNAMES that exceeds 32 characters and some firewalls versions/models (like Cisco PIX) will not allow larger CNAMES , in such cases try to have shorter name.

Point 11) Amazon ELB cannot Load Balance based on URL patterns

Amazon ELB cannot Load Balance based on URL patterns like other Reverse proxies. Example Amazon ELB cannot direct and load balance between request URLs  www.xyz.com/URL1 and www.xyz.com/URL2. Currently for such use cases you can use HAProxy in EC2.

Point 12) Amazon ELB can easily support more than 20K+ Concurrent reqs/sec

Amazon ELB is designed to handle unlimited concurrent requests per second. ELB is inherently scalable and it can elastically increase /decrease its capacity depending upon the traffic. According to a benchmark done by RightScale, Amazon ELB was easily able to scale out and handle 20K or more concurrent requests /sec. Refer URL: http://blog.rightscale.com/2010/04/01/benchmarking-load-balancers-in-the-cloud/

Point 13) Amazon ELB does not provide logs

Amazon ELB currently does not provide access to its log files for analysis. We cannot debug load balancing problems , analyze the traffic and access patterns; categorize bots / visitors etc currently because we do not have access to the ELB logs.This will also be a bottleneck for some organizations which has strong audit/compliance requirements to be met at all layers of their infrastructure. Amazon ELB can generate the logs and put in Amazon S3 buckets– (feature request to Amazon ELB product team)

Point 14) Monitoring Amazon ELB

Amazon ELB is an AWS building block and it does not currently provide access to its logs or Stats files for monitoring. Secondly, we cannot get full access to the Load Balancers launched inside the ELB tier and install any monitoring agents in it. This closed model of ELB makes us rely only on CloudWatch metrics for monitoring. Refer this URL for ELB metrics that can be currently monitored: http://harish11g.blogspot.in/2012/02/cloudwatch-elastic-load-balancing.html

Point 15) Amazon ELB and Compliance requirements

SSL Termination can be done at 2 levels using Amazon ELB in your application architecture .They are

• SSL termination can be done at Amazon ELB Tier, which means connection is encrypted between Client(browser etc) and Amazon ELB, but connection between ELB and Web/App EC2 is clear. This configuration may not be acceptable in strictly secure environments and will not pass through compliance requirements.
• SSL termination can be done at Backend with End to End encryption, which means connection is encrypted between Client and Amazon ELB, and connection between ELB and Web/App EC2 backed is also encrypted. This is the recommended ELB configuration for meeting the compliance requirements at LB level. 
• Important ELB-SSL Reference URLs 
• Refer this URL to understand how to configure SSL Offloading in Amazon ELB : http://harish11g.blogspot.in/2012/03/ssl-offloading-elastic-load-balancing.html
• How Amazon ELB-SSL support/offloading helps Cloud Admins ? :http://harish11g.blogspot.in/2010/10/amazon-elastic-load-balancing-support.html

Point 16) Amazon ELB and X.X.X.255 IP address

Sometimes ELB assigns its load balancers with IP address ending with X.X.X.255. Though it is technically fine, there are certain networks that will not properly route to an IP address ending in X.X.255 series. Unfortunately, it is not possible to exclude an IP address ending in .255 from ELB currently. It is possible, in such circumstances, some requests from certain users may face issues. Note this when you are debugging ELB for missing requests. 

Point 17) Amazon ELB inherently fault tolerant and Scalable service

Elastic Load Balancer does not cap the number of connections that it can attempt to establish with the load balanced Amazon EC2 instances. We can expect this number to scale with the number of concurrent HTTP, HTTPS, or SSL requests or the number of concurrent TCP connections that the Elastic Load Balancer receives. Since multiple Load balancers are launched in ELB tier, it is inherently fault tolerant as well. If you need a Scalable and and elastic LB layer , then ELB comes highly recommended. Amazon ELB can be deployed to support following HA architectures in AWS : http://harish11g.blogspot.in/2012/02/elastic-load-balancing-aws-deployment.html

Point 18) Amazon ELB + Amazon AutoScaling : No graceful connection termination

Amazon ELB can be configured with work seamlessly with Amazon AutoScaling and Amazon CloudWatch. The New EC2 instances launched by AutoScaling are added to the ELB for Load balancing automatically and whenever load drops; existing EC2 instances can be removed by Auto Scaling from ELB. Both Auto Scaling and ELB use CloudWatch Monitoring for enabling this functionality. The important point to remember while using this kind of integration is Amazon AutoScaling does not gracefully (without interruption to existing connections) remove Web/App EC2 from Amazon ELB. The connections are instantly dropped when the Web/App EC2 is removed and no grace period is given by ELB or AutoScaling. This behavior of Auto scaling can make dozens or hundreds of users to get error pages when they are using the application when such an event occurs in the backend infrastructure.

To know more about Amazon Auto Scaling : http://harish11g.blogspot.in/2011/04/auto-scaling-in-aws-cloud.html Harish Presentation @ AWS Summit on Amazon Auto Scaling http://harish11g.blogspot.in/2011/11/scale-new-business-peaks-with-amazon.html How Auto Scaling can save costs ? http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html

Article under progress

Other Load Balancing Articles

Monitoring ELB with Amazon CloudWatch

Configuring Amazon ELB using AWS Management Console

Amazon ELB Implementation Architectures

Amazon Elastic Load balancing benefits

Why do we need Elastic Load Balancing ?

Offloading SSL in Amazon ELB