Billion Messages - Art of Architecting scalable ElastiCache Redis tier

Whenever we are designing a highly scalable architectures on AWS running thousands of application servers and supporting millions of requests, usage of NoSQL solutions have become inevitable part. One such solution we often been using for years on AWS is Redis . We love Redis. 
AWS introduced ElastiCache Redis on 2013 and we started using the same since it eased the management and operational efforts.  In this article i am going to share my experience on designing large scale Redis tiers supporting billions of messages per day on AWS, step by step guide on how to deploy the same, what are the Implications you face at scale ? Best Practices to be adopted while designing sharded+replicated Redis Tiers etc.

Since we need to support billions of message requests per day and it was growing:

• the ElastiCache Redis tier was designed with Partitions( shards) to scale out as the customer grows
• the ElastiCache Redis tier was designed with Replica Slaves for HA and read scaling as the read volumes grow

When your application is growing at Rapid pace and lots of data are created every day, you cannot keep increasing (scaling up) the size of the ElastiCache Node. At one point you will hit the maximum memory capacity of your EC2 instance and you will be forced to partition.  Partitioning is the process of splitting your Key Value data into multiple ElastiCache Redis instances, so that every instance will only contain a subset of your Key Value pair. It allows for much larger ElastiCache Redis data stores, using the sum of the memory of many ElastiCache Redis Nodes. It also allows to scale the computational power to multiple cores and multiple EC2, and the network bandwidth to multiple EC2 network adapters. There are two widely used partition/shard implementation techniques that are available for ElastiCache Redis Tier :
Technique 1) Client side partitioning means that the Redis clients directly select the right ElastiCache Redis node where to write or read a given key. Many Redis clients implement client side partitioning, chose the right one wisely.
Technique 2) Proxy assisted partitioning means that your clients send requests to a proxy that is able to speak the Redis protocol, which in turn sends requests directly to the right ElastiCache Redis instance. The proxy will make sure to forward our request to the right Redis instance accordingly to the configured partitioning schema. Currently the most widely used Proxy assisted partitioning tool is Twemproxy , written by Manju Raj of twitter. Git hub link https://github.com/twitter/twemproxy . Twemproxy is a proxy developed at Twitter for the Memcached ASCII and the Redis protocol. Twemproxy supports automatic partitioning among multiple Redis instances and  currently it is the suggested way to handle partitioning with Redis.

In this article we are going to explore in detail about Proxy assisted partitioning technique for highly scalable and available Redis tier.

Welcome to Twemproxy
Twemproxy( nutcracker) is a fast single-threaded proxy supporting the Memcached ASCII protocol and more recently the Redis protocol.

Installing Twemproxy:
Download the Twemproxy package.
wget http://twemproxy.googlecode.com/files/nutcracker-0.3.0.tar.gz
tar -xf nutcracker-0.3.0.tar.gz
cd nutcracker-0.3.0
./configure
make
make install

Configuration:
Twemproxy (Nutcracker) can be configured through a YAML file specified by the -c or --conf-file command-line argument on process start. The configuration file is used to specify the server pools and the servers within each pool that nutcracker manages. The configuration files parses and understands the following keys:

• listen: The listening address and port (name:port or ip:port) for this server pool.
• hash: The name of the hash function.
• hash_tag: A two character string that specifies the part of the key used for hashing. Eg "{}" or "$$". Hash tag enable mapping different keys to the same server as long as the part of the key within the tag is the same.
• distribution: The key distribution mode.
• timeout: The timeout value in msec that we wait for to establish a connection to the server or receive a response from a server. By default, we wait indefinitely.
• backlog: The TCP backlog argument. Defaults to 512.
• preconnect: A boolean value that controls if nutcracker should preconnect to all the servers in this pool on process start. Defaults to false.
• redis: A boolean value that controls if a server pool speaks redis or memcached protocol. Defaults to false.
• server_connections: The maximum number of connections that can be opened to each server. By default, we open at most 1 server connection.
• auto_eject_hosts: A boolean value that controls if server should be ejected temporarily when it fails consecutively server_failure_limit times. See liveness recommendations for information. Defaults to false.
• server_retry_timeout: The timeout value in msec to wait for before retrying on a temporarily ejected server, when auto_eject_host is set to true. Defaults to 30000 msec.
• server_failure_limit: The number of consecutive failures on a server that would lead to it being temporarily ejected when auto_eject_host is set to true. Defaults to 2.
• servers: A list of server address, port and weight (name:port:weight or ip:port:weight) for this server pool.

For More details Refer: https://github.com/twitter/twemproxy

Running and Accessing Twemproxy 
To start the proxy just use the command “nutcracker” with the configuration file path specified or in its default path(conf/nutcracker.yml) .
Based on the configuration the twemproxy will be running and listening. Configure your application to point to the port and address instead of the Redis cluster.

Twemproxy Deployment models:

We usually deploy Twemproxy one one of the following models in AWS :
Model 1: Twemproxy as a separate Proxy Tier: In this model Twemproxies are deployed in separate EC2 instances, The application tier is configured to point to Twemproxies . The Twemproxy tier in turn maintains the mappings to the ElastiCache redis nodes. It is better to use instances with very good IO bandwidth for twemproxy tier in AWS. In case you feel the instance CPU is underutilized, you can launch multiple Twemproxy instances inside the same single EC2 instance as well.

Though the above model looks clean and efficient there are optimizations that can be applied to this architecture :
What happens when the twemproxy01 fails, how will the Application server instances know about it ?
Why should i pay additional for twemproxy EC2 instances, Can it be minimized ?

Model 2 : Twemproxy bundled with application tier EC2's: In this model twemproxies are bundled in the same box of the application server EC2 itself. Since two twemproxies are not aware of each others existence, it is easy to architect this model even in App->Auto Scaling mode. Every application server talks to the local twemproxy deployed in the same box this saves cost and avoids managing additional tier complexity as well.

Reference ElastiCache Redis + Twemproxy  deployment:
(This is a Reference deployment, the same can be scaled out to hundreds depending upon the need. It is a Redis Partitioned + replicated setup )
1. Two ElastiCache Redis nodes in AWS (twem01 and twem02)
2. Replication group for each ElastiCache redis nodes (twem01-rg and twem02-rg with one Read Replica each)
3. Two twemproxy servers running in separate EC2. (twemproxy01 and twemproxy02)
Once the above setup is done please note down the endpoints. We will be using the Replication group endpoint as the ElastiCache Redis endpoint for the twemproxy.

ElastiCache Redis Endpoints:
twem01-twem01.qcdze2.0001.usw2.cache.amazonaws.com:6379
twem02-twem02.qcdze2.0001.usw2.cache.amazonaws.com:6379
ElastiCache Redis Replication endpoints:
twem01-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379
twem02-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379

To test the Twemproxy we pumped following keys:
Pump KV data through the Twemproxy01 (1-2000 keys)
Pump KV data through the Twemproxy02(2001-4000 keys).

Configuration:
beta:
  listen: 127.0.0.1:22122
  hash: fnv1a_64
  hash_tag: "{}"
  distribution: ketama   #Consistent Hashing
  auto_eject_hosts: false
  timeout: 5000
  redis: true
  servers:
- twem01-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379:1 server1
- twem02-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379:1 server2

Test 1: Testing Key accessibility . Testing "GET" operation across both the Twemproxy Instances for few sample keys. 

Fetch 4 Keys spread across 4000 KV data from Twemproxy01  EC2 instance:
[root@twemproxy01 redish]# src/redis-cli -h 127.0.0.1 -p 22122
redis 127.0.0.1:22122> get 1000
"1000-data"
redis 127.0.0.1:22122> get 2000
"2000-data"
redis 127.0.0.1:22122> get 3000
"3000-data"
redis 127.0.0.1:22122> get 4000
"4000-data"
Fetch 4 Keys spread across 4000 KV data from Twemproxy02  EC2 instance:
[root@twemproxy02 redish]# src/redis-cli -h 127.0.0.1 -p 22122
redis 127.0.0.1:22122> get 1000
"1000-data"
redis 127.0.0.1:22122> get 2000
"2000-data"
redis 127.0.0.1:22122> get 3000
"3000-data"
redis 127.0.0.1:22122> get 4000
"4000-data"

From the above test it is evident that all 4000 KV data inserted using both Twemproxies are accessible from both Twemproxies( testing the sample) even though they are not aware among themselves. This is because of the same hashing and Key mapping translation done at Twemproxy level.

Test 2: Testing the ElastiCache Redis Availability and Fail over mechanism:

We are going to promote the twem01-rg replication group read replica to be the Primary Redis Node. After promotion we are going to test:

1. Whether the Twemproxy is able to recognize the newly promoted master
2. Whether the sample KV data is safely replicated and still accessible , to ensure failover is successful.

To promote ElastiCache Redis slave just click the promote Action and confirm or automate using API. During the promotion of Read Replica to master we observed that the transition happens very quickly and there is no timeout but the response time for the query is about 4-5 secs for about 3-4 minutes during the switch over. In the Twemproxy configuration we can set the timeout configuration, this value needs to be set accordingly so that during switch over there will be no connection refused. For the sample test we have set it as 5000

Repeat Test 1:

[root@twemproxy01 redish]# src/redis-cli -h 127.0.0.1 -p 22122
redis 127.0.0.1:22122> get 1000
"1000-data"
redis 127.0.0.1:22122> get 2000
"2000-data"
redis 127.0.0.1:22122> get 3000
"3000-data"
redis 127.0.0.1:22122> get 4000
"4000-data"
Fetch 4 Keys spread across 4000 KV data from Twemproxy02  EC2 instance:
[root@twemproxy02 redish]# src/redis-cli -h 127.0.0.1 -p 22122
redis 127.0.0.1:22122> get 1000
"1000-data"
redis 127.0.0.1:22122> get 2000
"2000-data"
redis 127.0.0.1:22122> get 3000
"3000-data"
redis 127.0.0.1:22122> get 4000
"4000-data"

From the above test it is evident that all 4000 KV data are replicated properly between master and slaves nodes and the transition between slave to master happened successfully with all the data.

Reporting

Nutcracker exposes stats at the granularity of server pool and servers per pool through the stats monitoring port. The stats are essentially JSON formatted key-value pairs, with the keys corresponding to counter names. By default stats are exposed on port 22222 and aggregated every 30 seconds.

Some best practices while designing highly scalable+available ElastiCache Redis Tier :

Practice 1 : Reduce the Number of Connections and pipeline messages:
Whenever the application instance gets a request to get/put value to the ElastiCache redis node, the client makes a connection to the Redis Tier. Imagine it is a heavy traffic site, then thousands of requests hitting translates to thousands of connections from the application instance to Redis Tier. Now when you add Auto- scaling to your application tier and you have few hundred servers scaled out , then imagine the connection complexity and overhead this architecture brings to the ElastiCache Redis Tier.

Best practice is minimize the number of connections made from your application instance to your ElastiCache redis node. Use Twemproxy in bundled mode with Application EC2 instance, this keeps the process in close proximity and reduces the connection overhead.  Secondly, Twemproxy internally uses minimal connections to ElastiCache Redis Instance by proxying multiple client connections onto one or few server connections.
Redis also supports pipelines, where multiple requests can be pipelined and sent on a single connection. In a simple test using large Application & ElastiCache node we were able to process 125K message/sec in pipeline mode, now imagine what you could achieve on bigger instance types on AWS. The connection minimization architectural setup of twemproxy makes it ideal for pipelining requests and responses and hence saving on the round trip time.  For example, if twemproxy is proxying three client connections onto a single server and we get requests - 'get key\r\n', 'set key 0 0 3\r\nval\r\n' and 'delete key\r\n' on these three connections respectively, twemproxy would try to batch these requests and send them as a single message onto the server connection.

Note : It is important to note that "read my last write" constraint doesn't necessarily hold true when twemproxy is configured with server_connections: > 1. Let us consider a scenario where twemproxy is configured with server_connections: 2. If a client makes pipelined requests with the first request in pipeline being set foo 0 0 3\r\nbar\r\n (write) and the second request being get foo\r\n (read), the expectation is that the read of key foo would return the value bar. However, with configuration of two server connections it is possible that write and read request are sent on different server connections which would mean that their completion could race with one another. In summary, if the client expects "read my last write" constraint, you either configure twemproxy to use server_connections:1 or use clients that only make synchronous requests to twemproxy.

Practice 2:  Configure Auto Ejection and Hashing combination properly
Design for failure is the mantra of cloud architecture. Failures are commons when things are distributed on scale. Though partitioning when using ElastiCache Redis as a data store or cache is conceptually the same on broad lines, there is a huge difference operationally on large scale systems. When you are using ElastiCache Redis as a data store you need to be sure that a given key always maps to the same instance, Whereas if you are using  ElastiCache Redis as cache if a given node is not available, then you can always start afresh using a different node in the hash ring with consistent hashing implementations.
To be resilient against failures, it is recommended that you configure Auto eject hosts false when you treat redis as a Data Store and true in when you treat redis as a cache.
resilient_pool:
  auto_eject_hosts: true
  server_retry_timeout: 30000
  server_failure_limit: 3
Enabling auto_eject_hosts: This property ensures that a dead ElastiCache redis Node can be ejected out of the hash ring after server_failure_limit: consecutive failures have been encountered on that node. A non-zero server_retry_timeout: ensures that we don't incorrectly mark a node as dead forever especially when the failures were really transient. The combination of server_retry_timeout: and server_failure_limit: controls the tradeoff between resiliency to permanent and transient failures.
Note that an ejected node will not be included in the hash ring for any requests until the retry timeout passes. This will lead to data partitioning as keys originally on the ejected node will now be written to another node still in the pool. If ElastiCache Redis is used as a cache (in memory) then in event of a Redis Node going down, the cache data will be lost. This cache miss can cascade performance problems to other tiers and altogether bring down your system on the cloud. To minimize KV cache miss,  you can design your hash ring with Ketama hashing on the Redis Proxy. This will minimize the Cache miss in event of cache node failure, also it decreases the overall re-balancing needed in your Redis tier.  In addition to helping hand on availability problems, Redis Proxy+Ketama can also help your Redis farm to Scale out and Scale down easily with minimal cache miss. To know more about Ketama on ElastiCache refer http://harish11g.blogspot.com/2013/01/amazon-elasticache-memcached-internals_8.html  . The below diagram illustrates a ElastiCache Redis Cache Farm with Consistent Hash Ring.

In short to minimize the cache miss when using auto eject with true it is recommended to use "Ketama Hashing ( Consistent Hashing Algorithm)" on your Twemproxy configuration. 

ElastiCache Redis as a Data Store:
What if the data stored in your Cache is important and needs to persisted across node failures and launch ? What if the date stored in your Cache cannot be lost and it needs to be replicated and promoted during failures?
Welcome to ElastiCache Redis as Data store. ElastiCache Redis offers features to persist the in memory cache data to disk and also replicate it to a slave for high availability. If ElastiCache Redis is used as a store (persistent), you need to keep the map between keys and nodes fixed, and a fixed number of nodes. Since the data stored is important when you treat ElastiCache Redis as a data store, in event one Redis node goes down, you should have immediate standby up and running in minutes.  You can architect ElastiCache Redis master with one or more replication Slave launched on different AZ from Master for High Availability in AWS. In event master node failure or master AZ failure, the slave Redis node can be promoted in minutes to act as master. This whole High availability design keeps the number of nodes on the hash ring stable and simple, Otherwise, you will end up building a system to re balance the keys (which is not easy) between nodes whenever there is a addition or removal of nodes during outages. In addition to above the ElastiCache Redis supports Partial Resynchronization with Slaves - If the connection between a master node and a slave node is momentarily broken, the master now accumulates data that is destined for the slave in a backlog buffer. If the connection is restored before the buffer becomes full, a quick partial resync will be done instead of a potentially longer full resync. This really saves network bottleneck during momentary failures.
In large scale systems you will often find some partitions are heavily used than others , in event the usage is read heavy in nature you can add upto 5 Read replicas for the ElastiCache Redis Master partition. Since these replicas are used only for read they do not affect the Hash ring structure. But Twemproxy lacks the support for read scaling with Redis Replica's. So in event when you face this problem, you will have to Scale up the capacity(instance/node type) of the Master and Slave of that partition alone.

If you are using ElastiCache redis as a Data store in the TwemProxy it is recommended to keep "auto_eject_hosts" property false so that in event of redis node failure it is not ejected from the hash ring. The hash ring can be built with both ketama or modula hash algorithms , since in event of Primary node failure, the Slave is going to be promoted and ring structure is going to be always maintained. But if you feel there is immense possibility for the number of primary node partitions to grow, or major failures to occu, it is better to choose ketama hash ring itself from beginning. The below diagram illustrates the architecture.

Practice 3: Configure the Buffer properly:
All memory for incoming requests and outgoing responses is allocated in mbuf in Twemproxy. Mbuf enables zero copy for requests and responses flowing through the proxy. By default an mbuf is 16K bytes in size and this value can be tuned between 512 and 16M bytes using -m or --mbuf-size=N argument. Every connection has at least one mbuf allocated to it. This means that the number of concurrent connections twemproxy can support is dependent on the mbuf size. A small mbuf allows us to handle more connections, while a large mbuf allows us to read and write more data to and from kernel socket buffers. Large Scale web/mobile applications involving millions of hits might have small size request/response and lots of concurrent connections to handle in their backend. So at such scenarios, when Twemproxy is meant to handle a large number of concurrent client connections, you should set chunk size to a small value like 512 bytes to 1K bytes using the -m or --mbuf-size=N argument.

Practice 4: Configure proper Timeouts
It is always a good idea to configure Twemproxy timeout: for every server pool, rather than purely relying on client-side timeouts. Eg:

resilient_pool_with_timeout:
  auto_eject_hosts: true
  server_retry_timeout: 30000
  server_failure_limit: 3
  timeout: 400
Relying only on client-side timeouts has the adverse effect of the original request having timed out on the client to proxy connection, but still pending and outstanding on the proxy to server connection. This further gets exacerbated when client retries the original request.


Benefits of using Twemproxy for Redis Scaling

• Avoids re inventing the wheel. Thanks Manju Raj (twitter).
• reduce the number of connections to your cache server by acting as a proxy
• shard data automatically between multiple cache servers
• support consistent hashing with different strategies and hashing functions
• be configured to disable nodes on failure
• run in multiple instances, allowing client to connect to the first available proxy server
• Pipelining and batching of requests and hence saving of round-trips

Disadvantages of Partitioning Model:

Point 1) Operations involving multiple keys are usually not supported. For instance you can't perform the intersection between two sets if they are stored in keys that are mapped to different Redis instances (actually there are ways to do this, but not directly).Redis transactions involving multiple keys can not be used.
Point 2) The partitioning granularity is the key, so it is not possible to shard a dataset with a single huge key like a very big sorted set. Ideally in such cases you should Scale UP the particular Redis Master-Slave to larger EC2 instance or pro grammatically stitch up the sorted set.
Point 3)When partitioning is used, data handling is more complex, for instance you have to handle multiple RDB / AOF files, and to make a backup of your data you need to aggregate the persistence files/snapshots from multiple EC2 Redis slaves.
Point 4) Architecting a partitioned + replicated ElastiCache Redis tier not complex. What is more complex is ? supporting transparent rebalancing of data with the ability to add and remove nodes at runtime. Systems like client side partitioning and proxies don't support this feature. However a technique called Presharding helps in this regard with limitations. Presharding technique ->Since Redis is lightweight, you can start with a lot of EC2 instances since the beginning itself. For example if you start with 32 or 64 EC2 instances (micro or small Cache Node instance type)  as your node capacity , it will provide enough room to keep scaling up the capacity when your data storage needs increase. It is not a highly recommended technique. But still can be used in production if your growth pattern is very predictable.

Future of highly scalable + available Redis tiers -> Redis Cluster

Redis Cluster is the preferred way to get automatic sharding and high availability. It is currently not production ready. Once Redis Cluster / Client  is available on Amazon ElastiCache, it will be the de facto standard for Redis partitioning. It uses a mix between query routing and client side partitioning.

References:
http://redis.io/documentation
https://github.com/twitter/twemproxy

This article was co-authored with Senthil

Other Amazon ElastiCache Articles :
Caching architectures using Memcached & Amazon ElastiCache
Highly Available ElastiCache Redis replication cluster in AWS VPC
Deep dive into Amazon Elasticache
Distributed Cache On Steroids: Amazon ElastiCache
Launching Amazon ElastiCache in 3 Easy Steps
AWS Performance Tip 3: Use Amazon ElastiCache to improve application performance

ElastiCache Redis : How to Backup & Launch from RDB Snapshots ?

ElastiCache Redis is a powerful KV store offering both in-memory and persistent store for Key Values.When we persist data in ElastiCache Redis, it is obvious that we need to constantly backup Redis to recover the data during failures. By default there is no option to easily create the snapshot from the running ElastiCache Redis cluster. We need to install Redis on Self Managed EC2 node and point it to ElastiCache Redis Master for the snapshot process. This post explains about the step by step method for Redis cluster snapshot creation.

Redis Snapshot Architecture :

• ElastiCache cluster with master redis node of size cache.m1.large in AZ1 and Replication group with one read replica node of size cache.m1.large in AZ2
• Self managed Redis installed on EC2 for snapshots on AZ1

Stage 1: Creating a Snapshot for ElastiCache redis Persistent store.

Step 1) Launch an EC2 instance on the same Availability zone where the ElastiCache Redis master node runs. 
If the Redis Snapshot EC2 node is temporary (or) replication/Backup performance is very critical then keep the Redis Snapshot node in the same AZ as the ElastiCache Redis Master Node.  
If Redis Snapshot node is going to permanently run and high availability of the backup mechanism is very critical, run this on different AZ from master redis node. 

Step 2)Download and install the Redis software on your newly launched EC2 instance. Make sure the Redis snapshot ec2 version and ElastiCache Redis Master version are compatible.

$ wget http://redis.googlecode.com/files/redis-2.6.4.tar.gz $ tar xzf redis-2.6.4.tar.gz $ cd redis-2.6.4 $ make

Step 3)Start the Redis Server on the newly launched EC2 instance. It will be listening on the port 6379

# src/redis-server  [1513] 10 Jan 10:25:45.083 * Max number of open files set to 10032  [1513] 10 Jan 10:25:45.085 # Server started, Redis version 2.6.4

Step 4)Open another terminal window and connect locally to the Redis Snapshot EC2 instance 

# src/redis-cli -h localhost -p 6379  redis localhost:6379>

Step 5)To synchronize the data between the local Redis Snapshot EC2 instance and the ElastiCache Redis master use the following command in the Redis Snapshot EC2. 

SLAVEOF redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com 6379

where "redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com 6379" is the endpoint of the ElastiCache Master Redis node.

Bringing a Additional Redis EC2 node for snapshots into the infrastructure is not ideal. It adds extra cost and manual labor effort to setup, automate and manage this Redis Snapshot EC2. I hope AWS ElastiCache team will release a mechanism where we can easily take the snapshots from the ElastiCache Cluster itself. 

Note: Before this step ensure that security group permissions are opened between the Redis Snapshot EC2 and ElastiCache Redis Cluster.

Step 6)Once the synchronization is done between Redis Snapshot node and ElastiCache Redis Master, you can take snapshot using the command BGSAVE or SAVE in the snapshot EC2 Node. The snapshot file named "dump.rdb" is created in the disk.

Step 7) You can optionally detach it from the ElastiCache cluster by using the command on redis-cli  “SLAVEOF NO ONE”. This step is recommended if your redis cluster size is few GB's and you are backing up persistent data once/4 times a day, as you can save costs by not unnecessarily running your Redis snapshot EC2 instance for 24 hours and automatically bring it up whenever needed.

We conducted a small test to check whether all the above steps are successfully reflected. 

We pumped the ElastiCache Redis cluster with 1.8GB of Data spanning few hundred millions of KV data with each record in few <KB size. We observed the time taken to sync the data is about ~5 minutes on m1.large capacity in same AZ deployment. 

• If your Redis is heavily used it is better to use Higher EC2 capacities for the ElastiCache Cluster and Snapshot node. Bigger the nodes, better the NW/IO and lesser the replication lag. 
• If you are running a small cluster with few GB's then creating the Redis Snapshot Node on demand will save costs. If your cluster is big, data is critical and backup frequencies are less, then run Snapshot EC2 node continuously for better performance.

The replication action with timing is detailed below:

[1595] 10 Jan 11:03:06.676 # Server started, Redis version 2.6.4  [1595] 10 Jan 11:03:06.676 * The server is now ready to accept connections on port 6379  [1595] 10 Jan 11:03:18.018 * SLAVE OF redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com:6379 enabled (user request)  [1595] 10 Jan 11:03:18.779 * Connecting to MASTER...  [1595] 10 Jan 11:03:18.975 * MASTER <-> SLAVE sync started  [1595] 10 Jan 11:03:18.975 * Non blocking connect for SYNC fired the event.  [1595] 10 Jan 11:03:18.977 * Master replied to PING, replication can continue...  [1595] 10 Jan 11:06:03.949 * MASTER <-> SLAVE sync: receiving 1943998490 bytes from master  [1595] 10 Jan 11:08:13.348 * MASTER <-> SLAVE sync: Loading DB in memory  [1595] 10 Jan 11:09:37.430 * MASTER <-> SLAVE sync: Finished with success  [1595] 10 Jan 11:09:37.441 * 100 changes in 300 seconds. Saving...  [1595] 10 Jan 11:09:39.970 * Background saving started by pid 1635  [1595] 10 Jan 11:16:44.860 # User requested shutdown...  [1595] 10 Jan 11:16:44.860 * Saving the final RDB snapshot before exiting.  [1595] 10 Jan 11:19:10.270 * DB saved on disk  [1595] 10 Jan 11:19:10.270 # Redis is now ready to exit, bye bye...

Test whether the same key values are synced between Master Redis Node in ElastiCache Cluster and Redis Snapshot EC2. The below steps illustrate the same: 

# src/redis-cli -h redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com -p 6379  redis redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com:6379> get 100sent  "1001234567890123456789012345678901234567890abcdefghijklmopqrstuvwxyz" redis redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com:6379> get 10sent  "101234567890123456789012345678901234567890abcdefghijklmopqrstuvwxyz"

Important Observation:
Always sync the Redis Snapshot EC2 node with the ElastiCache Redis master node and not with the read slave in the redis replication group. Following error will be displayed when trying to sync with the slave. 

#Connecting to MASTER... #MASTER <-> SLAVE sync started #Non blocking connect for SYNC fired the event. #Master replied to PING, replication can continue... # MASTER aborted replication with an error: ERR Can't SYNC with a slave

We tried the above approach to check whether we can reduce the workload on Redis Master node and take the backups from Slave redis Nodes. As AWS as rightly named it is " Redis Read Replica Nodes" and you cannot use it for snapshot replication. 

Stage 2: Launching ElastiCache Redis Cluster from the existing Snapshot taken above.

Step 1) Import the data to the S3 bucket (redisaws) and assign open/download permission for the email id aws-scs-s3-readonly@amazon.com. Note : The aws-scs-s3-readonly account is used exclusively for customers uploading Redis snapshot data from Amazon S3.

Step 2) Launch a new ElastiCache Redis cluster from the "dump.rdb" snapshot taken in the earlier stage 1 - step 6 from Redis Snapshot node. Lets understand what is RDB in detail.

• RDB is a very compact single file point in time representation of your Redis data and are perfect for backups. Being a single compact file it can be transferred to alternate AZ and AWS regions quite fast and is ideal during outages/DR. RDB also allows faster restarts of Redis Master with big datasets in new ElastiCache Clusters nodes. Both these points helps us achieve better RPO and RTO during DR scenario using RDB. 
• RDB needs to fork() often in order to persist on disk using a child process. Fork() can be time consuming if the dataset is big, and may result in Redis Master to stop serving clients for some millisecond or even for one second if the dataset is very big and the CPU performance is not great. Also since the Redis Snapshot instance in this architecture is not used for accepting reads/writes and it is primarily used only for replication + snapshot it will add again minimal load on the master node during replication. Factor the ElastiCache Master Redis node capacity taking both these parameters into consideration.
• Note : RDB is NOT good if you need to minimize the chance of data loss in case Redis Master stops working. It is suggested to use RDB model with ElastiCache Redis Replication group cluster for better Availability and integrity.

Step 3) Once the ElastiCache cluster is launched. Connect to the cluster endpoint and check for the existence of the same key values tested in the dump.

src/redis-cli -h rds.qcdze2.0001.usw2.cache.amazonaws.com -p 6379  redis rds.qcdze2.0001.usw2.cache.amazonaws.com:6379> get 10sent  "101234567890123456789012345678901234567890abcdefghijklmopqrstuvwxyz"  redis rds.qcdze2.0001.usw2.cache.amazonaws.com:6379> get 100sent  "1001234567890123456789012345678901234567890abcdefghijklmopqrstuvwxyz"

You can notice from the above that the key values are same and the new ElastiCache node was created properly with original data.


This post was co authored with Senthil 8K Miles

Architecting Highly Available ElastiCache Redis replication cluster in AWS VPC

In this post lets explore how to architect and create a Highly Available + Scalable Redis Cache Cluster for your web application in AWS VPC. Following is the architecture in which the ElastiCache Redis Cluster is assembled:

• Redis Cache Cluster inside Amazon VPC for better control and security
• Master Redis Node 1 will be created in AZ-1 of US-West
• Redis Read Replica Node 2 will be created in AZ-2 of US-West
• Redis Read Replica Node 3 will be created in AZ-3 of US-West

You can position all the 3 Redis Nodes in different Availability zones for Achieving High Availability (or) you can position Master + RR 1 in AZ1 and RR 2 in AZ2. This reduces the Inter - AZ latency and might give better performance for heavily used clusters.
Step 1: Creating Cache Subnet groups:
To create Cache Subnet group  navigate to the dashboard of ElastiCache, select Cache Subnet groups and then click "Create Cache Subnet group". Add the Subnet Id and the Availability Zone you need to use for the ElastiCache cluster.

We have created Amazon VPC spreading across 3 availability zones. In this post we are going to place the Redis Master and 2 Redis Replica Slaves in these 3 availability zones. Since Redis will be most of the times accessed by your application tier it is better if you place them on Private Subnet of your VPC.
Step 2: Creating Redis Cache Cluster: 
To create Cache Cluster navigate to the  dashboard of ElastiCache, select Launch Cache Cluster and provide the necessary details. We are launching it inside Amazon VPC, so we have to select the Cache Subnet group .
Note: It is mandatory to create Cache Subnet group before Launch if you need ElastiCache Redis cluster in Amazon VPC.

For test purposes i have used m1.small EC2 instance for the Redis. Since this is a fresh Redis installation, i have not mentioned S3 bucket from where the persistent Redis Snapshot will be used as input. On successful creation of the Cache Cluster you can see the details in the dashboard.
Step 3: Replication Group Creation:
To create Replication group select the option of Replication Groups from dashboard and then select the “Create Replication Group”

Select the master Redis node "redisinsidevpc" created previously as the primary cluster id of the Cache cluster.  Give the Replication group id and description as illustrated below.

Note: Replication Group should be created only after the Primary Cache Cluster node is UP and running, else you will get the error as shown below.

On the successful creation of the Replication group you can see the following details. You can observe from below screenshot that there is only one primary node in US-WEST-2A and zero Redis Read Replica's are attached to it.

Step 4: Adding Read Replica Nodes:
When you select the Replication group, you can see the option to add Redis Read Replica. We are adding 2 Redis Read Replica named Redis-RR1 (in US-West-2B) and Redis-RR2 (in US-WEST-2C). Both the Read replica's are pointed to the master node "redisinsidevpc". Currently we can add up to 5 Read replica Nodes for a Redis Master Node. This is more than enough to handle Thousands of messages per second. If you combine it with Redis Pipeline handling 100K messages per second from a node is like cake walk.
Adding Read Replica 1 in Us-West -2B

Adding Read Replica 2 in US-West-2c

On successful creation you can see the following details of Replication group in the dashboard. Now you can see there are 3 Redis nodes listed with Number of read Replica's as 2. Placing the Read Replica's and master node in multiple AZ will increase the high availability and protects you from node and AZ level failure. On our sample tests inter AZ Replication deployments had <2 second replication lag for massive writes on master and <1 second replication lag between master slave inside same AZ deployments. We pumped @100K messages per second for few minutes on m1.large Redis instance cluster. 
In event, if you need additional read scalability i recommend to use more read Replica slaves added to the master. 
In your application tier you need to use the primary Endpoint "redis-replication.qcdze2.0001.usw2.cache.amazon.aws.com:6379" shown below to connect to Redis. 

If you need to delete/reboot/Modify you can make it through the options available here.

Step 5: Promoting the Read replica:

You can also promote any node as the Primary cluster using the Promote/Demote option. There will be only one Primary Node.
Note: This step is not part of the cluster creation process.

This promotion has to be carried out with caution and proper understanding for maintaining data consistency. 

Post was co authored with Senthil 8KMiles

Other related posts:

Billion Messages - Art of Architecting scalable ElastiCache Redis tier

Caching architectures using Memcached & Amazon ElastiCache
Deep dive into Amazon Elasticache
Distributed Cache On Steroids: Amazon ElastiCache
Launching Amazon ElastiCache in 3 Easy Steps
AWS Performance Tip 3: Use Amazon ElastiCache to improve application performance

AWS Performance Tip 3: Use Amazon ElastiCache to improve application performance

Web Applications can be often made to perform better and run faster by caching critical pieces of data in memory.  Frequently accessed data, layers of HTML fragments, results of time-consuming/expensive database queries, search results, sessions, results of complex calculations and processes are usually very good candidates for cache storage.  In general application architectures that are read intensive will usually have better performance gains using cache tiers. In web scale applications, distributed caching has become a mandatory part of the architecture stack to improve performance.

AWS introduced ElastiCache for this purpose. ElastiCache is fully compliant with MemCached and applications already using MemCached can easily migrate their code to Amazon ElastiCache. ElastiCache serves data from RAM, and execute all the simple operations (such as SET and GET) with O(1) complexity. Amazon ElastiCache large node can do 40k+ RPS throughput easily, this approach is far more performing than relying on database for everything.Refer the following benchmark from garantiadata to understand how memcached/elasticache can add performance to your architecture. http://garantiadata.com/blog/its-true-even-modest-datasets-can-enjoy-the-speediest-performance. 

Using Amazon ElastiCache distributed caching tier in your architecture, you reduce the overall read load in your database and increase the overall performance of your application. Request/response latency can be reduced to few milliseconds using Amazon ElastiCache in your architecture.Since Amazon ElastiCache is a volatile in nature and the items have TTL's it is recommended to fall back to DB or source data stores in event of cache miss.

To know more about Amazon ElastiCache please deep dive into this article here : http://harish11g.blogspot.in/2013/01/amazon-elasticache-memcached-internals_8.html

 

AWS Performance Tip 2: Use Page Cache-Web Accelerators to reduce Latency

Tip : Deliver your content from the edge of your architecture stack.
Imagine you have online application with Multi-tiered architecture comprising of Security Tier, Load Balancing Tier, Web tier, Application tier and Caching tier etc. Information has to travel through all these tiers to hit the caching layer and deliver the response back many times. This might take couple of hundred's of milliseconds sometimes depending upon the processing needs and network latency in the AWS infra . This overall latency might not be acceptable for certain use cases which needs sub second content delivery to the end users. 
Instead of delivering your frequently accessed contents from the deep caching tier, You can re-architect your stack a little and try to deliver content from entry point itself. Solutions like Varnish, aiCache and NetScaler specialize in this caching and acceleration techniques and are widely used in AWS infra by architects. AMIs of these products are available on the AWS marketplace and many customers have already put them on production use. These products have the capability to cache/compress pages, objects, URL etc and by placing them at the entry of your architecture you will reduce the latency, save precious back end compute cycles and save costs by reducing the instance capacity needed at the backend.  In case you use NetScaler Amazon EC2 it comes with sophisticated Load balancing algorithms as well, this way you can cut an extra LB tier and merge both cache+LB together to achieve high performance at low latency. Since these products are at the entry and have heavy dependency on bandwidth and memory, it is recommended to use EC2 instances with High IO and Memory capacity for these products. 
This architecture implementation is a useful technique and can be widely adopted on segments like online newspapers/news, blogs ,content management systems, online classifieds use cases etc in AWS, which has lots of static content that does not change quite frequently and can be delivered again and again from the page cache-web accelerators tiers itself. 

Related Articles :

Cost of Latency and Route53 LBR
Varnish Deployment Architectures in AWS

AWS Performance Tip 1: Use Amazon CloudFront to reduce latency

Amazon CloudFront is a content delivery web service that helps you to distribute content to end users with low latency and high data transfer speeds. Amazon CloudFront has growing network of edge locations that cache copies of popular files(images, JS, CSS, videos etc) close to your viewers and makes sure that end-user requests are served by the closest AWS edge location.Since requests now travel shorter distances to request objects (lesser hops) + the additional optimizations done in Amazon CloudFront like TCP initial congestion window, reduces the overall latency and improves the performance to your users.
Imagine your user is accessing a content hosted in AWS Europe from Singapore, it takes ~120 milliseconds as latency for this delivery, whereas if the same content is delivered from AWS Singapore using Amazon CloudFront, the latency can be reduced to ~10 milliseconds. If your site is content heavy multiply the above with number of requests, hops etc and imagine the latency savings that Amazon CloudFront can provide. 

Internally Amazon CloudFront maintains persistent connections with the content origins so that changes/addition can be picked quickly. 

Following diagram illustrates current list of edge locations offered by Amazon Web Services. Keep watching  http://aws.amazon.com/cloudfront/#details for updates, more edge locations around the world are on your way.

Apache Solr 4 Cook Book Review

Book Review : Apache Solr 4 Cookbook Review
Author: Rafal Kuc
Reviewed by : Harish Ganesan and Vijay Olety
Book URL : http://www.packtpub.com/apache-solr-4-cookbook/book

Comments : 

I recently read this book and i am really impressed! This book provides good understanding of Apache Solr for both developers as well as consultants.

The book starts off well with an introduction to Apache Solr, the web / app servers required, the role of Zookeeper, why clustering your data is vital ?, the various directory implementations, performance-oriented caching mechanisms, a sample crawler module which coupled with Solr gives a complete end-to-end solution, the role of Apache Tika as a extracting toolkit and the ease of customizing Solr. From then on, the book dwells into the details.

The first step is indexing. It plays a vital part in the entire search solution. Data can be in the form on .txt, .pdf or any other format. It is imperative that all such formats are easily indexable. One of the widely used tools for extracting metadata and language detection is Apache Tika. Data can also be present in a database, for which the Data Import Handler is handy. It comes in two variants – full and delta. Every detail is nicely explained with examples which can make the development time faster. DIH also helps us to modify the data while importing which I felt is a pretty neat feature! One of the nicest features included in Solr 4 is the ability to update single field in a document. I am not sure why this was included in the earlier versions but it’s a classic case of better late than never.

The next step in the pipeline is the data analysis which is achieved through the use of analyzers and tokenizers. Various use cases include elimination HTML and XML tags, copying the contents of one field to another and stemming words amongst others. The detailing that has gone into explaining every concept, the examples and the associated step-by-step explanation is really helpful.

Now that the data is indexed and the data preparation is completed, it’s time to query Apache Solr! Searches can be performed on individual words or on a phrase. You can boost or elevate certain documents over others based on your requirements. Simple concepts such as sorting and faceting of results to complex ones such as ignoring typos using n-grams and detecting duplicates are very simple to understand and perform. Faceting, in particular, is gaining momentum as it helps in implementing the auto-suggest feature and narrowing down the search criteria. A newly introduced feature called the pivot faceting was a much needed one and it vastly simplifies certain use cases related to faceting. Solr provides immense capabilities when it comes to querying and this book explains each of them in great detail taking real-world examples.

We indexed and queried the data. But as our application scales, we have to get our hands dirty and start fine-tuning the performance metrics in order to give a good user experience to our customers. This is where caches and its various flavors and granularities starts to make sense. Cache always plays a major role in any deployment and it is necessary to monitor Solr at all times to gauge its performance. This book can done a great job in clearly explaining the various types of caches, the commit operation and its impact on searchers and how to overcome these. This topic is really important for any Solr real-world deployment and this book has not let me down!

Apache Solr 4.0 introduced the most-awaited SolrCloud feature that allows us to use distributed indexing and searching. Setting up of SolrCloud cluster along with a Zookeeper ensemble to enable replication, fault-tolerance and high availability along with disaster recovery is a piece-of-a-cake now. I really appreciate the time and effort spent on documenting and explaining how to set up two collections inside a single cluster. It was a nightmare to find information on this particular topic when we implementing SolrCloud for one of our customers. But I am rest assured that others referring this book will save precious time of theirs. Adding / deleting nodes from a cluster is no longer a tedious task as the entire process is automated through the presence of Zookeeper nodes. The in-depth knowledge of the author in these topics is clearly visible and is of great help to all the readers. A touch on Zookeeper Rolling Restart, though off-topic, might enable readers to get a complete birds-eye view of the entire cluster. Certain features such as soft commit and NRT search have been explained in detail afterwards (under Real-life Situations) but I felt that at least a mention earlier on would have provided a much needed continuity in that section. For the geeky readers like me, a detailed description about load balancing across shards and replicas and their customizations, if any, would have added an extra amount of spice to this well-cooked food!

As with any other tool, Solr deployment too will run into some kind of a problem. This section details the common problems that are encountered and effective ways to overcome these. Shrinking the size of the index and allocating enough memory in advance amongst others are some of the solutions explained in detail and is clearly documented in this book.

Lastly, as every developer would have wanted it, the real-world scenarios are described and the various Solr concepts that were explained in earlier sections are put together as part of a complete end-to-end solution.

Any one trying Solr 4.0 must read this book in its entirety before recommending a Solr production architecture. As mentioned above, there are a few suggestions which if incorporated in this book would benefit readers. All in all, this book will be really helpful for the developers and consultants alike!