AWS Cost Saving Tip 17: Add Amazon RDS Read Replica's on need basis and save costs

Many online applications are usually Read intensive in nature. Though the usage of Cache, NoSQL, CDN are becoming part of the core architecture in many such applications, still most of them are heavily dependent upon Traditional RDBMS (like MySQL, Oracle) for serving their requests.
Architects usually design their DB tier with Master + Read Slaves model to cater the read intensive needs of the application. Writes and Immediate Write + Reads are directed to Master and all other Read queries are balanced between Read Slaves. Translating this model to AWS infrastructure means, using RDS MySQL Master + Multiple Read Replica Slaves.

To know more about Load balancing Amazon RDS Read Replica's Click here

Scenario: Weekly load volatility impacts: 

Online Ticketing and Travel companies are usually prone to weekly fluctuations in their load pattern. Imagine during peak load they need five M2.4XLarge RDS MySQL Read Replica's to serve their read intensive site.

As an Infra Architect you have two options in the Elastic AWS world: 

Option 1: Run your DB tier with peak capacity all days of the week. ( May be suitable for Traditional infra environments, but a bad idea in AWS infra)

Option 2: Elastically increase the number of RDS Read Replica instances during the peak period and decrease the number of instances once the peak period is over. (AWS is built for leveraging such cases)

Number of EC2 Instances	Number of days in a week	EC2 Instance Type	Weekly($)	Monthly($)
5	7	M2.4XLarge	1860	7442
			1860	7442
2	4	M2.4XLarge	425	1701
5	3	M2.4XLarge	797	3189
			1222	4890

Cost for Option 1: Running all 5 of them for a Week costs : $1860
Cost for Option 2: Running with 2 RR normally for 4 days a week and elastically increasing to Five M2.4Xlarge during the peak days(i.e. three days a week) will cost : $1222

Savings: From the above analysis you can observe that elastically increasing/Decreasing Amazon RDS RR based on your load you can achieve ~30 % cost savings in a week for the mentioned use case.

To know more about "How Amazon Auto Scaling can save costs on your Web/App Tier ? Click here "

Scenario: Impact during Campaigns 

Imagine your online business demands big campaigns once every quarter. The campaign usually runs for 2 weeks and traffic increases 4-5X during these times. During normal period you run 1 RDS Read replica and during campaigns you need 5 RDS Read replicas. 

If the campaign is planned once a year during holiday season, companies lease hardware, but if the campaign is every quarter or more frequent it becomes irrelevant to lease-build-maintain peak capacity.

Number of EC2 Instances	Number of Months	EC2 Instance Type		Monthly($)
5	10	M2.4XLarge	All times	98877
				98877
1	10	M2.4XLarge	Other times	16479
5	2*	M2.4XLarge	During campaigns	16479
				32959

Instance Hours per day: 744

*Campaign weeks are aggregated to months

US-Oregon region - $2.215 (M2.4Xlarge RDS RR pricing as on posting)

Cost for Option 1: Running all 5 of them for a year costs :  98877 USD

Cost for Option 2: Running with 1 RR on regular weeks and elastically increasing to Five M2.4Xlarge during the peak weeks will cost : 32959 USD a year

Savings: From the above analysis you can observe that elastically increasing/Decreasing Amazon RDS RR you can achieve ~55 % cost savings in a year.

For in depth analysis on Amazon RDS Second tier Read Replica's Click here 

Other Cost Saving Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs 
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage 

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage 

Cost Saving Tip 16: Periodically remove your unwanted AWS resources and save costs (new)

Exploring Amazon RDS MySQL Second Tier Read Replica

AWS recently introduced Second Tier Replica for RDS MySQL.You can use this feature to shift the load from primary master DB to the replica in first tier and prepare your application for handling extremely high level of read traffic.  In this post we are going to explore this feature in following aspects :

• Steps to configure Multi-Tiered Amazon RDS MySQL Read replicas
• Second Tier Read Replica Deployment architectures
• Important Points to note

Amazon RDS MySQL version 5.6.12 on AWS WEST region was used. 

Steps to configure Multi- Tiered Amazon RDS MySQL Read Replicas

Configuration steps for the following architecture is given in this post :

Step 1: Creating Read Replica from Master and Place at Tier 1:

To create RDS MySQL Read replica navigate to the dashboard of Amazon RDS, select the Amazon RDS MySQL Master(named "sampledb") and use the option of “Create Read Replica”. 

Name the newly created Amazon RDS Read replica as "sampledb-level1" and place it in Tier 1. The Tier 1 Amazon RDS MySQL read Replica can be created in same AZ of Master or in a different AZ for High Availability. When the Tier replica is placed in Different AZ , you should factor few extra milliseconds of latency during replication.

Explore the status from Master DB: Post successful creation of the Tier 1 Read Replica, You can see the Read Replica Id's from exploring the details of the Master DB. Illustrated in Below Screen shot :

Explore the status from Tier 1 Read Replica : When you explore the Tier 1 Read Replica details, you will find it is pointing to the to Master DB. Illustrated in Below Screen Shot :

Step 2: Creating Second Tier Read Replica from Tier 1 Read replica :
To create Second Tier Read replica navigate to the dashboard of Amazon RDS, select the Amazon RDS Tier 1 Read Replica (sampledb-leve1) as the source and use the option of “Create Read Replica”.

Name the newly created Amazon RDS Read replica as "sampledb-level2" and place it in Second Tier 2. The Second Tier -2 Amazon RDS MySQL read Replica can be created in same AZ of Tier 1/Master or in a different AZ for High Availability. 

Explore the status from Tier 2 Read Replica : When you explore the Tier 2 Read Replica details, you will find it is pointing to the to Tier 1 - sampledb-level1 as replication source. Illustrated in Below Screen Shot :

To Load Balance AWS RDS Read Replica's Refer this article : http://harish11g.blogspot.com/2013/08/Load-balancing-Amazon-RDS-MySQL-read-replica-slaves-using-HAProxy.html

Second Tier Read Replica Deployment Architectures

One of the main complexity behind Multi-Level replication is that, if Tier 1 Read Replica if not properly architected/placed, it can be a single point of failure. Imagine a case where you need 4 Read Replicas for your Master DB. You can take following approaches as illustrated below while designing your infrastructure for this requirement.

In Architecture-1, Tier-1 Read replica is a Single Point of Failure. Instead if you split Tier-1 itself into two separate fleets it offers better availability than architecture-1. Since both Tier-1 RR put replication load on Master DB , you can by pass this using Hot Standby instance. Benefits of this approach is explained below in the best practice architecture. 

Best Practice High Availability Architecture for Second Tier Read Replica:

Condition 1 - Master DB Failure : Hot Standby becomes new master and Tier-1 Read Replicas points automatically to new source
Condition 2- Tier-1 Read Replica Failure : Entire Tier-1 and Associated Tier-2 has to be recreated. The Alternate Active Fleet of Tier-1+Tier-2 will serve the requests for high availability.
Condition 3 - Tier-2 Read Replica Failure : Only the non performing Tier-2 Read Replica instance has to be recreated
Condition 4- AZ NW problem: In event AZ-3 is failed, Requests are served by alternate fleet of Tier-1+Tier-2 in another AZ(AZ-4).

Multi-AZ Hot Standby Implementation is a recommended best practice when it comes to Multi-tiered Read replica implementation

• It is recommended to create Tier 1 Read Replicas from Multi-AZ DB instance to offload read queries from the source master DB instance for high traffic sites. If the source instance of a Multi-AZ deployment fails over to the secondary, any associated read replicas will be switched to use the secondary as their replication source automatically. This model guarantees high availability.
• Also when you initiate the creation of a Tier 1 read replica, Amazon RDS takes the DB Snapshot of your Standby DB instance (instead of Source DB) and begins replication. This model saves I/O suspension on your source DB during the snapshot process 

Other Points to Note:

Note : 
To Load Balance AWS RDS Read Replica's Refer this article : http://harish11g.blogspot.com/2013/08/Load-balancing-Amazon-RDS-MySQL-read-replica-slaves-using-HAProxy.html

P1) Circular replication are not allowed in this tiered replica creation process. 
P2)Third Tier cannot be created. Actually in production, very rarely we need third tier and it is not important feature
P3) When the "X" Tier replica is placed in Different AZ , you should factor few extra milliseconds of latency during replication.
P4) Before a DB instance (Master or Tier 1) can serve as a replication source, you must enable automatic backups on the source DB instance by setting the backup retention period to a value other than 0. This requirement does not apply to second tier replica as they are not source DB instance for another read replica.So create “Read Replica” option will appear only when the Backup Retention is set to minimum of 1 day for any level Read Replica creation. When we "create Read Replica" by default RR is created with Backup Retention set to 0. Use the modify option Illustrated in below screenshot and set the  Backup Retention Period.  :

P5) We have used t1.micro as RDS DB for article explanatory purpose. For production use cases please use proper instance types after proper capacity planning.
P6) AWS has released Parallel replica creation process, where you can create multiple Tier 1 and Tier 2 replicas in parallel. Since you no longer need to wait for one replica creation before starting the next one, it becomes easy to create multiple RR quickly. Without this feature, it would take hours to create a large Multi-tiered Replica setup.
P7) If a replica lags too far behind for your environment, the normal practice is to scale up or consider deleting and recreating the read replica. Imagine you have architecture where there are two Tier 1 and three Second Tier RR as illustrated in below architecture:

Now since Tier -1 RR is lagging , you are planning to delete and recreate the same. When you delete the Tier-1 RR, All the Second Tier Read Replica's will now become Standalone , Single AZ DB's. Once you have recreated the Tier -1 RR again and managed to retain the original end point as well, you cannot re-point these Second Tier RR to Tier -1 again. You need to recreate all 3 second tier from the Tier -1 again. For a application with heavy DB dependency and read traffic, it means 1/2 of the fleet is now down. This can lead to uncomfortable performance situation. This is not a ideal condition and AWS RDS team can take this in their Road Map. For such cases, it is better to create a new Tier -1 and Second tier fleet first , update your LB/App configs and then delete the old lagging fleet 
P8)Why it is better to have Multi-tiered Read Replica as the last resort in your architecture ?
8.a)Currently 30 Read Replica can be created overall for a master in two tiers. 30 Read replica is more than sufficient and usually turns out to be costly architecture approach. Use this approach only for cases, which demand heavy read and when application code cannot accommodate changes and are highly DB dependent.
8.b)It is recommended in DB world to stay away from Multi-level replication as much possible. Your architecture will be much simpler with one master and "X" replica slaves, rather than having tiered replica's. As we observed in above deployment architectures, the second tier replica slave will be a trouble to manage in event of replication delay, crashes and network problems affecting the Tier-1 RR or the Master DB. 
8.c)In case your application code can be redesigned, it is recommended to take following approaches before resorting to Second Tier replicas architecture

• Functional partition the RDS MySQL with Hot Stand By and Read Replica's
• Re-balance the DB load by using alternate data stores provided by AWS like DynamoDB, ElastiCache, CloudSearch etc.

If the above methods does not work for you, take the Second tier Read replica approach.

Thanks Senthil and Ram of 8KMiles for taking part in this analysis.

Configuring PHPMYADMIN on Amazon EC2 to manage Amazon RDS MySQL

This post is syndicated from Sanket's Blog :http://sanketdangi.com/post/56623052533/phpmyadmin-on-amazon-ec2-manage-amazon-rds

The installation procedure of PHPMYADMIN is very simple and there are numerous blogs available across internet which will help you to configure phpmyadmin.

Even, I followed same blogs/articles to configure phpmyadmin on Amazon EC2. But things didn’t clicked for me instantly. I figured out the solution for some of the problems and thought of writing a post on this.

There are two ways to configure phpmyadmin :

1. Download phpmyadmin package and install that.

2. Installing through repositories by enabling epel

I followed 2nd procedure.

Installation Steps :

1. To install phpmyadmin package :

# sudo yum --enablerepo=epel install phpmyadmin

2. Navigate to file "/etc/httpd/conf.d/phpMyAdmin.conf" and make sure you can access phpMyAdmin from outside (your local pc).

<Directory /usr/share/phpMyAdmin/>
   <IfModule mod_authz_core.c>
     # Apache 2.4
     <RequireAny>
#       Require ip 127.0.0.1
        Require ip <Add-your-IP>
       Require ip ::1
     </RequireAny>
   </IfModule>
   <IfModule !mod_authz_core.c>
     # Apache 2.2
     Order Deny,Allow
     Deny from All
#     Allow from 127.0.0.1
        Allow from <Add-your-IP>
     Allow from ::1
   </IfModule>
</Directory>

3. Edit mysql host in config.inc.php file

In order to access RDS database, we need to modify mysql host in config.inc.php file. Most of the blogs across web points to /usr/share/phpMyAdmin folder and copying config.sample.inc.php as config.inc.php. But modifying mysql host in this file didn’t work for me.

In order to get that working, I had to modify config.inc.php in /etc/phpMyAdmin.

# sudo vi /etc/phpMyAdmin/config.inc.php

Modify

$cfg['Servers'][$i]['host'] = 'localhost';

to

$cfg['Servers'][$i]['host'] = 'xxxxxxxxxxx.cny3hl5jk7fs.us-east-1.rds.amazonaws.com'; //Your-RDS-Instance-Endpoint

4. Perform httpd restart

# sudo yum install httpd

5. Browse PHPMYADMIN on your browser

http://<Elastic-IP-Address>/phpmyadmin

If you face below mentioned error :

#2002 - Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)

The server is not responding (or the local server’s socket is not correctly configured).

Then most probably, you haven’t modified mysql host in config.inc.php present in/etc/phpMyAdmin. Modifying mysql host in config.inc.php present in/usr/share/phpMyAdmin may not work for you.

Part 15: Amazon EBS Snapshot copying- Performance, Tips and How to

Most of us know that EBS snapshots have Amazon EC2 regional scope. In case you are planning to geographically expand or have data backed up on different Amazon EC2 region at regular intervals to minimize data loss and recovery time, you can use EBS snapshot copy feature for this purpose.
The copied EBS snapshot behaves the same like regular snapshots, i.e. it can be used to create new EBS volumes which can then be attached to an EC2 instance in the destination region. Since the data is moved across regions in the form of snapshots, you will be charged only for the data transferred to copy the snapshot and to store the copied snapshot at the destination region.
Following table illustrates the time it takes to copy EBS snapshots from Different Source and Destination Amazon EC2 regions. Data used was a 50 GB compressed file stored on an Amazon EBS volume which was taken as a snapshot and transferred using EBS snapshot copy process. Since AWS keeps improving their infrastructure every day, the time taken may vary in coming months/years. But the figures mentioned below can used as an reference to plan your Recovery time objectives and migrations:

EBS Snapshot Copy vs Amazon EC2 Regions matrix :

Source Region	Destination region	File Size/Type/EBS Volume Size	Time Taken
USA-East	Singapore	50 GB Compressed file in EBS Volume	1 hr 24 min
USA-East	US-West (Oregon)	50 GB Compressed file in EBS Volume	24 min
USA-East	Europe (Ireland)	50 GB Compressed file in EBS Volume	45 min

Points to note about Amazon EBS snapshot copy :

• Unlike the create snapshot operation, which is incremental in nature, the EBS copy snapshot operation copies all of the bytes in the snapshot every time. It is recommended to use EBS snapshot copy once a day for larger databases with Multi region DR to reduce data transfer cost . Only if your DB is not a large one and you need better RTO/RPO + your IT budget can afford it, you should reduce this frequency accordingly.
• EBS Snapshots now support incremental copies. The first time you copy an EBS snapshot of a volume to a particular Region, all of the data will be copied.  The second and subsequent copies of snapshots from the same volume to the same destination region will be incremental: only the data that has changed since the last copy will be transferred. As a result, the snapshot will transfer less data and complete more quickly than before. It is recommended to use EBS snapshot copy across regions frequently if you need better RTO/RPO for DR.  . To know more about how EBS Snapshots works ? click here
• Currently RDS Snapshot copy is not supported. In case you need to move data to different Amazon Region you will have to manage it through RDS dumps or implement your own programmatic replication in EC2+RDS, till AWS RDS team introduces this feature. Customers who need their database to be backed up on different region on AWS, adopt some of the following strategies, let us take MySQL as the DB and explore some techniques:
• Configure MySQL on Amazon EC2 with Master-Slave mode. Freeze the file system and take regular snapshots in the source region. Copy these snapshots once or twice a day depending upon the RTO/RPO and DR budget using EBS Snapshot copy feature.
• Configure MySQL on Amazon EC2 with Master-Slave mode. Have another Slave running on Destination region which asynchronously updates records from the Master in source region. Secure the entire transport using SSL or VPN between regions.

How to do Amazon EBS Snapshot Copy between Regions ?


In this article we are copying a Snapshot from US-EAST Virginia Region to Asia Pacific Singapore Region.
EBS Snapshot Copy is done using the AWS Management Console:
1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/
2. In the Navigation pane click Snapshots under the Amazon Elastic Block Store menu

3. In the EBS Snapshot page select the Snapshot ID you need to copy to destination region and Click Copy.

4. In the Copy snapshot dialog box you need to fill the following details.
Destination Region: The region where you want to copy the snapshot. In this case it is Asia Pacific (Singapore)
Description: Description of the snapshot. By default you can retain the one as shown in the below figure.

5. Click Yes, Copy button and the following dialog screen will be shown as confirmation.

6. To check the progress of the snapshot copy go to the Destination region (Asia Pacific Singapore in this case) and check the progress in the EBS snapshot page.

EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store
Part 2: Understanding Standard EBS Volumes
Part 3: Understanding EBS PIOPS Volumes
Part 4: Understanding EBS-Optimized Instances
Part 5: Understanding Latency in EBS
Part 7: 10% of your provisioned IOPS 99.9% of the time
Part 8: Performance Tuning - Pre Warming the EBS volume
Part 9: Performance Tuning - EBS Striping
Part 10: Performance Tuning - IO Block Size
Part 11: Understanding Amazon EBS Snapshots
Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud (new)
Part 13: Amazon EBS Security Best practices and tips
Part 14: How proper EBS Snapshot Retention strategy can save costs (new)
Part 15: Amazon EBS Snapshot copying- How-to, Performance and Tips (new)

Part 10: Performance Tuning Amazon Elastic Block Store - IO Block Size

The IOPS rate you get usually depends on the I/O size of your applications’ reads and writes. It is very important to know the I/O size your application operates and accordingly expect the performance from EBS Volumes. Provisioned IOPS volumes process your applications’ reads and writes in I/O sizes of 16KB blocks or less. Every increase in I/O size above 16KB will linearly increase the resources you need to achieve the same IOPS rate. For example, if you have provisioned a volume with 2000 IOPS, that means that it can handle 2000 16KB writes per second, or 1000 32KB writes per second, or 500 64KB writes per second, and so on. You can use Amazon CloudWatch to monitor your EBS Volumes throughput and I/O size for a while and decide what best suits for your case. 

Since PIOPS is especially designed for Database context, let us explore how it pans out in Amazon RDS. 

Following are some of the important points to note on PIOPS + RDS context.

P1) The theoretical maximum IOPS rate is also a function of database I/O page size and available channel bandwidth. Each page read or write constitutes one I/O operation. Database operations that read or write more than a single page will use multiple I/O operations for each database operation. For example, for a page size of 16 KB, if a read operation requires 32 KB, you would need 2000 IOPS to achieve 1000 read operations per second.

P2) If you provision an IOPS rate that is higher than the maximum or that is higher than your realized IOPS rate, you may still benefit from reduced latency and improvements in overall throughput.

Applicability of the following points and IOPS rates are based on the m2.4xlarge instance class with full duplex (1000 Mbit/second) and a workload that is perfectly balanced between reads and writes.

·         MySQL uses a page size of 16 KB and can do 12,500 IOPS. On a DB instance with a full duplex I/O channel bandwidth of 1000 megabits per second (Mbps), the maximum IOPS for page I/O is about 6,250 IOPS in each direction for 16 KB I/O. This is for a workload consisting of 50% reads and 50% writes and could reach 12,500 IOPS with 16 KB I/O.

·         Oracle uses a page size of 8KB and can do 25,000 IOPS. On a DB instance with a full duplex I/O channel bandwidth of 1000 megabits per second (Mbps), the maximum IOPS for page I/O is about 12,500 IOPS in each direction for 8 KB I/O. This is for a workload consisting of 50% reads and 50% writes and could reach 25,000 IOPS with 8 KB I/O.

·         SQL Server uses a page size of 8KB and can do 10,000 IOPS. On a DB instance with a full duplex I/O channel bandwidth of 1000 megabits per second (Mbps), the maximum IOPS for page I/O is about 5,000 IOPS in each direction for 8 KB I/O. This is for a workload consisting of 50% reads and 50% writes and could reach 10,000 IOPS with 8 KB I/O.

Note : Please refer the AWS Documentation for latest PIOPS 

EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store

Part 2: Understanding Standard EBS Volumes

Part 3: Understanding EBS PIOPS Volumes

Part 4: Understanding EBS-Optimized Instances

Part 5: Understanding Latency in EBS

Part 7: 10% of your provisioned IOPS 99.9% of the time

Part 8: Performance Tuning - Pre Warming the EBS volume

Part 9: Performance Tuning - EBS Striping

Part 10: Performance Tuning - IO Block Size

Part 11: Understanding Amazon EBS Snapshots

Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 

Part 13: Amazon EBS Security Best practices and tips