AWS Cost Saving Tip 9: Have EBS Snapshots Deletion- Retention strategy in place

Amazon EBS snapshots are incremental backups, meaning that every snapshot only copies the blocks in the volume that were changed since the last snapshot.  The TOC and only changed blocks are copied (in compressed form) to the S3 in subsequent snapshots. If you have a volume with 10 GB of data, but only 2 GB of data have changed since your last snapshot, only the 2 GB of modified data is written to Amazon S3 during the snapshot process. When the blocks are copied they are compressed before getting stored in Amazon S3.

To know more about How Amazon EBS Snapshots work ? Refer   http://harish11g.blogspot.in/2013/04/understanding-Amazon-Elastic-block-store-EBS-snapshots.html

It is usually very complex to predict the cost of storage associated with EBS snapshots though some use perfmon, Disk Monitor, IoTop etc to guess as close a possible. 

In this article based on my experience have suggested some patterns to understand EBS snapshot costing in detail and how to avoid leakages and save money while using them.

Imagine you have 1.5 TB of EBS volumes of which ~ 1TB is occupied.  Let us explore the patterns based on above assumptions:

Read only DB: Though practically you will not find databases that are completely read-only for understanding purposes let us imagine there is one. Whenever you take a snapshot from this database after first full snapshot copy (~1 TB), it won’t occupy much storage space for subsequent snapshots because there is literally no change in this database, so no changed blocks and hence no copy to Amazon S3 also occurs. Imagine you have and automated Snapshot mechanism every day for this database, you can go ahead and delete all the old copies of the snapshot regularly and retain only the latest copy.  Even if you miss to delete the old copies you will not end up much with leakage in cost in this case, because you will be paying for the 1 TB snapshot (first time) and not much for the subsequent snapshot storage.

Normal Read-Write DB:  

Read-Write ratio (90:10) is a normal pattern that can be observed in many Databases. Imagine you have a ~1 TB EBS (used) and everyday there is 7-10% data change on the volume, assuming  efficient compression in place and close to ~30GB changed data is copied to S3. Assuming the first full snapshot process will take ~1 TB snapshot storage space in S3 and then every day the incremental will be ~30 GB to Amazon S3. If retention period is 30 days for the snapshots, then 900 GB of snapshot storage will be additionally added in S3 totaling to ~1.9 TB in 30 days.  

In case the IT team does not have a mechanism in place to delete the snapshots regularly, then over a year they would have aggregated following cost leakage:

1024 GB (full snapshot) + 11824 GB (Aggregated @900 GB a month) = 11824 GB of snapshot storage consumed. This equates to ~7933 USD in cost @0.095 $ per GB of snapshot storage cost in Amazon S3.

Is there a better way to cut this leakage and reduce the cost in Amazon S3? Yes, since snapshots are incremental in nature, the old snapshots can be deleted anytime. The IT team just needs to have proper mechanism in place to manually or automatically delete the older snapshots. Depending upon the application characteristics, they should have proper retention periods (with deletion) and snapshot version maintenance strategy in place. This way they can efficiently manage the snapshot storage and reduce the cost leakage. Imagine the same IT team has a 30 days retention strategy and mechanism in place for deleting the snapshots; now let us revisit the costs:

1024 GB (full snapshot) + 900 GB (maintained @900 GB a month) = 1924 GB of snapshot storage consumed. This equates to ~2290 USD in cost @0.095 $ per GB of snapshot storage cost in Amazon S3. This snapshot retention/deletion process in place easily translates to 70% reduction in cost leakage and savings.  Refer the below table for cost comparison and savings

 

More savings can be achieved in some use cases if the retention periods are much more compact. If your application has more write ratio, then efficient snapshot deletion strategy in place can help you save more costs.

To know more about Amazon EBS in detail ? Refer   http://harish11g.blogspot.in/2013/04/Understanding-Amazon-Elastic-block-store-intro.html

Frequency of snapshots depends purely on the RTO/RPO of your DB. Some of the common patterns I have observed are 5-10 minutes, 1 hour and 1 day for Snapshots. You need to have file system like XFS to freeze while taking snapshots for consistency. Take snapshots from the Slaves and incase if you take it once a day, Schedule it when least activity is there in your DB.    


Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place
            

Part 13: Amazon EBS Security Practices & Tips

Security Practice 1: IAM your EBS Volumes

Amazon EBS volume access can be restricted to the AWS Account that created the volume and only to the users under the AWS Account with AWS IAM controls and policies. For Example : You can use AWS IAM and grant access to an user with EBS operations and deny all other AWS Accounts/users, the permission to view or access that EBS volume. IAM is a very powerful and emerging feature of AWS and should be adopted wherever appropriate controls are needed.

To understand more about Amazon EBS refer article : http://harish11g.blogspot.in/2013/04/Understanding-Amazon-Elastic-block-store-intro.html

Security Practice 2: Wiping Data from Amazon EBS Volumes: 

Amazon EBS volumes will be presented to you as raw unformatted block devices that had been wiped prior to being made available for your use. But some businesses have stringent data security policies and follow much more sophisticated guidelines for Media sanitization. For such cases, Amazon EBS provides you the ability to wipe the data in a specific method as well. For example : You can follow DoD 5220.22-M (“National Industrial Security Program Operating Manual “) or NIST 800-88 methodology for wiping the data in EBS Volumes.  You should basically conduct a specialized wipe procedure prior to deleting the volume for compliance with your established requirements.

Security Practice 3: Sharing Amazon EBS snaphsots

We all know that, Amazon EBS volume snapshots can be made publicly available to other AWS Accounts as well.  Though Amazon EBS volume snapshots can be shared , it does not provide other AWS Accounts with the permission to alter or delete the original snapshot. The right is explicitly reserved for the AWS Account that created the volume as part of the security controls. Since Amazon EBS snapshot is a block-level view of an entire EBS volume , sometimes data that is not visible in the file system on EBS Volume, may be present in the EBS snapshot. 

To know more about how EBS snapshots work, refer article:   http://harish11g.blogspot.in/2013/04/understanding-Amazon-Elastic-block-store-EBS-snapshots.html

If you want to create shared snapshots, you should ensure that sensitive data or files had been deleted in EBS volume properly. 

Security Practice 4: Storing AWS Credentials on an EBS Snapshot Securely

This is an important security practice to be carried in combination with Amazon IAM in EBS snapshots. The following article written by shlomo swidler illustrates the same in detail :

http://shlomoswidler.com/2010/07/storing-aws-credentials-on-an-ebs-snapshot-securely.html

EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store
Part 2: Understanding Standard EBS Volumes
Part 3: Understanding EBS PIOPS Volumes
Part 4: Understanding EBS-Optimized Instances
Part 5: Understanding Latency in EBS
Part 7: 10% of your provisioned IOPS 99.9% of the time
Part 8: Performance Tuning - Pre Warming the EBS volume
Part 9: Performance Tuning - EBS Striping
Part 10: Performance Tuning - IO Block Size
Part 11: Understanding Amazon EBS Snapshots
Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 
Part 13: Amazon EBS Security Best practices and tips

Part 11: Understanding Amazon Elastic Block Store Snapshots

We understand that EBS volumes have redundancy built-in, which means that they will not fail if an individual drive fails. But their redundancy is limited to Availability Zone scope. EBS does not replicate data automatically across multiple availability zones like other AWS services (S3, DynamoDB, RDS etc).

The durability of EBS is illustrated as follows by AWS in their site

“The durability of your EBS volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives. “

Technically we can overcome this by mirroring EBS Volumes but still it will not solve if there is a failure at AZ level. This constraint strongly suggests that for safe guarding your data you need to take backups and store them in multiple availability zones. Some of the common challenges involved in the backup process include the time it takes to create data copies, the disk space required, the impact on server operations during the copy process.  New generation Storage arrays have the ability to speed up dramatically the backup process by using a technique called as “Snapshot”.

A snapshot is the state of a system (like LUN-level copies of data) at a particular point in time. One of the most common types is Differential snapshots -> which allows for fast creation and reduced disk space consumption. Some common implementations of differential snapshots include copy-on-write or allocate-on-write; Better implementations of these techniques create copies instantly, allow the copies to be used read-write, permit many copies to co-exist and be active at the same time etc

Amazon EBS snapshots are incremental backups, meaning that every snapshot only copies the blocks in the volume that were changed since the last snapshot.  The TOC and only changed blocks are copied (in compressed form) to the S3 in subsequent snapshots. If you have a volume with 10 GB of data, but only 2 GB of data have changed since your last snapshot, only the 2 GB of modified data is written to Amazon S3 during the snapshot process. 

AWS does not disclose the internal of their snapshot technology but based on our understanding with storage systems let us explore how it works:

Step 1) when you take snapshot of an EBS volume for the first time, it is a full snapshot, but it only copies the blocks in the EBS volume that contains data.  During the first snapshot, the full TOC and all blocks containing data (A, B, C, D, and E) is moved asynchronously to S3.

Step 2) Imagine in meantime, blocks D and E were changed and F is newly added from the snapshot 1. When you take snapshot 2, this time the TOC and only the changed blocks D1, E1 and F are moved to S3.

Step 3) when you take snapshot 3, blocks E and F are changed and G is newly added as per diagram. This time the TOC and only the changed blocks E2, F1 and G are moved to S3.

Step 4) since snapshot 3 is the recent and contains the latest data, you can go ahead and delete older snapshots like 1 and 2.  The capacity occupied by blocks like D, E, F, E1 are no more relevant, and they are released and not charged by AWS.

You can observe that the above mechanism is much more cost effective because you pay only for what had changed. Second, the overall capacity of the backup is efficiently used and third snapshots are fast to take than traditional backups. You should note that taking a snapshot can impact the rate of IOPS you get from your volume while your snapshot is pending; this is usually few milliseconds->seconds depending upon the changes occurred between snapshots.

In Amazon infrastructure, Snapshots are usually used for achieving some of the following objectives:

• Expand the size of a EBS volume
• Create multiple duplicate (copies) volumes inside an AZ
• Create volumes across Amazon Availability Zones inside an Amazon EC2 region (in event of failure)
• Create similar volumes across Amazon EC2 regions using EBS snapshot copy mechanism. This feature will help you during geographic expansion, data center migration, and disaster recovery.

Since EBS snapshots can be taken regardless of whether or not the volume is attached to a running Amazon EC2 instance, it is strongly recommended to either detach the volume or freeze all writes before taking snapshot to prevent data loss. Not all the times we can detach a volume for taking snapshots, imagine you are running a database or Solr Search in EC2, these services need to run continuously and this option is not feasible and might prove very costly.  In Amazon cloud it is a recommended practice to use file systems like XFS which provides option to freeze writes for a while and take the snapshot consistently. XFS can is very useful when we use EBS Striping (RAID 0) as well.

A snapshot of an EBS volume writes a copy of the volume data in Amazon S3 (Not Buckets). S3 is an excellent option for snapshot storage because

• S3 is a separate infrastructure than EBS storage, hence it improves the availability factor and reduces the dependency in event of EBS failure 
• EBS volumes have availability zone scope and can be attached only to Amazon EC2 instances launched in same AZ. On the other hand, since the snapshots are stored in S3, you can create a new volume from them in any AZ inside the Amazon EC2 region.
• Since the snapshots are not stored directly in buckets, you cannot access them using S3 API’s, you can only list the snapshots using the EC2 API
• On the other hand one negative I have observed is that: Accessing data for the first time from Amazon S3 snapshot might cause latency during the initial loading period i.e. whenever you create new volumes from existing Amazon S3 snapshots; they load lazily in the background. But if your EC2 instance accesses data that hasn’t yet been loaded from S3, the volume immediately downloads the requested data from S3, and continues loading the rest of the data in the background. In case you are trying to access S3 snapshots from the private subnet inside VPC, make sure your NAT instance capacity is right sized to reduce the latency during loading.

EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store

Part 2: Understanding Standard EBS Volumes

Part 3: Understanding EBS PIOPS Volumes

Part 4: Understanding EBS-Optimized Instances

Part 5: Understanding Latency in EBS

Part 7: 10% of your provisioned IOPS 99.9% of the time

Part 8: Performance Tuning - Pre Warming the EBS volume

Part 9: Performance Tuning - EBS Striping

Part 10: Performance Tuning - IO Block Size

Part 11: Understanding Amazon EBS Snapshots

Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 

Part 13: Amazon EBS Security Best practices and tips