Saturday, September 21, 2013

AWS Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage

Tip 1: Use Multi-part upload and range retrieval:
You store data in Amazon Glacier as an archive. An archive is any object, such as a photo, video, or document that you store and usually represents a single file or several files. It is a base unit of storage in Amazon Glacier. You can upload an archive in a single request. For large archives, Amazon Glacier provides a multipart upload API that enables you to upload an archive in parts and enables range retrieval in future.
In reality many times, you may need to retrieve a small selection of those files from a archive, in which case you could retrieve only the ranges of the archive that contained the required files.You may also choose to perform range retrievals in order to reduce or eliminate your retrieval fees. If you exceed your free retrieval allowance, you pay a retrieval fee that is based on your peak retrieval rate. Spreading out a retrieval of an archive in smaller parts could therefore allow you reduce your retrieval fees, by reducing your peak retrieval rate.

Tip 2: Spread your retrievals and avoid cost leakage:
Let’s assume you are storing 75 TB of data in Amazon Glacier and you would like to retrieve 140 GB. The amount you pay is determined by how fast you retrieve the data. 
Option 1: Storing 75 TB and retrieving 140 GB in 4 hours costs $21.60.  
Option 2: Storing 75 TB and retrieving 140 GB in 8 hours costs $10.80 (half the cost of option 1)
Option 3: Storing 75 TB and retrieving 140 GB evenly over 28 hours costs NIL because you would no longer exceed your daily free retrieval allowance and would therefore not be charged a Retrieval Fee. Imagine how much cost is leaked if option 1 is followed frequently instead option 3 where the billing is lower or free most times. This can be avoided by better understanding of Amazon Glacier, planning your retrieval in advance with proper spread.

In Amazon Glacier, the suggested practice to lower costs is to plan your retrieval strategy in advance by spreading out the retrievals over longer periods of time. This will reduce or eliminate the retrieval fees and lowers your billing.
Use the aggressive retrieval mechanisms only on use cases where timing is important, Example : You have an old archived news footage in Amazon Glacier, and suddenly based on the current event this old new footage needs to be retrieved, related and delivered. In this case you can use peak retrieval rate and other times you can spread your retrieval strategy.

To explore Log Analysis and Archive with Amazon S3 and Glacier. Refer this detailed article series.

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

No comments:

Need Consulting help ?


Email *

Message *

All posts, comments, views expressed in this blog are my own and does not represent the positions or views of my past, present or future employers. The intention of this blog is to share my experience and views. Content is subject to change without any notice. While I would do my best to quote the original author or copyright owners wherever I reference them, if you find any of the content / images violating copyright, please let me know and I will act upon it immediately. Lastly, I encourage you to share the content of this blog in general with other online communities for non-commercial and educational purposes.