AWS Cost Saving Tip 12: Add Spot Instances with Amazon EMR

In continuation to my post on "How elastic thinking can save costs on Amazon EMR cluster ?" i have explored in this post how we can exploit Amazon EMR by introducing Spot EC2 into the cluster and achieve more cost savings.

Most of us know that Amazon Spot EC2 instances are usually good choice for Time-flexible and interruption-tolerant tasks. These instances gets traded frequently on a Spot market price and you can fix your Bid Price using AWS API's or AWS Console. Once free Spot EC2 instances are available for your Bid Price, AWS will allot them for use in your account. Spot instances are usually available way cheaper than On-Demand EC2 instances most of the times. Example: On-Demand m1.xlarge per hour price is 0.48 USD and on spot market you can find them sometimes @ 0.052 per hour. This is ~9 times cheaper than the on-demand price; imagine if you can bid competitively and get hold of spot EC2 even around 0.24 USD most of the times, you are saving 50% from the on-demand price straight away. In Big data use cases usually you might need lots of EC2 nodes for processing, adopting such techniques can vastly make difference in your infra cost and operations in long term. I am sharing my experience on this subject as tips and techniques you can adopt to save costs while using EMR clusters in Amazon for big data problems. 
Note : While dealing with spot you can be sure that you will never pay more than your maximum bid price per hour.  

To know more about real implementation of these tips, read the following case study. Lock, Stock and X Smoking EC2's. 

Tip 1: Make right choice (Spot vs On-Demand) for the cluster components
Data Critical workloads: For workloads which cannot afford to lose data you can have the Master + Core on Amazon On-Demand EC2 and your task nodes on Spot EC2. This is the most common pattern while combining Spot and On-Demand on Amazon EMR cluster.  Since task nodes are operating on spot prices depending upon your bidding strategy you can save ~50% costs from running your task nodes using On-Demand EC2. You can further save(if you are lucky) by reserving your Core and Master Nodes , but you will be tied to an AZ. According to me this is not a good or common technique, because some AZ's can be very noisy with high spot prices.  
Cost Driven workloads: When solving big data problems, sometimes you might have to face scenarios where cost is very important than time. Example: You are processing archives of old logs as low priority jobs, where cost of processing is very important and usually with abundant time left. Such cases you can have all the Master+Core+Task run on Spot EC2 to get further savings from the data critical workloads approach. Since all the nodes are operating on spot prices depending upon your bidding strategy you can save ~60% or more costs from running your nodes using On-Demand EC2. The below mentioned table published by AWS gives an indication of the Amazon EMR + Spot combinations that are widely used:

Tip 2: There is free lunch sometimes
Spot Instances can be interrupted by AWS when the spot price reaches your bidding price. What interruption means is that, AWS can pull out the Spot EC2's assigned to your account when the price matches/exceeds. If your Spot Task Nodes are interrupted you will not be charged for any partial hour of usage by AWS i.e. if you have started the instance @ 10:05 am and if your instances are interrupted by spot price fluctuations @ 10:45 am you will not be charged for the partial hour of usage. If your processing exercise is totally time insensitive, you can keep your bidding price at closer level to spot price which are easily interrupt-able by AWS and exploit this partial hours concept. Theoretically you can get most of the processing done through your task nodes for free* exploiting this strategy.

Tip 3: Use the AZ wisely when it comes to spot
Different AZ's inside an Amazon EC2 region has different spot prices for the same Instance type. Observe this pattern for a while, build some intelligence around the price data collected and rebuild your cluster in the AZ with lowest price. Since the Master+Core+Task need to run on the same AZ for better latency, it is advisable to architect your EMR clusters in such a way they can be switched(i.e.recreate) to different AZ's according to spot prices. If you can build this flexibility in your architecture you can save costs by leveraging the Inter AZ price fluctuations. Refer the below images for Spot Price variations in 2 AZ's inside the same Region for same time period. "Make your choice wisely time to time"

Tip 4: Keep your Job logic small and store intermediate outputs in S3
Breakdown your complex processing logic into small jobs and design your jobs and tasks in EMR cluster in such a way that they run for very small period of time (example few minutes). Store all the intermediate job outputs in Amazon S3. This approach is helpful in EMR world and gives you following benefits:

• When your Core+ Task nodes are interrupted frequently, you can still continue from the intermediate points. Data accessed from S3.
• You now have the flexibility to recreate the EMR clusters in multiple AZ depending upon the Spot price fluctuations
• You can decide the number of nodes needed for your EMR cluster(even every hour) depending upon the data volume, density and velocity

All the above 3 points when implemented contribute to elasticity in your architecture and there by helps you save costs in Amazon cloud. The above recommendation is not suitable for all Jobs, it has to be carefully mapped with right use cases by the architects.

To know more about real implementation of the above tips, read the following case study. Lock, Stock and X Smoking EC2's. 

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

Lock, Stock and X Smoking EC2's - Cloud Connect 2013 Presentation

This case was presented @ Cloud Connect 2013 . You can view this in Slide share as well.

To know more about Amazon EMR + Spot refer the following articles. They will set the technical context right before you go through this presentation. Posts are listed below:

Post 1: How elastic thinking saves cost in Amazon EMR Clusters ?
Post 2: Add Spot Instances with Amazon EMR

Cloud Connect 2013- Lock Stock and x Smoking EC2's from Harish Ganesan

Case :
Mobile advertising company based in USA. They have Forbes 1000 clientele using their mobile advertising product. The company approached us for architecture and implementation help for their Clicks, Ad and other analytic's hosting and processing on AWS.

The LOCK ?
Variety of unstructured logs and semi structured files have to be processed for data analysis. It includes logs from CDN, server logs,  XML Files, Text files, Geo Data Files and Structured DB records. The logs flow every hour from variety of sources into AWS. It amounts to few hundred GB's -> ~ 1 TB @ peak hours/seasons.

The STOCK ?
Mobile Ad company wanted an efficient architecture and infrastructure on AWS for collecting, storing, processing and share the data. They wanted to avoid cost leakages because of bad architectural practices in AWS and save $$$ wherever possible.

Challenges:
Their analysis patterns can be categorized into Hourly, Monthly and Historical. Some of the major challenge were:

• How do we Transfer, Store, Analyze and Share ?
• How to optimize costs at this scale ?

Solution:

• Architect their entire front end analysis module in AWS. Use the AWS technologies like ELB, EC2, R53, RDS ,Cache for delivering their analysis  results to online users.
• Use Amazon Elastic MapReduce with Spot EC2 instances for their back-end processing jobs.
• Automate their infrastructure using Chef, Scripts and Custom Java Modules wherever applicable. 

Stage 1: Transfer the Data :
First time data was transferred using AWS Import/Export. On going data was transferred to AWS using Tsunami UDP. High Bandwidth EC2's were installed with Tsunami UDP on AWS side for receiving data faster. The Data collected is stored temporarily in the Receiver EC2's , some pre-processing tasks were run on them and files are moved to S3 Buckets post that.
For more details about using Tsunami UDP and Aspera on AWS refer these articles: Tsunami UDP on AWS and Aspera on AWS

Some of the other popular models are :
AWS Direct Connect: Establishes a dedicated connection between Data centers to AWS using Direct Connect. This model is little costlier and is suitable only enterprises.
WAN Optimization: Use WAN optimization tools like Riverbed Steel head, aryaka etc to speed up the transfer between source endpoints to AWS.

Stage 2: Storage of Data:  
Temporarily data was stored on Receiver EC2's. After Pre-processing tasks they were moved to S3. Amazon S3 is a default choice because of its inherent fault tolerance and scalability features. Around ~2 TB of compressed logs are stored in Amazon S3 Daily for processing. Amazon S3 Reduced Redundancy Storage option was used for storing intermediate log outputs. S3 Automatic Object Expiry was used for efficiency and cost savings. Archival data is moved to Amazon Glacier periodically.

Stage 3: Analysis using Amazon EMR + Spot EC2
Amazon EMR was used for log processing and analysis. Amazon Elastic MapReduce (EMR) is a web service that helps customers with big data processing using Hadoop framework on EC2 and S3. Amazon Elastic MapReduce lets customers focus on crunching data instead of worrying about time-consuming set-up, management or tuning of Hadoop clusters or the EC2 capacity upon which they operate. This in built automation provided by AWS already saves huge labor cost for the customers. At peak hours some jobs run with 2000 Mappers/ 750 Reducers. For peak periods ~250 m1.xlarge or equivalent EC2 capacity was used for processing the logs. We developed a Custom EMR manager to introduce Spot EC2 into the Amazon EMR equation. Our Spot Bidding strategy was either on-Demand price or 20% above On-demand price. Since the Spot prices vary in different AZ's we collected the past price history, current market price etc and chose the right AZ's appropriately. Choosing right AZ sometimes mean creating the entire EMR Cluster on low priced AZ (All Master,Core & Task should reside on Same AZ).  The Master and Core Nodes were running On-Demand EC2 and Task Nodes were running on Spot EC2 or On-Demand EC2. The Custom EMR Manager has the capability to increase the Core nodes depending upon the Log Data Volume + Create the entire EMR cluster (with revised Core+task Node numbers) in New AZ depending upon the Spot prices. This decision was made by the Custom EMR manager periodically depending upon the log data volume fluctuations and Log Volume patterns.   

Learning:

• Spot+On Demand EC2 for EMR is deadly combination for Cost Savings. Though this combination is suitable for Time in-sensitive jobs, but if some intelligence and imagination can be applied in your architecture and design, you can use this for certain time sensitive jobs as well.
• Bigger files are better, so we merge the files into Bigger chunks before proccesing in Amazon EMR.
• In Pre processing stage, we split files with ratio of 1 file per mapper. This gives us better time predictability during processing. Also the file transfer was faster between S3 and EMR because of this manageable size splits.
• Data is compressed at all possible levels. Snappy and .lzo compression was used.
• We dynamically increase/decrease the task nodes using the Custom job manager. If no Spot EC2 available for Task nodes, then Custom EMR manager adds on-Demand task nodes to the cluster.
• The jobs were designed to be small in processing logic size and intermediate output data is stored in S3. This way we can re-create the cluster with reproducible data. Only Processing data was kept in EMR cluster, rest of the data were kept in S3.
• The EMR clusters can be sized according to the log data volumes. This intelligence was built in the custom EMR manager based on the past patterns, Data Density tests etc, during pre processing phase.
• Certain Intermediate reducers were designed to create number of result files according to number of Mappers in next jobs. This helped us in optimum utilization of capacity inside a hour.
• Decision to re-size clusters are made nearest to the hour. If AWS brings per minute pricing , it will help such use cases to save more costs.
• Every API call and millisecond matters in big data programming, tune the MR code and test the 3rd party APIs for performance before integrating them into your code.
• Understand NW capacities of to/fro data transfer to S3, m1.Xlarge EMR Nodes (Memory Sizing, Mapper / reducer numbers) and work leveraging the strengths of AWS.
• If your processing requirements are totally time insensitive, you can bid very close to spot price and have AWS interrupt(mostly) within the hour and literally get entire processing stuff done for free. (If AWS interrupts the Spot EC2 you will not be charged for any partial hour of usage). This strategy was not used by us. If any of you use this, please write to me, will be happy to discuss.

Value :

• EMR with Spot brought ~56% cost savings from pure On-Demand model for Core+ Task Nodes.
• Customer CXO's were happy !!!