AWS Cost Saving Tip 18: Using AWS Tagging with Graffiti Monkey to Saves costs

Tagging AWS assets and resources is a simple and an important feature that helps IT teams streamline their process of managing the environments. We can categorize AWS resources in variety of ways using the AWS Tagging feature. For example: 

• Environment based Tags: Environment based tags can be used by customers who have Larger AWS footprints. These tags will help you to distinguish between production, development, and staging infrastructure deployed on AWS
• Application based Tags: Used to describe the set of disparate resources (or clusters) that work together to deliver a product or service to a customer. Can be applied on Small to Big AWS Production environments to identify the resources used
• Role based Tags– Used to describe the function of a particular node (web server, database server, load balancer, etc.). Owner – Used to identify the individual who is responsible for the instance. Both these cases are usable for small AWS foot prints.

How it helps ?
Tags can be very useful in understanding our AWS Billing. When you run large foot prints on AWS, constantly we need to monitor and manage our resource usage. Any IT operations team which manages AWS resources, constantly need to work on the following questions to optimize and manage their operations efficiently. Which AWS resources are used ? Which resources are lying idle (volumes, snapshots) in detached state ? Which Tiers or environments are consuming more billing than others? Which owner is exceeding the AWS cost allocated ? In short by starting to tag the AWS resources we will know how those assets are translating to monthly cost and take optimization decisions. Tagging is an simple but an essential feature that helps us to reduce and keep costs in control by improving our management efficiency in AWS.

AWS Tags in Detail:
Each tag consists of a key and a value, both of which the user can define. Currently AWS does not provide facility to TAG all AWS resources. Some of them can be Tagged only using API actions or the Command Line. The following Screen Shot shows all AWS resources and the restrictions that apply to them. Resources with tagging restrictions of None can be tagged with API actions, the CLI, and the console.

The following diagram illustrates how tagging works. In this example, i am assigning two tags to each of my EC2 instances, one called Owner and another called Stack. Each of the tags also has an associated value.
Owner = 8KMiles , Stack = Development, Stack = Production as illustrated in the below diagram

You can assign tags using the AWS Management Console, the Amazon EC2 command line interface (CLI), and the Amazon EC2 API.You can define tags from the EC2 console by selecting the relevant instance and selecting the “Tags” option.  Alternatively, you can select the instance, click “Actions” and select “Add/Edit Tags.” Below Screenshot shows how tags can be assigned using AWS Management Console

We have assigned Name, Owner , Environment and Contact for the 8KMiles Production instance using the AWS Tagging feature. The above is not the usual EC2 Managed Service naming standard , but just an simple illustration of the tag use case.
AWS Tagging Constraints and the Cost Leakage:
Currently AWS Tagging has some limitations as well. For example each tag associated with a single AWS resource, applies only to that particular resource and it is not propagated to the dependent resources attached to it. Example: Imagine if you have an EC2 instance, most of the times this EC2 instance is attached with dependent resources like EBS volumes, Snapshots etc. Usually there will be 1-2 EBS Volumes on normal scenarios and up to 8 EBS Volumes or more on RAID Scenarios. Number of snapshots are more in number and usually becomes unmanageable. Imagine how this problem will be propagated if you are running a Large AWS foot print with thousands of EC2 instances. 
You have to find each EBS volumes associated with a particular EC2 and Tag it separately.Managing this is easier said than done and usually slightest mis- management of these resources will lead to cost leakage. What if i can use a tool to automate tagging for dependent AWS resources. It will surely reduce the manual labor associated and help us remove all the dependent resources when we are removing the EC2. This avoids leakages and reduces cost of Cloud Operations in very dynamic AWS environments.

Welcome to Graffiti Monkey. 
The Graffiti Monkey goes around tagging things automatically. By looking at the tags of an EC2 instance, it copies those tags to the EBS Volumes that are attached to it, and then copies those tags to the EBS Snapshots as well. This small automation reduces hours of Manual labor on large AWS footprints, helps us efficiently manage the cloud infrastructure and reduce the cost leakages.

Step 1: Installation  of Graffiti Monkey 

Easiest way to install Graffiti Monkey using the usual PyPI channels.Please refer the below screenshot

link for installing pip.http://www.pip-installer.org/en/latest/installing.html

Step 2: Using Graffiti Monkey

It will identify the tags associated with EC2 instance and applies them to dependent resources like EBS volumes and snapshots saving hours of manual labor.

This Post is co authored with Nitheesh 8Kmiles

Other Cost Saving Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs 
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage 

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage 

Cost Saving Tip 16: Periodically remove your unwanted AWS resources and save costs (new)

AWS Cost Saving Tip 17: Add Amazon RDS Read Replica's on need basis and save costs (new)

Deeper Health Checks and Problems in Load Balancing in AWS

Health Checks are one of the essential mechanisms that helps you to keep N-Tiered system highly available. Usually a simple script or program is deployed on the Web/App Server. The Health check component of Load Balancer is configured to frequently call this script in Web/App Server in a light weight protocol. Based on the response from the script/program the Load balancer decides the status of the Web/App Servers and accordingly direct the requests to healthy Web/App servers. This is a usual mechanism that is followed in all popular load balancers like Amazon ELB, Netscaler, HAProxy and NGinx in AWS cloud. This sounds simple and straight forward right, but some of the customers i have consulted follow a much deeper Health check diagnostic mechanism and it might have problems when migrated to AWS cloud. 
Let us explore this case in detail :

What is the architecture ?
A simple multi-tiered architecture with : A  load balancer deployed at the front. The Web/App Server has the health script/program. The database is MySQL deployed Master+ Slave mode.  

What is deeper Health Check ?
The script/program deployed in the Web/App Server is little intelligent; when it is called by load balancer it performs simple operations and checks the status of the Database. So when you get a response back from the health check script / program you are verifying whether the health of DB and Web/App server is sound at the load balancer tier.

What is the problem scenario ?
Imagine when migrating this infrastructure to AWS you have adopted the standard architecture pattern consisting of :

• Amazon ELB is used as the Load balancer
• Web/App Server in auto scaling mode
• MySQL moved to Amazon RDS+Multi-AZ with RR

Now let us explore this problem in detail : 

• Imagine the any of the following condition in your production, network between database and Web/App is down intermittently for few minutes or RDS MySQL is elevating the Hot Standby as new Master. In such scenarios, the health check response actually timeouts at Database level, whereas the Load Balancer will mark the even the healthy App Servers as unhealthy because of the deeper health checks. This is not good especially for Amazon Auto Scaled scenario's where Amazon ELB marks Web/App EC2 as unhealthy because of deeper health check and Amazon Auto Scaling keeps restarting the Web/App EC2 auto automatically to maintain minimum healthy farm. This unwanted effect can cascade the overall availability and surely not good for the production in AWS. So in short Deeper Health checks are not surely recommended for complex N-Tier systems that follows Auto scaling/healing and Service oriented architecture patterns in AWS. 
• Usually the purpose of health check is to check the status of next tier or service consumed by a particular tier.  Deeper health checks is heavy weight and it usually takes much more time to respond because majority of your tiers are exercised in this process. If we set this frequency too aggressive, then health checks itself will eat lots of your CPU. So the frequency of the health checks and the response time out have to be set considerably large. Also during heavy traffic scenario, such heavy weight calls can be queued and you might not get faster response in deeper health checks. 
• Deeper health checks are usually suitable for simple and fixed infrastructures. When your infrastructure is non elastic , the decisions are taken manually by the ops team after analyzing the particular failing tier. For Elastic Auto scaled workloads in AWS it is better to isolate the health checks of load balancing tier separate from Deeper Health checks that can be used for assessing the availability of the infrastructure.

AWS Cost Saving Tip 12: Add Spot Instances with Amazon EMR

In continuation to my post on "How elastic thinking can save costs on Amazon EMR cluster ?" i have explored in this post how we can exploit Amazon EMR by introducing Spot EC2 into the cluster and achieve more cost savings.

Most of us know that Amazon Spot EC2 instances are usually good choice for Time-flexible and interruption-tolerant tasks. These instances gets traded frequently on a Spot market price and you can fix your Bid Price using AWS API's or AWS Console. Once free Spot EC2 instances are available for your Bid Price, AWS will allot them for use in your account. Spot instances are usually available way cheaper than On-Demand EC2 instances most of the times. Example: On-Demand m1.xlarge per hour price is 0.48 USD and on spot market you can find them sometimes @ 0.052 per hour. This is ~9 times cheaper than the on-demand price; imagine if you can bid competitively and get hold of spot EC2 even around 0.24 USD most of the times, you are saving 50% from the on-demand price straight away. In Big data use cases usually you might need lots of EC2 nodes for processing, adopting such techniques can vastly make difference in your infra cost and operations in long term. I am sharing my experience on this subject as tips and techniques you can adopt to save costs while using EMR clusters in Amazon for big data problems. 
Note : While dealing with spot you can be sure that you will never pay more than your maximum bid price per hour.  

To know more about real implementation of these tips, read the following case study. Lock, Stock and X Smoking EC2's. 

Tip 1: Make right choice (Spot vs On-Demand) for the cluster components
Data Critical workloads: For workloads which cannot afford to lose data you can have the Master + Core on Amazon On-Demand EC2 and your task nodes on Spot EC2. This is the most common pattern while combining Spot and On-Demand on Amazon EMR cluster.  Since task nodes are operating on spot prices depending upon your bidding strategy you can save ~50% costs from running your task nodes using On-Demand EC2. You can further save(if you are lucky) by reserving your Core and Master Nodes , but you will be tied to an AZ. According to me this is not a good or common technique, because some AZ's can be very noisy with high spot prices.  
Cost Driven workloads: When solving big data problems, sometimes you might have to face scenarios where cost is very important than time. Example: You are processing archives of old logs as low priority jobs, where cost of processing is very important and usually with abundant time left. Such cases you can have all the Master+Core+Task run on Spot EC2 to get further savings from the data critical workloads approach. Since all the nodes are operating on spot prices depending upon your bidding strategy you can save ~60% or more costs from running your nodes using On-Demand EC2. The below mentioned table published by AWS gives an indication of the Amazon EMR + Spot combinations that are widely used:

Tip 2: There is free lunch sometimes
Spot Instances can be interrupted by AWS when the spot price reaches your bidding price. What interruption means is that, AWS can pull out the Spot EC2's assigned to your account when the price matches/exceeds. If your Spot Task Nodes are interrupted you will not be charged for any partial hour of usage by AWS i.e. if you have started the instance @ 10:05 am and if your instances are interrupted by spot price fluctuations @ 10:45 am you will not be charged for the partial hour of usage. If your processing exercise is totally time insensitive, you can keep your bidding price at closer level to spot price which are easily interrupt-able by AWS and exploit this partial hours concept. Theoretically you can get most of the processing done through your task nodes for free* exploiting this strategy.

Tip 3: Use the AZ wisely when it comes to spot
Different AZ's inside an Amazon EC2 region has different spot prices for the same Instance type. Observe this pattern for a while, build some intelligence around the price data collected and rebuild your cluster in the AZ with lowest price. Since the Master+Core+Task need to run on the same AZ for better latency, it is advisable to architect your EMR clusters in such a way they can be switched(i.e.recreate) to different AZ's according to spot prices. If you can build this flexibility in your architecture you can save costs by leveraging the Inter AZ price fluctuations. Refer the below images for Spot Price variations in 2 AZ's inside the same Region for same time period. "Make your choice wisely time to time"

Tip 4: Keep your Job logic small and store intermediate outputs in S3
Breakdown your complex processing logic into small jobs and design your jobs and tasks in EMR cluster in such a way that they run for very small period of time (example few minutes). Store all the intermediate job outputs in Amazon S3. This approach is helpful in EMR world and gives you following benefits:

• When your Core+ Task nodes are interrupted frequently, you can still continue from the intermediate points. Data accessed from S3.
• You now have the flexibility to recreate the EMR clusters in multiple AZ depending upon the Spot price fluctuations
• You can decide the number of nodes needed for your EMR cluster(even every hour) depending upon the data volume, density and velocity

All the above 3 points when implemented contribute to elasticity in your architecture and there by helps you save costs in Amazon cloud. The above recommendation is not suitable for all Jobs, it has to be carefully mapped with right use cases by the architects.

To know more about real implementation of the above tips, read the following case study. Lock, Stock and X Smoking EC2's. 

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

Lock, Stock and X Smoking EC2's - Cloud Connect 2013 Presentation

This case was presented @ Cloud Connect 2013 . You can view this in Slide share as well.

To know more about Amazon EMR + Spot refer the following articles. They will set the technical context right before you go through this presentation. Posts are listed below:

Post 1: How elastic thinking saves cost in Amazon EMR Clusters ?
Post 2: Add Spot Instances with Amazon EMR

Cloud Connect 2013- Lock Stock and x Smoking EC2's from Harish Ganesan

Case :
Mobile advertising company based in USA. They have Forbes 1000 clientele using their mobile advertising product. The company approached us for architecture and implementation help for their Clicks, Ad and other analytic's hosting and processing on AWS.

The LOCK ?
Variety of unstructured logs and semi structured files have to be processed for data analysis. It includes logs from CDN, server logs,  XML Files, Text files, Geo Data Files and Structured DB records. The logs flow every hour from variety of sources into AWS. It amounts to few hundred GB's -> ~ 1 TB @ peak hours/seasons.

The STOCK ?
Mobile Ad company wanted an efficient architecture and infrastructure on AWS for collecting, storing, processing and share the data. They wanted to avoid cost leakages because of bad architectural practices in AWS and save $$$ wherever possible.

Challenges:
Their analysis patterns can be categorized into Hourly, Monthly and Historical. Some of the major challenge were:

• How do we Transfer, Store, Analyze and Share ?
• How to optimize costs at this scale ?

Solution:

• Architect their entire front end analysis module in AWS. Use the AWS technologies like ELB, EC2, R53, RDS ,Cache for delivering their analysis  results to online users.
• Use Amazon Elastic MapReduce with Spot EC2 instances for their back-end processing jobs.
• Automate their infrastructure using Chef, Scripts and Custom Java Modules wherever applicable. 

Stage 1: Transfer the Data :
First time data was transferred using AWS Import/Export. On going data was transferred to AWS using Tsunami UDP. High Bandwidth EC2's were installed with Tsunami UDP on AWS side for receiving data faster. The Data collected is stored temporarily in the Receiver EC2's , some pre-processing tasks were run on them and files are moved to S3 Buckets post that.
For more details about using Tsunami UDP and Aspera on AWS refer these articles: Tsunami UDP on AWS and Aspera on AWS

Some of the other popular models are :
AWS Direct Connect: Establishes a dedicated connection between Data centers to AWS using Direct Connect. This model is little costlier and is suitable only enterprises.
WAN Optimization: Use WAN optimization tools like Riverbed Steel head, aryaka etc to speed up the transfer between source endpoints to AWS.

Stage 2: Storage of Data:  
Temporarily data was stored on Receiver EC2's. After Pre-processing tasks they were moved to S3. Amazon S3 is a default choice because of its inherent fault tolerance and scalability features. Around ~2 TB of compressed logs are stored in Amazon S3 Daily for processing. Amazon S3 Reduced Redundancy Storage option was used for storing intermediate log outputs. S3 Automatic Object Expiry was used for efficiency and cost savings. Archival data is moved to Amazon Glacier periodically.

Stage 3: Analysis using Amazon EMR + Spot EC2
Amazon EMR was used for log processing and analysis. Amazon Elastic MapReduce (EMR) is a web service that helps customers with big data processing using Hadoop framework on EC2 and S3. Amazon Elastic MapReduce lets customers focus on crunching data instead of worrying about time-consuming set-up, management or tuning of Hadoop clusters or the EC2 capacity upon which they operate. This in built automation provided by AWS already saves huge labor cost for the customers. At peak hours some jobs run with 2000 Mappers/ 750 Reducers. For peak periods ~250 m1.xlarge or equivalent EC2 capacity was used for processing the logs. We developed a Custom EMR manager to introduce Spot EC2 into the Amazon EMR equation. Our Spot Bidding strategy was either on-Demand price or 20% above On-demand price. Since the Spot prices vary in different AZ's we collected the past price history, current market price etc and chose the right AZ's appropriately. Choosing right AZ sometimes mean creating the entire EMR Cluster on low priced AZ (All Master,Core & Task should reside on Same AZ).  The Master and Core Nodes were running On-Demand EC2 and Task Nodes were running on Spot EC2 or On-Demand EC2. The Custom EMR Manager has the capability to increase the Core nodes depending upon the Log Data Volume + Create the entire EMR cluster (with revised Core+task Node numbers) in New AZ depending upon the Spot prices. This decision was made by the Custom EMR manager periodically depending upon the log data volume fluctuations and Log Volume patterns.   

Learning:

• Spot+On Demand EC2 for EMR is deadly combination for Cost Savings. Though this combination is suitable for Time in-sensitive jobs, but if some intelligence and imagination can be applied in your architecture and design, you can use this for certain time sensitive jobs as well.
• Bigger files are better, so we merge the files into Bigger chunks before proccesing in Amazon EMR.
• In Pre processing stage, we split files with ratio of 1 file per mapper. This gives us better time predictability during processing. Also the file transfer was faster between S3 and EMR because of this manageable size splits.
• Data is compressed at all possible levels. Snappy and .lzo compression was used.
• We dynamically increase/decrease the task nodes using the Custom job manager. If no Spot EC2 available for Task nodes, then Custom EMR manager adds on-Demand task nodes to the cluster.
• The jobs were designed to be small in processing logic size and intermediate output data is stored in S3. This way we can re-create the cluster with reproducible data. Only Processing data was kept in EMR cluster, rest of the data were kept in S3.
• The EMR clusters can be sized according to the log data volumes. This intelligence was built in the custom EMR manager based on the past patterns, Data Density tests etc, during pre processing phase.
• Certain Intermediate reducers were designed to create number of result files according to number of Mappers in next jobs. This helped us in optimum utilization of capacity inside a hour.
• Decision to re-size clusters are made nearest to the hour. If AWS brings per minute pricing , it will help such use cases to save more costs.
• Every API call and millisecond matters in big data programming, tune the MR code and test the 3rd party APIs for performance before integrating them into your code.
• Understand NW capacities of to/fro data transfer to S3, m1.Xlarge EMR Nodes (Memory Sizing, Mapper / reducer numbers) and work leveraging the strengths of AWS.
• If your processing requirements are totally time insensitive, you can bid very close to spot price and have AWS interrupt(mostly) within the hour and literally get entire processing stuff done for free. (If AWS interrupts the Spot EC2 you will not be charged for any partial hour of usage). This strategy was not used by us. If any of you use this, please write to me, will be happy to discuss.

Value :

• EMR with Spot brought ~56% cost savings from pure On-Demand model for Core+ Task Nodes.
• Customer CXO's were happy !!!

AWS Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ?

Introduction: Amazon Elastic MapReduce (EMR) is a web service that helps customers with big data processing using Hadoop framework on EC2 and S3. Amazon Elastic MapReduce lets customers focus on crunching data instead of worrying about time-consuming set-up, management or tuning of Hadoop clusters or the EC2 capacity upon which they operate. This in built automation provided by AWS already saves huge labor cost for the customers.

What does the word Elastic mean in Hadoop/EMR context ? Ans: You can dynamically increase the number of processing nodes depending upon the volume/velocity of the data. Adding or removing servers takes minutes, which is much faster than making similar changes in clusters running on physical servers. Let us explore this in detail and analyse how it will help you save costs in AWS Big data processing. 

Components:
Before getting into the savings part, lets understand the composition of an Amazon EMR Cluster. An Amazon EMR cluster consists of following server components. They are:
Master Node: This node Manages the cluster, it coordinates the distribution of the MapReduce executable and subsets of the raw data, to the core and task nodes. There is only one master node in a cluster. You cannot expand or reduce your Master Node in the EMR Cluster.
Core Node(s): A core node is an EC2 instance that runs Hadoop map/reduce tasks and stores data using the Hadoop Distributed File System (HDFS). Core nodes are managed by the master node.  You can add more core nodes to running cluster, but you cannot remove them from a cluster because since it stores data you have an risk of losing data. 
Task Node(s): As the name suggests these nodes run tasks and they map to equivalent of Hadoop slave node.  These nodes are optional in nature.  Task nodes are managed by the master node. While a cluster is running you can increase and decrease the number of task nodes. Because they don't store data and can be added and removed from a cluster, you can use task nodes to manage the EC2 instance capacity your cluster, by increasing capacity to handle peak loads and decreasing it later when there is no load.

Analysis :
Imagine the log volume flow is not constant and it varies every hour, some hours you receive few hundred GB's and some hours few GB's of logs for processing. For peak hours your use case needs around 192 mappers/72 reducers and normal hours you need ~64 mappers/24 reducers or less. The peak and normal numbers can be arrived based on the analysis done on the past data. This elasticity in log volume scenario is a usual occurrence in many big data projects and it is source of cost leakage. Simple approach what many architects take is that they run their cluster infrastructure @ peak capacity always since the operation is time sensitive, but this might not be an optimal approach in amazon cloud-big data world. Since you can elastically increase/decrease the number of nodes in an Amazon EMR cluster it is optimal if you can size the number of nodes dynamically every hour. Since you pay by usage in amazon cloud, having this elasticity built in your architecture will save costs. 

Based on the number of mappers/reducers required, we have chosen the node capacity to be in m1.xlarge EC2 units. So during Peak hours you will need 24 processing nodes and normal (avg) hours it will be reduced to 8 processing nodes. 

Elastic Approach-1: Vary the Task Nodes: 
In this approach, number of Master and Core nodes are maintained constant. 1 - master node and 4- core nodes are used for processing and data storage always. The task nodes are increased and decreased between 4->20 every hour depending upon the log volume flow. Since the data is present in the core nodes and only tasks/jobs are assigned in the task nodes, adding/removing task nodes will not cause problems. You can engineer a custom Job manager using AWS API's and manage this entire cluster easily. If you do a simple math that in average only 8 processing nodes are needed (4 core + 4 task nodes) and during peak hours you need ( 4 core + 20 task nodes) with this approach you can save ~60 % costs by not running your cluster in ALWAYS peak capacity. This model is a recommended approach for many elastic big data use cases in AWS. Refer the below table for cost savings:

Scenario	No. of Processing Nodes	Hourly rate	Node Type	Monthly
Peak hours	24	0.48	M1.Xlarge	~8570.88
Normal hours	8	0.48	M1.Xlarge	~2856.96

  

Elastic Approach-2: Vary both the Core and Task Nodes:
In this approach, the number of both Core and Task nodes are varied dynamically. Since the Core nodes can be only increased and cannot be decreased in a running cluster(because it could lead to data loss), this approach is recommended only for advanced use cases. Since the entire data is stored in S3(is reproducible) and can be moved to the EMR cluster every hour, using the custom Job manager an entire cluster can be created(even every hour) depending upon the log data volume (GB's). Example: Imagine first hour: 4 Core + 10 Task nodes are used for processing, second hour data volume is increased and 4 Core + 20 Task nodes are added in the cluster, third/fourth hour etc there is hardly few GB's of data flow and only 8 Mappers/3 reducers are needed for processing, instead of running 20 task + 4 core nodes(of prev hour),  a new EMR cluster can be created with just 1 master and 1-2 Core nodes. This approach requires engineering a custom job manager using AWS API's for managing the cluster. Though this approach is little complex to engineer, it saves more cost than approach-1 on medium to long term.

Note:The approaches illustrated are not theoretical in nature. I have put both the above techniques to production use for some customers and they are already seeing huge cost savings. 

Coming Soon - Adding Spot to this equation gives brutal savings ... 

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Top 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Top 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Top 12: Add Spot Instances with Amazon EMR 
Cost Saving Top 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Top 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Top 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

Amazon EC2 AMI Copy Feature - Performance, Tips and How To

An Amazon Machine Image (AMI) is a pre-configured operating system image that is used to create an EC2 instance within the Amazon cloud environment.  AMI’s are unique per region and they not span across multiple regions, But AMI’s can be migrated across regions using the EC2 AMI copy function available as part of AWS Console, API and CLI.  This feature helps customers to maintain consistent Amazon EC2 Multi region deployments and achieve higher availability. Since user-created AMIs are assigned a unique AMI ID within the region, when you configure Auto Scaling Launch configuration or AWS CloudFormation templates in target Amazon EC2 region, it is recommended to update the AMI ID references in these configurations/templates to match the ones that exist in the target region. 

Following table illustrates the time it takes to copy EC2 AMI from Different Source and Destination Amazon EC2 regions. Data used was a 50 GB compressed file. Since AWS keeps improving their infrastructure every day, the time taken may vary in coming months/years. But the figures mentioned below can used as an reference to plan your Recovery time objectives and migrations:

EC2 AMI Copy for Different Amazon EC2 Regions Matrix:

Source Region	Destination region	Size	Time Taken
USA-East	Singapore	50 GB	1 hr 21 min
USA-East	US-West (Oregon)	50 GB	27 min
USA-East	Europe (Ireland)	50 GB	41 min

Important points to note while implementing this are:

• Permissions and user-defined tags applied to the AMI in source region will not be copied to the target region during the AMI copy process. You will have to apply the permissions and tags in the target region after the copy process is completed.
• Imagine you have stored the database connection strings, ElastiCache End points, external IP address and other Integration end points in the AMI of the source region. When you migrate this AMI to the target region these end points might not exist. Worst scenario, if these source end points are accessible over internet like S3, CloudSearch assets etc, you will end up accessing resources from target region itself. This will cause latency issues, performance problems and cost leakage.  
• Some best practices to be followed are:
• Prior to copying an AMI from Source region, you must manually ensure that the contents of the source AMI are updated to support running in a different region. It is recommended to add this step as part of your operation process during Cross region migration activities to avoid problems
• Automate the download of configuration information to EC2 from S3 using user data and other scripts in the respective regions. This way when the migration happens to target region, the configuration files and scripts can be downloaded to EC2 instance from the S3 of the target region. This mechanism keeps the end points in tune with respective regions.
• If you are running Java based environment, use NetFlix Archaius to dynamically source properties from multiple configuration stores inside an Amazon EC2 region. Archaius includes a set of java configuration management APIs and is primarily implemented as an extension of Apache's Commons Configuration Library.  Refer https://github.com/Netflix/archaius/wiki/Overview

How to use EC2 AMI Copy feature between Amazon EC2 Regions ?

Step 1) Login to the AWS Console at https://console.aws.amazon.com/ec2/

In the navigation pane, click the AMI menu under the Images section

Step 2) Select the AMI ID to copy from the source Amazon EC2 region, click Actions Button and select Copy action.

Step 3) In the EC2 AMI Copy page > Mention the destination region, Name of the AMI and Description and then click Copy

Destination Region: The Target region to which the AMI should be copied. In this case we are copying to Amazon EC2 Asia Pacific (Singapore).

Name: Name of the New Target AMI

Description: Description about the AMI. By default below description will be available for use

Step 4) When the Copy process is initiated a new AMI ID will be provided in the screen. Once noted, Click Done button

Step 5)  To check the progress of the copy process, Choose the EC2 Console of the Asia-Pacific(Destination) region and view the new AMI.

  

In the above figure we can see that Copy Status is pending. Once the copy process is completed the status will be changed to "available" from "pending".

Step 6) If you wish to cancel the copy process you can right click on the New AMI in the Source Region and select Deregister option. This will stop the AMI copy process.

Amazon EC2 Multi Region Setup and Migrations -> Security groups

Security Groups are like firewall rules that you configure to restrict access to assets deployed in Amazon Web Services infrastructure. Security groups can exist only within a scope of an Amazon EC2 region (Example security groups configured in AWS US-EAST region is not available on AWS APAC region). For Multi region region setups/migrations in AWS , one common best practice is to export security groups  from primary region and configure the same in target AWS region as well. 

Point 1) You can use the following command to export a copy of the definitions of existing security groups; this will ease the migration effort:

ec2-describe-group –H -–region <Source Region Name>  > security_groups.txt

Point 2) Alternatively you can use the following script to import the security groups in the Target AWS region. This script does not work when you have VPC security group. This script uses the command “ec2-describe-group” to query the existing security groups in a region and creates a shell script, which can be used to re-create in a different region.
Instructions:

• Download the script from the following location: http://ry4an.org/unblog/static/attachments/ec2-security-group-tools.tar.gz
• Set the environment variables which are necessary to run the “ec2-describe-group” command. i.e., AWS_SECRET_KEY and AWS_ACCESS_KEY
• In the file “create-firewall-script.pl” set the AWS region and AWS Account Number. The AWS region name should be set as the region from which you needed to import the security groups. E.g., if you need to import from us-east-1 to us-west-2. The create-firewall-script.pl should be updated with the region as  us-west-2
• Use the following command to run the script, this will generate a shell script. The shell script contains the security groups details and the commands which are needed to create security group. Type the following ec2-describe-group | ./create-firewall-script.pl > create-firewall.sh 
• Just use the command “sh create-firewall.sh ” to run the shell script. This will create the security groups in the target region.
• By using the below command, you can generate security groups details  as an image. Type the following command :                                      ec2-describe-group | ./visualize-security-groups.pl > groups.png

Point 3) Use centralized governance tools like Dome9, cloudaware etc to manage the security groups , changes and management efficiently

It is recommended implement one or more above points proactively for Multi region setups in AWS for improving the Recovery Time Objectives (RTO) during DR.

Configuring Aspera On-Demand on AWS for Huge File transfers

The Aspera On-Demand Server contains the core features from Aspera's Connect Server,  Aspera’s management application, Aspera On-Demand Console for Amazon Web Services (AWS). The On-Demand Server is stored as an Amazon Machine Image (AMI) in AWS account.
Note: Aspera Inc will only share private Amazon Machine Image (AMI) once you have placed a purchase or evaluation order with them.

Aspera On Demand Core Features:

• fasp Transport Server: A transport server that accepts incoming Aspera connections.
• ascp command-line:  A command-line transfer program to initiate Aspera transfers.
• Simple Sync:  A command-line synchronization program.
• Connect Server Web UI:  A web-based transfer server.
• On-Demand Console Features:
• Transfer monitoring and control: View transfers, pause / resume / cancel, change transfer rates
• Transfer Initiation: Initiate and schedule transfer jobs remotely.
• Email Notification: Notify users of transfer events with customizable messages.
• Reporting: Create detail and summary reports of transfer activity.

Aspera On Demand Server Overview:
The Aspera On-Demand Server is packaged as an Amazon Machine Image (AMI). The Aspera AMI contains the Connect Server, and the Aspera On-Demand Console for AWS. When booting up the Aspera AMI, you can use the Connect Server for fasp™ file transfers, and the On-Demand Console for AMI's transfer server management.

How to access Aspera On-Demand Console ?

Type the following on the browser :
Browser : http://<EC2 instance's Public DNS>/aspera/console 
Login: admin/(instance id)
When logged in to On-Demand Console as an administrator, we should see the following items in the Console menu:

• Dashboard: Overview of current and recent transfers, and simple Aspera EC2 node monitoring
• Activity: Lists transfer activities involving at least one managed nodes
• Nodes: Manage the On-Demand Server (managed node) and the unmanaged nodes
• Users: Create and manage On-Demand Console users
• Groups: Create and manage On-Demand Console groups
• Transfer: Initiate a file transfer between Aspera EC2 nodes
• Reports: Generate reports on transfers
• Preferences: Configure your account preferences
• Config: Configure the On-Demand Console in this module
• Search: Search for the most recent transfers that associate to key word(s) entered by users. Search Results shows up to 20 of the most recent transfers.

Instructions to set up Aspera On-Demand server in AWS:


Step 1: Register the Aspera On demand server
Before using the Aspera On-Demand Server, make sure you have registered the Aspera On-Demand for AWS. You can find the registration page here: http://asperasoft.com/ondemand/reg

Step 2: Configure the AWS security groups for File Transfer
create a Security Group that opens the necessary ports on firewall settings for the Aspera On-Demand Server. 
>Go to the Security Groups in the AWS Console Navigation sidebar, click Create Security Group.
> Create a new group for the Apsera On-Demand Server, open the following ports:
HTTP - TCP/80
HTTPS - TCP/443
Custom (SSH) - TCP/33001
Custom (fasp) UDP/33001
You can use your existing key pair for the Aspera AMI. If you wish to generate a new key pair, go to the Navigation sidebar, click Create Key Pair. Enter the key pair name and click Create, the private key will be downloaded to your computer.

Step 3: Launch the Aspera AMI:
In the AWS Management Console, go to the AMIs in the Navigation sidebar, search for the AMI with the keyword aspera, or the exact name referred in the subscription-confirmation page. Launch the AMI. Choose the EC2 instance type according to your transfer needs. Note: If you need large amounts of file to be transferred in short time period then larger instance type is recommended.

Step 4: Log in the On-Demand Console:
When the Aspera AMI is launched, locate it under the Instances in the AWS Console Navigation sidebar. You can obtain the following On-Demand Server information:
With Login & Password 
Address:  Public DNS

Step 5: Transfer Files with the Aspera On-Demand Server:
When the On-Demand Server is set up, and at least one system/transfer account is created, you can transfer files with the On-Demand Server through the Web UI, fasp™ Transfer Server, or the On-Demand Console. 
If you have added the un-managed nodes into the On-Demand Console, it can be used to initiate transfers between nodes when the Console user has the permission to start transfers. It provides two types of transfer methods:
A) Simple Transfer:  one-time transfer session that requires entering all transfer info.
B) Smart Transfer: A transfer template that stores all the transfer settings and can easily be reused.

Simple Transfer:To start a Simple Transfer in the Transfer screen, click Simple Transfer. Fill out the Start a Simple Transfer form. When finished, click Transfer to initiate the transfer.

Description: Enter the name and comment of the transfer. This can be helpful if you want to search for this transfer later.
Source: Select the source endpoint and files to transfer.
Selected Source Items: The selected files and folders to transfer will be added into the Selected Source Items panel.

Destination:  Select the destination endpoint and the directory to receive the file.
Destination Directory:  The selected folder to receive the files will be added into this panel.

Console Transfer Options: In both the Simple and Smart Transfer, we can find more transfer settings in the More Options panel:
Connection: fasp™ Port (UDP)  Specify the UDP port for fasp file transfers.
Transfer >
Target rate         : Specify the transfer target rate.
Minimum rate      : Set the transfer minimum rate
Bandwidth policy : Choose a transfer policy among fixed/high/fair/low.
Retry policy        :  Check the option to enable the retry policy, and specify the number of attempts and the duration.

Session Details: We can view the session details in the console once the transfer is over.

Aspera On-Demand vs SCP Comparison Matrix:

The following table illustrates the comparison (time taken) between Aspera On-demand and SCP For transfering data between Amazon EC2 USA-EAST and  US-West region.

Note :

• 50 GB Single compressed file when uncompressed translates to 500 GB file size. 
• Default settings of Aspera On-demand on AWS was used 

Source : US-East  | Target: US-West
Instance Type	File Size / Type	Aspera On-Demand (time)	SCP (time)
m1.large	5 GB Single File Compressed	3m 21s	-
m1.xlarge	50 GB Single File Compressed	35m 32s	-
m2.4xlarge	50 GB Single File Compressed	33m 58s	2hr 1min

 References :

http://cloud.asperasoft.com/aspera-on-demand/

http://asperasoft.com