Loading Big Index Data into newly launched Amazon CloudSearch engine

Search tier is the most critical section of many online verticals like travel, e-commerce, classifieds etc. If users cannot search products efficiently they will not make their buying decisions properly, which in turn massively affects the revenues of these companies. Most of them are usually powered by Apache Solr, FAST , Autonomy, ElastiSearch etc.  AWS also has a Search Service called CloudSearch which is a fully-managed service in the cloud that makes it easy to set up, manage, and scale a search solution for your website. Amazon CloudSearch relieves you from the worry of hardware provisioning, setup, and maintenance. As your volume of data and traffic fluctuates, Amazon CloudSearch automatically scales to meet your needs.

In AWS infrastructure Apache Solr has been the king and the software to beat till now, recently it has got heavy competitor in the form of Amazon CloudSearch - API 2013-01-01. 
API version 2013-01-01 of Amazon CloudSearch is internally powered by customized version of Apache Solr Engine, and it is specifically designed for running highly scalable and available search on Amazon Web Services Cloud. This 2013 CloudSearch API has lots of similarities with Apache Solr and customers can easily migrate to this version and leverage the benefits of Amazon Cloud Infrastructure. We are already hearing many AWS customers are planning their migration from FAST, Solr and A9 Engine into the Amazon CloudSearch - 2013-01-01 API engine. 

My team is already migrating couple of customers into this Amazon CloudSearch 2013-01-01 API and i have shared our experience on this process for the benefit of AWS community.

Reference Migration Architecture and requirements:

In this article i am going to explore how to 

• Migrate a 300+ GB index containing close to 247+ million records distributed in 105 searchable fields in a highly scalable /parallel manner in AWS infrastructure.
• 300 + GB index file is stored in Amazon S3
• Custom Data loader program built on Amazon Elastic MapReduce is used for parallel loading
• Around ~6 Search.M2.2Xlarge are created with 2 partitions and 5 replication count
• Around 10+ M1.large EMR Core nodes are for Data loading. This loader can be increased to hundreds of nodes depending upon the volume and velocity of data pump required.
• Amazon CloudSearch Infrastructure provisioning, Automated partitioning, replication count are handled by AWS.   

Lets get into the details below:

Step 1)Create a new Amazon CloudSearch Domain: We have named the search domain as "bigdatasearch" and chose the search instance type as search.m2.2xlarge.  Since we are planning to pump and query a 300 GB index with millions of document, it did not make sense for us to chose a smaller instance type of Amazon CloudSearch.  Usually the base instance type can be selected based on the number and size of the documents you are planning to maintain in the Amazon CloudSearch. 
Note: Here we have chosen replication count as 5.  This is little strange in a distributed architecture because usually more replication count for the master decreases the speed of document upload. But when we were playing with Amazon CloudSearch we observed that it is increasing the speed of uploads. In addition we also observed the following :

• If we keep the replication count 0 or less, use a smaller search instance type and pump documents in parallel from multiple nodes, either the Amazon CloudSearch Server is failing sometimes or error rates are high.
• If we keep the replication count 0 or less , use a larger search instance type and pump documents in parallel from multiple nodes, internally Amazon Cloud Search itself is creating 3-5 nodes and it shows in the replication count. Waiting to discuss with AWS SA folks on this behavior.

We will be utilizing distributed uploading technique which we custom built using Amazon Elastic MapReduce to pump data to the Amazon CloudSearch server. This technique enables us to write more Index data in parallel.  

Step 2) Select how you would like to create the Amazon CloudSearch Schema: Here we have chosen Manual setup, since we already have schema to be migrated to Amazon CloudSearch.

Next step is to Add index fields to create your Amazon CloudSearch Schema configuration.

Step 3)Adding Amazon CloudSearch Index Fields: Once all the fields have been configured in the schema, click on continue button. In the schema file used we have 100+ fields to be indexed for this particular search domain.

Step 4) Review the setup configurations and launch:
We have 100+ Index fields with scaling options instance type as m2.2xlarge and replication count 5 in the "bigdatasearch" domain.

Step 5 ) Wait till the Amazon CloudSearch Infrastructure is provisioned for you on the back. Usually it takes 10 minutes, it will also list if there is any error encountered when creating the index fields.

Once the Amazon CloudSearch infrastructure is provisioned at the back end , you should notice the "bigdatasearch" domain is "Active". The search and Document endpoints are published and currently no of searchable document is "0". There is only 1 CloudSearch Index Partition (Shards) and 5 search.m2.2xlarge instances.

Step 6)Configuring Synonyms: We have 2+ MB of Synonyms which needs to be configured into the Amazon CloudSearch domain. For this, we used Cloud Search cli-toolkit to upload synonyms to Cloud Search.

cs-configure-analysis-scheme -d bigdatasearch --name customanalysisscheme --lang en -e cloudsearch.ap-southeast-1.amazonaws.com --synonyms customsynonyms.txt

Since the volume of index data is huge (300+ GB) we have created a Custom Data Loader built on Amazon Elastic MapReduce to pump the data in parallel into Amazon CloudSearch. Since it is built on Amazon Elastic MapReduce,  we can use the same program without modification for scale to upload TB's of index into the search system with hundreds of Data loader EMR core/task nodes. 

Step 7) Create Amazon Elastic MapReduce Data Loader Cluster Configuration:

Step 8) Configure the Elastic MapReduce (EMR) Capacity: We are using 10 M1.Large core node instances for uploading the data from inside AWS VPC. Depending upon the Data size (GB->TB) and Upload hours we can increase the EMR core nodes capacity and number to speed up the data pump (upload) process.

To know more about How Spot instances can save cost on Amazon EMR ? refer URL AWS Cost Saving Tip 12: Add Spot Instances with Amazon EMR

Step 9)Add Custom data loader program Jar to EMR: 
We have exported the data from a MSSQL server as flat UTF-8 dump file and stored it in Amazon S3. We are giving the 300+ GB Dump file as the input for the Amazon EMR CloudSearch Data Loader program to upload into Amazon CS in parallel. Buckets configurations of the Data Loader jar, Input, output and log files are configured in this screen

Step 10) Configure Amazon CloudSearch Access Policies:  We need to open Cloud Search security group access policies to accept upload requests from EMR cluster inside VPC. Configure static IP’s of all the instances or IP range of the data loader clients

Step 11)Run the Amazon Elastic MapReduce Data loader job :

Step 12) Analyzing the Amazon EMR Data loader Job Output:
Output of the JOB can be seen in the AWS EMR JOB logs. Here are few details:

• “Map output records” in the log tells how many records are inserted into the Amazon CloudSearch , we can observe close to 247,681,520 documents(247+ million) are pumped.
• “Bytes Read” in the output tells what is size of data set which the JOB has read. We can observe 322387978332 bytes which is equivalent to 300+ GB of index in the Amazon CloudSearch
• The entire pumping process took ~30 hours with 10 m1.large core nodes for us. We observed that increasing the number of Data loader EMR nodes or their capacity improves the upload speed drastically.

Step 13) Clean up : Reset Replication Count to level of HA needed ideally 1-2 nodes. Once the Job is completed, Revert back the Security Access Policies in Amazon cloud search. Terminate the EMR Cluster and clean any leftover resources.

Step 14) Analyzing the CloudSearch Dashboard :
We observed that it takes some time for cloud search to reflect actual count of the indexed documents.

After the pumping of 300 + GB index you can observe that currently 2 Amazon CloudSearch partitions ( shards) are used to distribute 247+ million documents with 100+ index fields. This is tremendous cost savings compared to A9 powered Amazon CloudSearch. Amazon CloudSearch has automatically created shards based on the volume of data pumped in to the system. This is cool !!!, it reduces the maintenance headache of the infra admins. If the Amazon CloudSearch team can make this partition concept as configurable parameter in future it will be useful. 

Step 15) Executing a Sample Search queries: We are executing a some sample product search queries on the "bigdatasearch" domain to check whether everything is fine. Distributed query was fired and Results came Sub Second from one of the partitions.

In short, It is cost effective compared to old A9 powered CloudSearch, Automated scaling of replication counts for request scalability, automated scaling of partitions for data scalability relieves the infra admin headaches, strong apache Solr pedigree and its long list of feature additions in coming months will make it more interesting.  

After working with this service few weeks, we feel it is going to become the major search service on AWS in coming years, giving tough fight for Apache Solr and ElastiSearch deployments on EC2. 

This article was co authored with Ankit @8Kmiles.

AWS Cost Saving Tip 12: Add Spot Instances with Amazon EMR

In continuation to my post on "How elastic thinking can save costs on Amazon EMR cluster ?" i have explored in this post how we can exploit Amazon EMR by introducing Spot EC2 into the cluster and achieve more cost savings.

Most of us know that Amazon Spot EC2 instances are usually good choice for Time-flexible and interruption-tolerant tasks. These instances gets traded frequently on a Spot market price and you can fix your Bid Price using AWS API's or AWS Console. Once free Spot EC2 instances are available for your Bid Price, AWS will allot them for use in your account. Spot instances are usually available way cheaper than On-Demand EC2 instances most of the times. Example: On-Demand m1.xlarge per hour price is 0.48 USD and on spot market you can find them sometimes @ 0.052 per hour. This is ~9 times cheaper than the on-demand price; imagine if you can bid competitively and get hold of spot EC2 even around 0.24 USD most of the times, you are saving 50% from the on-demand price straight away. In Big data use cases usually you might need lots of EC2 nodes for processing, adopting such techniques can vastly make difference in your infra cost and operations in long term. I am sharing my experience on this subject as tips and techniques you can adopt to save costs while using EMR clusters in Amazon for big data problems. 
Note : While dealing with spot you can be sure that you will never pay more than your maximum bid price per hour.  

To know more about real implementation of these tips, read the following case study. Lock, Stock and X Smoking EC2's. 

Tip 1: Make right choice (Spot vs On-Demand) for the cluster components
Data Critical workloads: For workloads which cannot afford to lose data you can have the Master + Core on Amazon On-Demand EC2 and your task nodes on Spot EC2. This is the most common pattern while combining Spot and On-Demand on Amazon EMR cluster.  Since task nodes are operating on spot prices depending upon your bidding strategy you can save ~50% costs from running your task nodes using On-Demand EC2. You can further save(if you are lucky) by reserving your Core and Master Nodes , but you will be tied to an AZ. According to me this is not a good or common technique, because some AZ's can be very noisy with high spot prices.  
Cost Driven workloads: When solving big data problems, sometimes you might have to face scenarios where cost is very important than time. Example: You are processing archives of old logs as low priority jobs, where cost of processing is very important and usually with abundant time left. Such cases you can have all the Master+Core+Task run on Spot EC2 to get further savings from the data critical workloads approach. Since all the nodes are operating on spot prices depending upon your bidding strategy you can save ~60% or more costs from running your nodes using On-Demand EC2. The below mentioned table published by AWS gives an indication of the Amazon EMR + Spot combinations that are widely used:

Tip 2: There is free lunch sometimes
Spot Instances can be interrupted by AWS when the spot price reaches your bidding price. What interruption means is that, AWS can pull out the Spot EC2's assigned to your account when the price matches/exceeds. If your Spot Task Nodes are interrupted you will not be charged for any partial hour of usage by AWS i.e. if you have started the instance @ 10:05 am and if your instances are interrupted by spot price fluctuations @ 10:45 am you will not be charged for the partial hour of usage. If your processing exercise is totally time insensitive, you can keep your bidding price at closer level to spot price which are easily interrupt-able by AWS and exploit this partial hours concept. Theoretically you can get most of the processing done through your task nodes for free* exploiting this strategy.

Tip 3: Use the AZ wisely when it comes to spot
Different AZ's inside an Amazon EC2 region has different spot prices for the same Instance type. Observe this pattern for a while, build some intelligence around the price data collected and rebuild your cluster in the AZ with lowest price. Since the Master+Core+Task need to run on the same AZ for better latency, it is advisable to architect your EMR clusters in such a way they can be switched(i.e.recreate) to different AZ's according to spot prices. If you can build this flexibility in your architecture you can save costs by leveraging the Inter AZ price fluctuations. Refer the below images for Spot Price variations in 2 AZ's inside the same Region for same time period. "Make your choice wisely time to time"

Tip 4: Keep your Job logic small and store intermediate outputs in S3
Breakdown your complex processing logic into small jobs and design your jobs and tasks in EMR cluster in such a way that they run for very small period of time (example few minutes). Store all the intermediate job outputs in Amazon S3. This approach is helpful in EMR world and gives you following benefits:

• When your Core+ Task nodes are interrupted frequently, you can still continue from the intermediate points. Data accessed from S3.
• You now have the flexibility to recreate the EMR clusters in multiple AZ depending upon the Spot price fluctuations
• You can decide the number of nodes needed for your EMR cluster(even every hour) depending upon the data volume, density and velocity

All the above 3 points when implemented contribute to elasticity in your architecture and there by helps you save costs in Amazon cloud. The above recommendation is not suitable for all Jobs, it has to be carefully mapped with right use cases by the architects.

To know more about real implementation of the above tips, read the following case study. Lock, Stock and X Smoking EC2's. 

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

Lock, Stock and X Smoking EC2's - Cloud Connect 2013 Presentation

This case was presented @ Cloud Connect 2013 . You can view this in Slide share as well.

To know more about Amazon EMR + Spot refer the following articles. They will set the technical context right before you go through this presentation. Posts are listed below:

Post 1: How elastic thinking saves cost in Amazon EMR Clusters ?
Post 2: Add Spot Instances with Amazon EMR

Cloud Connect 2013- Lock Stock and x Smoking EC2's from Harish Ganesan

Case :
Mobile advertising company based in USA. They have Forbes 1000 clientele using their mobile advertising product. The company approached us for architecture and implementation help for their Clicks, Ad and other analytic's hosting and processing on AWS.

The LOCK ?
Variety of unstructured logs and semi structured files have to be processed for data analysis. It includes logs from CDN, server logs,  XML Files, Text files, Geo Data Files and Structured DB records. The logs flow every hour from variety of sources into AWS. It amounts to few hundred GB's -> ~ 1 TB @ peak hours/seasons.

The STOCK ?
Mobile Ad company wanted an efficient architecture and infrastructure on AWS for collecting, storing, processing and share the data. They wanted to avoid cost leakages because of bad architectural practices in AWS and save $$$ wherever possible.

Challenges:
Their analysis patterns can be categorized into Hourly, Monthly and Historical. Some of the major challenge were:

• How do we Transfer, Store, Analyze and Share ?
• How to optimize costs at this scale ?

Solution:

• Architect their entire front end analysis module in AWS. Use the AWS technologies like ELB, EC2, R53, RDS ,Cache for delivering their analysis  results to online users.
• Use Amazon Elastic MapReduce with Spot EC2 instances for their back-end processing jobs.
• Automate their infrastructure using Chef, Scripts and Custom Java Modules wherever applicable. 

Stage 1: Transfer the Data :
First time data was transferred using AWS Import/Export. On going data was transferred to AWS using Tsunami UDP. High Bandwidth EC2's were installed with Tsunami UDP on AWS side for receiving data faster. The Data collected is stored temporarily in the Receiver EC2's , some pre-processing tasks were run on them and files are moved to S3 Buckets post that.
For more details about using Tsunami UDP and Aspera on AWS refer these articles: Tsunami UDP on AWS and Aspera on AWS

Some of the other popular models are :
AWS Direct Connect: Establishes a dedicated connection between Data centers to AWS using Direct Connect. This model is little costlier and is suitable only enterprises.
WAN Optimization: Use WAN optimization tools like Riverbed Steel head, aryaka etc to speed up the transfer between source endpoints to AWS.

Stage 2: Storage of Data:  
Temporarily data was stored on Receiver EC2's. After Pre-processing tasks they were moved to S3. Amazon S3 is a default choice because of its inherent fault tolerance and scalability features. Around ~2 TB of compressed logs are stored in Amazon S3 Daily for processing. Amazon S3 Reduced Redundancy Storage option was used for storing intermediate log outputs. S3 Automatic Object Expiry was used for efficiency and cost savings. Archival data is moved to Amazon Glacier periodically.

Stage 3: Analysis using Amazon EMR + Spot EC2
Amazon EMR was used for log processing and analysis. Amazon Elastic MapReduce (EMR) is a web service that helps customers with big data processing using Hadoop framework on EC2 and S3. Amazon Elastic MapReduce lets customers focus on crunching data instead of worrying about time-consuming set-up, management or tuning of Hadoop clusters or the EC2 capacity upon which they operate. This in built automation provided by AWS already saves huge labor cost for the customers. At peak hours some jobs run with 2000 Mappers/ 750 Reducers. For peak periods ~250 m1.xlarge or equivalent EC2 capacity was used for processing the logs. We developed a Custom EMR manager to introduce Spot EC2 into the Amazon EMR equation. Our Spot Bidding strategy was either on-Demand price or 20% above On-demand price. Since the Spot prices vary in different AZ's we collected the past price history, current market price etc and chose the right AZ's appropriately. Choosing right AZ sometimes mean creating the entire EMR Cluster on low priced AZ (All Master,Core & Task should reside on Same AZ).  The Master and Core Nodes were running On-Demand EC2 and Task Nodes were running on Spot EC2 or On-Demand EC2. The Custom EMR Manager has the capability to increase the Core nodes depending upon the Log Data Volume + Create the entire EMR cluster (with revised Core+task Node numbers) in New AZ depending upon the Spot prices. This decision was made by the Custom EMR manager periodically depending upon the log data volume fluctuations and Log Volume patterns.   

Learning:

• Spot+On Demand EC2 for EMR is deadly combination for Cost Savings. Though this combination is suitable for Time in-sensitive jobs, but if some intelligence and imagination can be applied in your architecture and design, you can use this for certain time sensitive jobs as well.
• Bigger files are better, so we merge the files into Bigger chunks before proccesing in Amazon EMR.
• In Pre processing stage, we split files with ratio of 1 file per mapper. This gives us better time predictability during processing. Also the file transfer was faster between S3 and EMR because of this manageable size splits.
• Data is compressed at all possible levels. Snappy and .lzo compression was used.
• We dynamically increase/decrease the task nodes using the Custom job manager. If no Spot EC2 available for Task nodes, then Custom EMR manager adds on-Demand task nodes to the cluster.
• The jobs were designed to be small in processing logic size and intermediate output data is stored in S3. This way we can re-create the cluster with reproducible data. Only Processing data was kept in EMR cluster, rest of the data were kept in S3.
• The EMR clusters can be sized according to the log data volumes. This intelligence was built in the custom EMR manager based on the past patterns, Data Density tests etc, during pre processing phase.
• Certain Intermediate reducers were designed to create number of result files according to number of Mappers in next jobs. This helped us in optimum utilization of capacity inside a hour.
• Decision to re-size clusters are made nearest to the hour. If AWS brings per minute pricing , it will help such use cases to save more costs.
• Every API call and millisecond matters in big data programming, tune the MR code and test the 3rd party APIs for performance before integrating them into your code.
• Understand NW capacities of to/fro data transfer to S3, m1.Xlarge EMR Nodes (Memory Sizing, Mapper / reducer numbers) and work leveraging the strengths of AWS.
• If your processing requirements are totally time insensitive, you can bid very close to spot price and have AWS interrupt(mostly) within the hour and literally get entire processing stuff done for free. (If AWS interrupts the Spot EC2 you will not be charged for any partial hour of usage). This strategy was not used by us. If any of you use this, please write to me, will be happy to discuss.

Value :

• EMR with Spot brought ~56% cost savings from pure On-Demand model for Core+ Task Nodes.
• Customer CXO's were happy !!!

AWS Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ?

Introduction: Amazon Elastic MapReduce (EMR) is a web service that helps customers with big data processing using Hadoop framework on EC2 and S3. Amazon Elastic MapReduce lets customers focus on crunching data instead of worrying about time-consuming set-up, management or tuning of Hadoop clusters or the EC2 capacity upon which they operate. This in built automation provided by AWS already saves huge labor cost for the customers.

What does the word Elastic mean in Hadoop/EMR context ? Ans: You can dynamically increase the number of processing nodes depending upon the volume/velocity of the data. Adding or removing servers takes minutes, which is much faster than making similar changes in clusters running on physical servers. Let us explore this in detail and analyse how it will help you save costs in AWS Big data processing. 

Components:
Before getting into the savings part, lets understand the composition of an Amazon EMR Cluster. An Amazon EMR cluster consists of following server components. They are:
Master Node: This node Manages the cluster, it coordinates the distribution of the MapReduce executable and subsets of the raw data, to the core and task nodes. There is only one master node in a cluster. You cannot expand or reduce your Master Node in the EMR Cluster.
Core Node(s): A core node is an EC2 instance that runs Hadoop map/reduce tasks and stores data using the Hadoop Distributed File System (HDFS). Core nodes are managed by the master node.  You can add more core nodes to running cluster, but you cannot remove them from a cluster because since it stores data you have an risk of losing data. 
Task Node(s): As the name suggests these nodes run tasks and they map to equivalent of Hadoop slave node.  These nodes are optional in nature.  Task nodes are managed by the master node. While a cluster is running you can increase and decrease the number of task nodes. Because they don't store data and can be added and removed from a cluster, you can use task nodes to manage the EC2 instance capacity your cluster, by increasing capacity to handle peak loads and decreasing it later when there is no load.

Analysis :
Imagine the log volume flow is not constant and it varies every hour, some hours you receive few hundred GB's and some hours few GB's of logs for processing. For peak hours your use case needs around 192 mappers/72 reducers and normal hours you need ~64 mappers/24 reducers or less. The peak and normal numbers can be arrived based on the analysis done on the past data. This elasticity in log volume scenario is a usual occurrence in many big data projects and it is source of cost leakage. Simple approach what many architects take is that they run their cluster infrastructure @ peak capacity always since the operation is time sensitive, but this might not be an optimal approach in amazon cloud-big data world. Since you can elastically increase/decrease the number of nodes in an Amazon EMR cluster it is optimal if you can size the number of nodes dynamically every hour. Since you pay by usage in amazon cloud, having this elasticity built in your architecture will save costs. 

Based on the number of mappers/reducers required, we have chosen the node capacity to be in m1.xlarge EC2 units. So during Peak hours you will need 24 processing nodes and normal (avg) hours it will be reduced to 8 processing nodes. 

Elastic Approach-1: Vary the Task Nodes: 
In this approach, number of Master and Core nodes are maintained constant. 1 - master node and 4- core nodes are used for processing and data storage always. The task nodes are increased and decreased between 4->20 every hour depending upon the log volume flow. Since the data is present in the core nodes and only tasks/jobs are assigned in the task nodes, adding/removing task nodes will not cause problems. You can engineer a custom Job manager using AWS API's and manage this entire cluster easily. If you do a simple math that in average only 8 processing nodes are needed (4 core + 4 task nodes) and during peak hours you need ( 4 core + 20 task nodes) with this approach you can save ~60 % costs by not running your cluster in ALWAYS peak capacity. This model is a recommended approach for many elastic big data use cases in AWS. Refer the below table for cost savings:

Scenario	No. of Processing Nodes	Hourly rate	Node Type	Monthly
Peak hours	24	0.48	M1.Xlarge	~8570.88
Normal hours	8	0.48	M1.Xlarge	~2856.96

  

Elastic Approach-2: Vary both the Core and Task Nodes:
In this approach, the number of both Core and Task nodes are varied dynamically. Since the Core nodes can be only increased and cannot be decreased in a running cluster(because it could lead to data loss), this approach is recommended only for advanced use cases. Since the entire data is stored in S3(is reproducible) and can be moved to the EMR cluster every hour, using the custom Job manager an entire cluster can be created(even every hour) depending upon the log data volume (GB's). Example: Imagine first hour: 4 Core + 10 Task nodes are used for processing, second hour data volume is increased and 4 Core + 20 Task nodes are added in the cluster, third/fourth hour etc there is hardly few GB's of data flow and only 8 Mappers/3 reducers are needed for processing, instead of running 20 task + 4 core nodes(of prev hour),  a new EMR cluster can be created with just 1 master and 1-2 Core nodes. This approach requires engineering a custom job manager using AWS API's for managing the cluster. Though this approach is little complex to engineer, it saves more cost than approach-1 on medium to long term.

Note:The approaches illustrated are not theoretical in nature. I have put both the above techniques to production use for some customers and they are already seeing huge cost savings. 

Coming Soon - Adding Spot to this equation gives brutal savings ... 

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Top 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Top 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Top 12: Add Spot Instances with Amazon EMR 
Cost Saving Top 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Top 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Top 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

Amazon RedShift - Is it keeping its promise ?

Amazon launched much awaited RedShift @ AWS re:invent last year. Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service. It comes under Massively Parallel Processing (MPP) system and it complements and works well with the EMR and DynamoDB products of AWS. It is optimized for datasets ranging from a few hundred gigabytes to a petabyte and it is simple and cost effective.Most attractive part of Amazon RedShift is its promise to cost less than than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions in the market currently.
Amazon Redshift was customized by AWS team to deliver queries faster and give good I/O performance for virtually any size dataset by using columnar storage technology and parallelizing + distributing queries across multiple nodes. AWS team made Amazon Redshift easy to use by automating most of the common administrative tasks associated with provisioning, configuring, monitoring, backing up, and securing a data warehouse. This automation in turn saves lots of costly labor cost associated with the above activities in maintaining and operating a MPP system.

Recently AK team analysed and tested Amazon Redshift for its promise and have published a detailed report on their blog. They tested variety of Amazon RedShift nodes with data sets ranging from 80GB to 2.4TB in size equating to 2B -> 57B rows in the main fact table.

Some of the points observed by AK team were :

• Starting an Amazon RedShift cluster, regardless of what size and Instance type, took between 3 and 20 minutes. 
• Each Amazon RedShift xlarge node could load about 3.17MB/sec of compressed S3 data or 78k rows/sec and each 8xlarge node could load about 23.8MB/sec or 584k rows/sec.
• The overall COPY speed scaled linearly with cost and with data size, measured over hundreds of chunked loads.
• For loading, AK team observed linear scaling per dollar and observed just a bit worse than the 8:1 price ratio between the small and big nodes. 
• Backing up 4-5TB Amazon RedShift cluster reliably took 2-3 hours, they were getting about 400MB/sec to S3 from their 2-node 8xlarge/16-node xlarge clusters
• AK team saw an effective rate of about 175MB/sec transfer when upgrading from 2-node 8xlarge/16-node xlarge to 4-node 8xlarge/32-node xlarge clusters, meaning they looked up the amount of space Amazon Redshift reported using on disk and then divided that by the time between hitting the resize button and being able to run the first query on the new cluster. For the data sets, that meant 6-7 hours of degraded query performance (but not downtime!) followed by a seamless transition.

I enjoyed reading this excellent analysis and this article is a must read for Amazon RedShift users :  You can find the original article at 

http://blog.aggregateknowledge.com/2013/05/16/aws-redshift-how-amazon-changed-the-game/

- Thanks AK team for the detailed Analysis

Cloud, Mobile and Big Data Stories: Oct- Dec 2012

Interesting Stories(tweets) on Cloud, BigData and Mobile which you may have missed between Oct - Dec 2012. Collection of my tweets, RT, happenings and thoughts from Industry leaders during this period.

1. Top 100 Business, Leadership and Technology Twitter Accounts You Must Follow http://huff.to/RMK57u  #socialmedia
2. Big Data: Big Data Can Fight Fraud in Health Insurance Exchanges: Opera http://dataaware.net/bigdata/20514 
3. It takes much for than 10,000 hours to master the art of listening, don't kid yourself. @fastcompany #lessons2013
4. How the Cloud Changes Software Consulting - http://NYTimes.com  http://ow.ly/goIdi  RT:@mtrzil
5. By 2020, there will be 5,200 GB of data for every person on Earth --> http://www.computerworld.com/s/article/9234563/By_2020_there_will_be_5_200_GB_of_data_for_every_person_on_Earth …
6. @ForbesTech: How #cloud computing is accelerating context-aware coupons, offers and promotions http://bit.ly/RXj6q1 
7. 16 of the most useful #cloud management tools http://shar.es/hQpx4  RT:@agflem
8. "Percona and the MariaDB Foundation" http://dzone.com/NOLP 
9. Economics of Choosing Amazon ElastiCache Node Type http://bit.ly/ZAUM0c  #aws #elasticache #caching #cloud
10. Economics of Choosing Amazon ElastiCache Node Type http://bit.ly/ZAUM0c  #aws #elasticache #caching #cloud
11. Part 4: Understanding Amazon ElastiCache Internals: Economics of Choosing Cache Node Type http://bit.ly/ZAUM0c  #aws #elasticache
12. Making the Case for Cloud - http://slidesha.re/Wv980r  -> One of the best #cloud slide sets ever by @jinman #aws
13. To make your holiday complete, a developer preview of the new #AWS command line interface: http://amzn.to/12F8op2  - 1 tool to rule them all.
14. #AWS CloudFormation Templates for Visual Studio and #Eclipse: http://bit.ly/XVZlvM  #visualstudio
15. The #AWS Data Pipeline is now available for use: http://bit.ly/U1Wobx  - Process and move data, EC2 and on-premises.
16. The new #EC2 High Storage Instance Family - http://bit.ly/12tMbLp  - 48 TB of instance storage, 120 GiB of RAM, 35 ECU) #aws
17. #Riak #opensource #database is optimized for use on #AWS by @Basho. Try it now on @AWSMarketplace http://amzn.to/UNEt9h  #bigdata
18. Checking out "How journalists are using data" on AnalyticBridge: http://ning.it/TdYkm7 
19. Cloudant On Rackspace: Data Layer For The Big Apps Era: Rackspace customer and Cloud Tools par... http://bit.ly/SUmvWw  #cloud @rackcloud
20. Google Analytics Metrics and Dimensions http://www.lunametrics.com/blog/2012/12/20/google-analytics-metrics- …
21. Job Stability vs. Job Satisfaction? http://bit.ly/TfxiHC 
22. StarChart - Growth metrics for 1,730 Australian startups. - proud that #AWS powers many of those cool companies /via HN http://wv.ly/TpYfZ0 
23. New for Amazon CloudSearch - Analytics Dashboard and Real-Time Rank Expression comparison: http://amzn.to/V5weIq  #aws
24. New analytics and rank comparison tools for #AWS CloudSearch: http://aws.amazon.com/cloudsearch/whats-new/ …
25. Building Amazon #cloud apps that span the world is now much easier http://ow.ly/gfps3  #AWS
26. Ousted Windows exec Steven Sinofsky to teach at Harvard Business School this spring. http://bit.ly/ZlUH08 
27. Big Graph Data on Hortonworks Data Platform: http://hortonworks.com/blog/big-graph-data-on-hortonworks-data-platform/?utm_source=twitter&utm_campaign=cpt&utm_medium=status … …
28. Amazon watcher Newvem starts charging to monitor your cloud http://bit.ly/ZiLo1e  #AWS #Cloud
29. RT @christillian: Single pane of glass across your clouds via @ComodIT: Unlock the power of the Hybrid Cloud http://www.comodit.com/2012/12/18/hybrid-cloud-orchestration-with-comodit/ …
30. What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud.
31. I am almost finish with article for IT World has interview with all 7 customer of Azure cloud.
32. In devops only certification is when you have work 48 hour with not sleep.
33. Little know fact is 90% of Big Data is result in No Data.
34. In devops is success 5% inspiration, 15% git clone, 25% copy and paste and 55% bulls shit.
35. 90% of problem in devops can able solve with one of 3 thing: 1) SSD 2) CDN 3) AWS
36. Only 10% in devops are know how of work with Big Data. Only 1% are realize they are need 2 Big Data for fault tolerance.
37. Advantage of Agile is can able of allow you get nowhere much faster.
38. Introducing IT chargeback to meter private cloud use < Our enterprise reporting is a step in the right direction http://ow.ly/gdqxM 
39. Top 12 Big Data Stories of 2012 http://trib.al/VPdkDmx  @CIOonline
40. If you missed any of my 2012 posts on #Agile, #BigData, or #CIO / IT #leadership, this post sums it up - http://blogs.starcio.com/2012/12/big-data-agile-cio-2012-in-review.html …
41. 10gen's MongoDB Monitoring Service (MMS) is a tool designed to help develop and manage MongoDB applications. http://soc.ai/2eP 
42. How tomorrow’s mobile-centric data centers will look http://dlvr.it/2g6pxr  @gigaom
43. Vendor neutral #Cloud ROI Calculator from @MomentumSI. Download now! http://hub.am/12nuEEO 
44. Read Preferences with the MongoDB PHP driver http://buff.ly/Tw44Ej 
45. 7 Companies That Are Capitalizing on the Mayan Apocalypse http://feedproxy.google.com/~r/adrants/~3/PJ9uKP_C7zk/7-companies-that-are-capitalizing-on-the.php …
46. RT @newvem: Amazon watcher Newvem starts charging to monitor your cloud http://lnkd.in/dVyynA  < congrats!
47. Why you should expect more online outages but less downtime http://gigaom.com/cloud/why-you-should-expect-more-online-outages-but-less-downtime/ …
48. High Performance Web Applications by @awscloud http://www.slideshare.net/AmazonWebServices/high-performance-web-applications … #AWS #Cloud #Amazon
49. RT @gigaom : Amazon adds storage snapshot feature to bolster disaster recovery - http://gigaom.com/cloud/amazon-adds-storage-snapshot-feature-to-bolster-disaster-recovery/ …
50. Great resource! 8 IBM white papers on cloud not to be missed: http://bit.ly/RoF0SC  via IBM Benelux cloud leader @schoutene
51. #bigdata alert: Oracle Upgrades NoSQL Database, Big Data Appliance - Software - - InformationWeek http://bit.ly/UHXEAD 
52. In case you missed it, new blog posted last evening: "Cloud Predictions and Trends for 2013" http://www.neovise.com/cloud-computing-predictions-for-2013 … #cloud
53. 3 Data Integration Predictions for 2013 | Data Integration Blog http://www.dataintegrationblog.com/data-integration-david-linthicum/3-data-integration-predictions-for-2013/ …
54. New! EBS Snapshot Copy. Perfect for global expansion, disaster recovery and cross-region app migration: http://aws.typepad.com/aws/2012/12/ebs-snapshot-copy.html … #aws
55. RT @DavidLinthicum 3 Data Integration Predictions for 2013 | Data Integration Blog http://bit.ly/T5UHMO  < good points
56. #Cloud disaster recovery: 5 key steps to secure your data http://bit.ly/UNSZOE 
57. I greatly appreciate the wonderful reviews of our new book, thank you all http://amzn.to/QhAEd1 
58. 2013 Predictions for Big Data http://prsm.tc/MO8yLS  via @prismatic
59. AWS announces EBS Snapshot Copy, a Step towards Easier Data Center Migration and Disaster Recovery #cloud #aws
60. Google’s Gmail Outage Is a Sign of Things to Come - Businessweek http://buswk.co/WkXdCh 
61. In case you missed it: #MySQL monitoring with CopperEgg custom metrics (Live Demo): http://bit.ly/UBVYcM 
62. Are Cloud Data Security Fears Overblown? A Sensible View. - Forbes http://onforb.es/SV6de0  #cloud #security
63. #AWS Week in Review - December 10, 2012 - http://bit.ly/YcuOi1  - Spotathon, new case studies, AWS Billing Report, Auto Scaling, Marketplace.
64. .@CAInc releases top 5 IT predictions for 2013. http://bit.ly/WivnGz  #ITpredictions #bigdata #IAM #mobile
65. DJ Patil: Data Science is Storytelling #datastream #blog http://ow.ly/g7kDD 
66. TechNet Radio: Cloud Innovators – (Part 4) Private Cloud IaaS http://bit.ly/ULUo8o  #cloud #Azure #fb
67. Volvo signs with Ericsson for cloud-based in-car apps, probably including Spotify http://dlvr.it/2fVLhw  #cloud
68. SaaS valuation boom slowed but remained strong in 2012 — #cloud Computing ... - GigaOM http://dlvr.it/2fVMbX 
69. Big Data's Surprising Uses: From Lady Gaga To CIA http://dld.bz/bTAyh  #BigData
70. Why HP, Dell and IBM Are on the Wrong Side of Internet History http://prsm.tc/lpqySC 
71. #BigData for Health Care infographic - where data originates, how it moves, & 6 ways it can transform healthcare http://bit.ly/T2C0bL 
72. Top CEOs Share how Big Data Is Transforming Our Health, Wealth and Security #bigdata http://bit.ly/UDBXll 
73. Autodesk embraces cloud computing - http://bit.ly/Uc5XTZ  #cloud #cloudComputing
74. Reading: "Does MPLS Make Sense in #Cloud Data Centers?" by Kireeti Kompella ▸ http://bit.ly/Vx11KC  #CloudComputing #SDN
75. In B2B, customer is 60% through buying decision before contacting #sales http://huff.to/SX2X1B  #business #CMO
76. The CTO of @GigaSpaces, @natishalom, predicts a major #Cloud & #BigData shakeout is coming in 2013 ▸ http://bit.ly/11L7CY5  #CloudComputing
77. According to 7P, the #cloud "makes economic sense" for all of its customers: http://ow.ly/g3xTS  #VMware
78. Jongo .3 Released http://buff.ly/TqWOcV  #java #nosql #mongodb
79. #Cloud Predictions for 2013 via @Forrester: http://spr.ly/6017pcMD  #cloudcomputing
80. Why Cloud Computing Projects Fail by @DavidLinthicum on @slideshare http://www.slideshare.net/Linthicum/why-cloud-computing-projects-fail …
81. So What If You Have Big Data? Without Data-Driven Processes And Products, It’s Useless http://techcrunch.com/2012/12/15/so-ive-got-big-data-now-what/ … @TechCrunch #bigdata
82. Big Data News of the Week: The Business of Organizing the World's Data - Forbes http://onforb.es/UIBqQ6 
83. Gartner: Amazon, HP cloud SLAs are http://shar.es/hHBMQ  via @networkworld
84. What's Amazon's enterprise strategy for the cloud? http://shar.es/hHT1j  via @networkworld
85. Enterprise Big Data Cloud Launched: Infochimps Enterprise Cloud #bigdata http://bit.ly/QZLCFE 
86. Optimizing Your MongoDB Database on AWS - YouTube http://cln.gs/SWs1Vo  #reinvent #aws #mongodb #nosql
87. Ray Kurzweil joins Google to work on machine learning, language processing http://gigaom.com/2012/12/14/ray-kurzweil-joins-google-to-work-on-machine-learning-language-processing/ … by @elizakern
88. Your company is only as good as the company you keep. @fastcompany
89. Big Data Really Has Helped Big Business http://rww.to/Wbl9nB  #bigdata
90. Log archive & analysis with Amazon S3 and Glacier - Part I http://www.raghuramanb.com/2012/12/aws-log-archive-amazon-s3-glacier-part-1.html … … #aws #cloud #s3 #glacier @raghuramanb
91. Log archive & analysis with Amazon S3 and Glacier - Part I http://www.raghuramanb.com/2012/12/aws-log-archive-amazon-s3-glacier-part-1.html … #aws #cloud #s3 #glacier
92. Reading: "Transforming Three Vs of #BigData into Three Ws of Business Analytics" by @tonycosentinovr ▸ http://bit.ly/RqG2xo  #Cloud
93. MT: My interview at SysconTV during @CloudExpo "@Akamai Director Gary @Ballabio talks to SYS-CON.tv http://bit.ly/UEbAwk  #Cloud"
94. Dell Declares OpenStack its Primary Cloud Platform - Dell just released a technical preview of its private cloud wid... http://ow.ly/2tJZTp 
95. @samsaffron @igrigorik of course you can ! A detailed post about HAProxy and websocket: http://blog.exceliance.fr/2012/11/07/websockets-load-balancing-with-haproxy/ … just add bind :443 ssl crt foo
96. Log archive & analysis with Amazon S3 and Glacier - Introduction @raghuramanb http://www.raghuramanb.com/2012/12/log-archive-analysis-amazon-s3-glacier.html …
97. @Gartner_Inc predicts 10% of overall #IT #security will be delivered in the #cloud by 2015: http://spr.ly/6015pYJ3 
98. Be the best, but get ready to be attacked. Only mediocrity is safe. - @paulocoelho
99. Architecting High Availability @ Load balancing layer in AWS http://harish11g.blogspot.in/2012/10/high-availability-haproxy-amazon-ec2.html … #aws #elb #cloud
100. Comparison Analysis:Amazon ELB vs HAProxy on Amazon EC2 http://harish11g.blogspot.in/2012/11/amazon-elb-vs-haproxy-ec2-analysis.html … #aws #elb #cloud
101. This Is What You Talked About on Twitter in 2012 http://bit.ly/YX2iSU 
102. Data Science: Big data skills Should data scientist be your next job | TechRepublic http://dataaware.net/datascience/20154 …
103. Every programmer should create, at least once: OS, editor, database, roguelike, interpreter, compiler and robot.
104. #cloudBees a Finalist for 2012-13 #cloud Awards - The Herald | http://HeraldOnline.com  http://dlvr.it/2csC3Z 
105. AWS Security Group Whitelist App with DynamoDB http://bit.ly/TS0Z3j  #challenge
106. 100% of marketers agree brand building is a company-wide effort, but only 2 in 5 have a consistent approach http://s.forr.com/fb6S 
107. Using Big Data to suck the life out of my son, and money from my pocket (true story) http://buff.ly/RmWNcQ  <My @regvulture piece
108. Survey Sees Big Year Ahead for Big Data - Enterprises are recognizing big data as mission critical, increasingly nee... http://ow.ly/2tGApU 
109. Webinar with @Eucalyptus tomorrow at 2 PM ET. Learn how to succeed with #hybridclouds http://ow.ly/g0qyx 
110. 13 Big Data Vendors To Watch In 2013 -- InformationWeek http://ow.ly/g0CzY  #bigdata
111. Azure blob storage and security - Sarbanes Oxley requirements http://bit.ly/VAEYCh  #cloud #Azure #fb
112. Redefining the value of data - disruptive trends 2012-2015 http://bit.ly/UeOdZ8  @briansolis [human side] #CMO #CIO #socbiz
113. Big data is the future focus of the enterprise #cloud http://bit.ly/UzruYR 
114. Launching Amazon ElastiCache in 3 Easy Steps http://bit.ly/10PkNGQ  #aws #elasticache #caching #memcached
115. Caching architectures using Amazon ElastiCache & Memcached http://bit.ly/UkpiZl  #aws #elasticache #memcached
116. Amazon ElastiCache - Tips to scale out Nodes: http://bit.ly/RkUdUW  #aws #elasticache #caching #memcached
117. Figuring out AWS pricing giving you a headache? Check out AWS Costs Cheat Sheet by @cloudvertical - http://bit.ly/awscsts  #cloud #aws #ec2
118. if u wanna know how it feels to have a tank roll over ur foot, join a start up. If u want the tank to roll over all ur limbs, b the founder.
119. "This CEO Is Creating The Next Billion Dollar Open Source Company" http://ow.ly/g02WE  << Good presentation and quotes by @martenmickos
120. GigaSpaces survey shows need for tools for fast big data, strong interest in big data in cloud - http://ow.ly/fZLEx  #cloud
121. #bigdata alert: Is 'Big data' red hot, and so not, for 2013? - Upstart (blog) http://bit.ly/Ude2ZG 
122. Lessons From The Hobbit: #cloud Computing Education - #cloudTweaks News http://dlvr.it/2ckDSq 
123. Distributed Cache On Steroids: Amazon ElastiCache http://www.sitepoint.com/amazon-elasticache-cache-on-steroids/ …
124. Amazon ElastiCache and Consistent Hashing: http://bit.ly/RkUdUW  #aws #elasticache #caching #memcached
125. Drivers Of Cloud Computing Adaption http://www.cloudtweaks.com/2012/11/drivers-of-cloud-computing-adaption/ …
126. 10 Industries That Will Gain From Adopting The Cloud | CloudTweaks http://www.cloudtweaks.com/2012/09/10-industries-that-will-gain-from-adopting-the-cloud/ …
127. Big data gets Amazoned with Redshift http://bit.ly/Szjk2W  — yep, it massively reduces the cost comparison point or data warehousing
128. The customer may not be right all of the time, but they always have the right to their opinion. Be respectful. #custserv
129. Distributed Cache On Steroids: Amazon ElastiCache http://www.sitepoint.com/amazon-elasticache-cache-on-steroids/ … via @sitepointdotcom
130. Amazon ElastiCache and Elasticity Implications explored : http://bit.ly/RkUdUW  #aws #elasticache #caching #memcached
131. AT&T Partners With Akamai on CDN Suite Akamai Technologies (AKAM) and AT&T announced a strategic alliance to deliv... http://ow.ly/2tFjvx 
132. Dear manager, if you're not promoting people, you're not leading people. A leader's #1 job is to create more leaders.
133. "Doesn't matter if the glass is half empty or half full. All that matters is that you're the one pouring the water." - @mcuban v @aimeegroth
134. Amazon adds big data cloud functions, and drops its prices! http://www.thecloudcircle.com/article/amazon-adds-big-data-cloud-functions … - #cloud #Amazon #AWS
135. Samsung captured 21.8% market share based on shipments in 3Q12. Apple led all vendors in value with at $34.1 billion. http://bit.ly/X05AhB 
136. Zenprise Acquisition Strengthens @Citrix's BYOD Stack - UCStrategies http://dlvr.it/2cXFFd 
137. Developers and operations are still very much stuck in their own worlds. How can we bridge the gap? http://ow.ly/fXmBj 
138. “@Dana_Gardner: #GigaSpaces survey shows need for tools for fast big data, strong interest in #bigdata in #cloud http://bit.ly/VLNcO0 ”
139. Why Google Ingress Augmented Reality game is a data gold mine http://bit.ly/TMbimj 
140. Will Adobe's new cloud strategy pay off? http://bit.ly/UPOa9K  #Cloud #CloudComputing
141. #Webcast Recap by @TimGasper : Top Strategies for Successful #BigData Projects http://bit.ly/VIfHMF 
142. Techblog - all the Netflix videos and slides from AWS Re:Invent http://techblog.netflix.com/2012/12/videos-of-netflix-talks-at-aws-reinvent.html … …”
143. Huawei invades Nokia’s home turf with $90m R&D investment http://gigaom.com/europe/huawei-invades-nokias-home-turf-with-90m-rd-investment/ …
144. Amazon VPC Essentials: http://goo.gl/FEjt9  #aws #vpc #ec2
145. Silicon Alley - Amazon Is Offering A $50 Discount On The Kindle Fire HD (AMZN): Amazon is offering a $50 discoun... http://bit.ly/Z55XN9 
146. Hadoop on Windows Azure: Hive vs. JavaScript for processing big data: Hadoop, which was designed for this purpos... http://bit.ly/VLrWrP 
147. Session Synchronization patterns in AWS http://bit.ly/WItXCc  #aws #session #rds #terracotta #elasticache #dynamodb
148. Caching architectures using Amazon ElastiCache & Memcached http://bit.ly/UkpiZl  #aws #elasticache #memcached
149. Launching Amazon ElastiCache in 3 Easy Steps http://bit.ly/10PkNGQ  #aws #elasticache #caching #memcached
150. Distributed Cache On Steroids: Amazon ElastiCache http://bit.ly/TPemkW  #aws #elasticache #caching #memcached
151. Amazon ElastiCache and Elasticity Implications explored : http://bit.ly/RkUdUW  #aws #elasticache #caching #memcached
152. Cloud: Top 10 news stories of 2012: CloudStack vs. OpenStack and more http://cloudaware.net/cloud/41905 
153. The White House Proves That Everyone Needs Social Media! http://bit.ly/RUZOkA 
154. 7 Predictions for Cloud Computing in 2013 That Make Perfect Sense http://onforb.es/XIFtkl  #cloud
155. loads of hype around this, but makes sense of it #bigdata http://bit.ly/SKDzLh 
156. "Big Data and Cloud Computing [are] two sides of the same tech coin." http://cloudtimes.org/2012/12/08/forrester-cloud-predictions-for-2013/ …
157. 10 Reasons To Deploy In The Cloud http://kb.esds.co.in/10-reasons-to-deploy-in-the-cloud/ …
158. Both CQL (Cassandra) and Kiji (HBase) constrain the underlying Bigtable map-of-maps schema to a map-of-predefined-structs. Pretty limiting.
159. Big Data: Is Big Data getting Bigger? | Wall Street Oasis http://dataaware.net/bigdata/20128 
160. Big data in the cloud? Be careful what you pay for - ZDNet http://tinyurl.com/d6t6kmt  #amount #Cloud
161. I just voted for 'Hadoop On-Demand on CloudStack' - what do you think? http://uservoice.com/a/Xl7cv 
162. Study shows differences between cloud users and non-users: http://ops.vu/SAi4g7 
163. Good Top 10 #Cloud Predictions for 2013. But, cloud brokers & #integration hubs will explode? been there done that. http://www.computerworlduk.com/slideshow/cloud-computing/3414733/10-cloud-predictions-for-2013/?goback=%2Egde_1855006_member_192612248 …
164. .@merv takes a good look at Amazon #Redshift on his @Gartner_inc blog #Cloud #DW - http://goo.gl/TLHCO 
165. How to: Work with @MongoDB in Datameer http://bit.ly/NAxOym  #JSON #NoSQL
166. Big Data: Getting Over Big Data Fatigue in the C Suite http://dataaware.net/bigdata/20104 
167. How Obama’s data scientists built a volunteer army on Facebook http://dlvr.it/2c1mfJ 
168. Student says " I am very discouraged. What should I do?" Master says, "encourage others." Zen Proverb [via @corybooker]
169. Cloudera Raises $65M For Big Data Analytics, Plans  ed Partner Program: Cloudera is on a roll in the hot ma... http://bit.ly/U2L7rb 
170. Performance improvements in Cassandra 1.2: http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 …. There is a _lot_ to like here.
171. @ValaAfshar Why Companies Fail: 1. Culture. 1A: Bad hiring. 1B: Poor Leadership. The End.
172. Architecture, Strategy at Center of First #AWS Conference http://zite.to/SUAIPL  #cloud
173. Big Data, EHR Driving Healthcare IT Innovation http://trib.al/fHn3XSb  @CIOonline
174. Distributed Cache On Steroids: Amazon ElastiCache http://bit.ly/VlmJ3Z  #aws #elasticache #caching @cloudspring
175. Amazon Virtual Private Cloud (VPC) essentials http://bit.ly/RC0o6B  . Good article @raghuramanb #aws #vpc
176. BigData Trends: Public Cloud Cost Study http://www.slideshare.net/bigdatalandscape/big-data-trends-cloud-cost-benchmark-study … by @vcdave #cloudcomputing
177. How Enterprises are Interacting with Big Data, Social Media - http://ow.ly/fSesf  via @dhinchcliffe
178. Distributed Cache On Steroids: Amazon ElastiCache http://bit.ly/VlmJ3Z  #aws #elasticache #caching
179. Demo: Cycle Computing Spins Up Big AWS Clusters on-the-fly at SC12 http://bit.ly/TSOTDr 
180. The evolution of NoSQL http://lnkd.in/RsJaXs 
181. Distributed Cache On Steroids: Amazon ElastiCache http://bit.ly/VlmJ3Z  #aws #elasticache #caching @cloudspring
182. The future of Internet Business http://linkd.in/VlmxSq 
183. Amazon Virtual Private Cloud (VPC) essentials http://bit.ly/RC0o6B . Good article @raghuramanb #aws #vpc
184. Nice read: How Price Sensitive are Public Cloud #Consumers? via @DavidLinthicum http://bit.ly/TzZO5g  #PublicCloud #Costbenefits
185. Otka gets $25M more to take on cloud identity management http://bit.ly/R4JJbZ 
186. EMC and VMware spin out big data/cloud division http://shar.es/6loCr  via @BButlerNWW w/ thoughts from @Stu
187. Cloud Computing: 10gen & SoftLayer Tie Up on MongoDB - 10gen, the company commercializing MongoDB, and SoftLayer, th... http://ow.ly/2tyKGV 
188. RT @jamesurquhart RT @bmichelson: great pictures inside Amazon's distribution center -- http://imgur.com/a/q1WIO  via @imgur < Wow. < Wow Squared
189. Meet the faces of big data with interactive iPad app http://dlvr.it/2Zqhj1 
190. Gartner: Big Data it will drive $232 Billion In IT spending through 2016 [CMO will drive spend]. @hubspot #CIO #CMO
191. New: Slick MongoDB managed service from Softlayer and 10gen http://dlvr.it/2ZprrQ 
192. The future of big data: 4 grams of DNA can hold 1.82 trillion gigabytes i.e All the data the world produces in a year. http://onforb.es/TqOtVj 
193. Gartner: 10 critical IT trends for the next five years - http://bit.ly/S0bfWm  #CIO #CFO #GartnerDC
194. Werner Voegels on Cost-Aware Architectures — "Exploit economies of scale" and have Revenues > Costs http://ow.ly/fLj2g 
195. Team Obama Schools GOP Operatives on 'Big Data' Usage #bigdata http://bit.ly/Rw4nl7 
196. Facebook architecture http://slidesha.re/VfCM3m 
197. Is 2013, the year of Hybrid cloud ? http://bit.ly/TAXDlm  #cloud #hybrid
198. If you learn from defeat, you haven't really lost. Zig Ziglar
199. Big Data: Big data market to grow more than 2 times by 2014 http://dataaware.net/bigdata/19921 
200. Free shipping cost Amazon $2.8 billion in the past year. They'll even ship this 1,672-pound safe for free: http://on.mktw.net/TruFAV 
201. CIOs must educate the board on the increasingly important role of technology http://bit.ly/Vbg2RM  @HarvardBiz #ceo #cio
202. Social Analytics: New Uses of Social Intelligence http://bit.ly/SxdHn8 
203. A GigaOM conversation with GE CEO Jeff Immelt http://ow.ly/2tvHTa 
204. Amazon ElastiCache - Now With Auto Discovery - http://bit.ly/113AF8I  #aws #memcached
205. The fireside chat with Jeff Bezos at #aws #reinvent this afternoon is on youtube: http://wv.ly/TvNB4Z 
206. 90% of all data on the planet has been created in the past two years. http://bit.ly/VbeAyU  @cmswire #socbiz #cio #cmo
207. 11 Hot News Stories You May Have Missed http://bit.ly/TAr3fs 
208. 10 interesting tech predictions for 2013 | @Forbes http://onforb.es/VbgNu9 
209. AWS Andy Jassy announced 2013 Premier tier consulting partners at partner summit #reinvent http://twitter.com/jinman/status/273524990090018816/photo/1pic.twitter.com/ChuuPJy1  Thanks @8kmiles team
210. RT @jinman: #AWS Andy Jassy announced 2013 Premier tier consulting partners at partner summit #reinvent pic.twitter.com/08uYPwx7
211. Launching Amazon ElastiCache in 3 Easy Steps http://bit.ly/10PkNGQ  #aws #elasticache #cloud
212. Session Synchronization patterns in AWS http://bit.ly/WItXCc  #aws #session #rds #terracotta #elasticache #dynamodb
213. Designing High Availability at Load Balancing Layer in AWS cloud http://bit.ly/PXynWx  #aws #ec2 #haproxy #elb #cloud
214. Caching architectures using Amazon ElastiCache & Memcached http://bit.ly/UkpiZl  #aws #elasticache #memcached
215. Comparison Analysis : Amazon ELB vs HAProxy on EC2 http://bit.ly/T6Ik3p  #aws #elb #ec2 #haproxy #cloud
216. Caching architectures using Amazon ElastiCache & Memcached http://bit.ly/UkpiZl  #aws #elasticache #memcached
217. Eucalyptus Empowers Enterprises With Best Practices for Cloud-Based Development and Test Strategies http://yhoo.it/TMsR6i  #cloud
218. Amazon enjoys big etail lead as Cyber Monday hits http://dlvr.it/2Xgt4b 
219. The future of IT rests in an enlightened approach to tech vs. classic IT practices http://gtnr.it/RcWctW  @Gartner_inc #CIO
220. Amazon ELB vs HAProxy on EC2 - Analysis in depth http://bit.ly/T6Ik3p  #aws #elb #ec2 #haproxy #cloud
221. Learn how to deploy #public #cloud apps #AWS #ec2 with #Puppet Enterprise. Short video here: http://bit.ly/pe2_5cpa  #puppetize
222. To be clear, this is a bucket full of awesome. http://amzn.to/Tum3YB  #illumos #zfs #dtrace #aws
223. Improving the Cloud - More Efficient Queuing with #SQS - #AWS http://wv.ly/YULyst 
224. Nice: Wakari, a Python data analysis environment running on #AWS EC2: http://continuum.io/blog/introducing-wakari …
225. A 'spot' light on cost - big data, machine learning supremos TellApart reduce costs on #AWS by 75% with spot instances: http://aws.typepad.com/aws/2012/10/new-ec2-second-generation-standard-instances-and-price-reductions-1.html …
226. New S3/Glacier feature: Archiving of Amazon S3 data to Amazon #Glacier: http://bit.ly/TDOh3c  #aws #s3
227. RT @jinman New #AWS Whitepaper: Development and Test Environments on AWS #Cloud #SDLC #dev #test http://j.mp/W4k6sS  PDF Download
228. Global manufacturer ditches underperforming IBM in favour of IaaS provider http://dlvr.it/2ThrWf 
229. “@CloudAve: ClickSoftware – Great Case of an AWS Cloud Adoption: Part 1, Operations http://j.mp/TJfZOb ”
230. Partnering with @mapr and driving adoption of #Hadoop in the enterprise http://bit.ly/PTyzVF  #BigData
231. Pick the right #cloud for your app. RightScale Solutions Architect Brian Adler shares how: http://ow.ly/fhXf7  via VentureBeat
232. Our latest infographic: Top Concerns on Securing Virtual Infrastructures http://trendmic.ro/WYTwCR  #cloudsecurity
233. Backend-As-A-Service Provider Kinvey Adds Facebook Open Graph Integration For Mobile Apps http://tcrn.ch/RxuYeb  by @sarahintampa
234. Nice whitepaper from @simon_elisha et al: Migrating #AWS Resources to a New Region: http://media.amazonwebservices.com/AWS_Migrate_Resources_To_New_Region.pdf …
235. How to launch your own open source web-service with GIT and AWS http://ow.ly/fdSnu 
236. Caching architectures using Amazon ElastiCache & Memcached http://bit.ly/UkpiZl  #aws #elasticache #memcached cc @jeffbarr @awscloud
237. Smart managers are always looking for teachable moments - first for themselves, then for those they serve.
238. Smart managers know that every person that learns from them can potentially widen their circle of influence.
239. "Happiness does not depend on what you have or who you are, it solely relies on what you think." - Buddha #quote
240. Caching architectures using Amazon ElastiCache & Memcached http://bit.ly/UkpiZl  #aws #elasticache #memcached cc @KnowYourCloud @IAmOnDemand
241. Caching architectures using Memcached & Amazon Elasticache http://bit.ly/UkpiZl  #aws #memcached #elasticache
242. Newvem Introduces Its “Cloud Smart Meter” iOS App for iPhones and iPads http://bit.ly/RLrNNT  cc @James_Staten @natishalom @harish11g
243. Designing Highly available Load Balancing Layer in AWS cloud http://bit.ly/PXynWx  #aws #elb #haproxy CC @jeffbarr @awscloud
244. ketama consistent hashing http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consistent_hashing_algo_for_memcache_clients …
245. Designing High Availability at Load Balancing Layer in AWS cloud http://bit.ly/PXynWx  #aws #ec2 #haproxy #elb #cloud
246. Two-Thirds of the World's Data Centers Going to Cloud: Cisco http://lnkd.in/7q3sb9 
247. Architecting High Availability at HAProxy Load Balancing Layer in AWS cloud http://bit.ly/PXynWx  #aws #ec2 #haproxy #elb #cloud
248. High Availability at Load Balancing Layer in AWS cloud http://bit.ly/PXynWx  #aws #ec2 #haproxy #elb #cloud
249. Using DynamoDB with Amazon Elastic MapReduce : Articles & Tutorials : Amazon Web Services http://amzn.to/SA1MCE  #AWS #cloud
250. Building a real time, solr-powered recommendation engine http://www.slideshare.net/treygrainger/building-a-real-time-solrpowered-recommendation-engine …
251. Gartner sees cloud computing, mobile development putting IT on edge http://sns.mx/RrlOy3 
252. Host your blog on #S3 + #CloudFront using Jekyll - http://bit.ly/RC82fn  - "The initial HTML page takes an insane 9ms to load!" #aws
253. Data Mining 3.4 billion Web pages for $100 of EC2 http://blog.luckyoyster.com/post/33592990831/data-mining-the-web-100-worth-of-priceless … #AWS
254. An interesting & potentially useful piece of analysis on EC2 I/O http://blog.scalyr.com/2012/10/16/a-systematic-look-at-ec2-io/ … #aws #ec2 #ebs
255. The AWS Report - Balan Subramanian, Amazon Simple Workflow Service: For today's episode of The AWS R... http://bit.ly/VWn5Sc  #AWS #Cloud
256. Good read: Caching in theory and practice http://sos.me/mKvAQ/0  #caching #algorithms
257. ‘Big data’ startup Splice Machine breathes new life into SQL, pulls in $4M | VentureBeat http://ow.ly/eyr8O 
258. See the new innovations, products & partnerships unveiled by @Citrix to help companies realize the promise of the cloud http://bit.ly/V6zJL4 
259. Google tries wowing the world with a look at its data centers http://dlvr.it/2L057v 
260. An investment in employees, is an investment in your company's brand. #CSO2012
261. SAP HANA in-memory database placed on Amazon Web Services http://dlvr.it/2L07g7 
262. News From the Front (LinkedIn TalentConnect) By the time you read this, the embargo will have passed and Linkedin wi... http://bit.ly/RcXUWV 
263. Twitter and Airbnb Founders Want to Build a Smarter City #bigdata http://bit.ly/WM94pK 
264. Cost of Latency and Amazon Route53 LBR http://bit.ly/Sh2TET  #aws #route53 #LBR #latency
265. Session Synchronization patterns in AWS http://bit.ly/WItXCc  #aws #session #rds #terracotta #elasticache #dynamodb
266. Sharding Apache Solr on AWS http://bit.ly/wJzFns  #ec2 @aws #solr
267. Cost of Latency and Amazon Route53 LBR http://bit.ly/Sh2TET  #aws #route53 #LBR #latency #cloud
268. Web Session Synchronization patterns in AWS cloud http://bit.ly/WItXCc  #aws #session #rds #terracotta #elasticache #dynamodb
269. RT @equinix: State of the Cloud. Guest blog by @rightscale CEO @michaelcrandell. http://ow.ly/e0yOV  #cloud #cloudcomputing #IaaS
270. Diff Tools for MySQL Configurations http://j.mp/V6ZOy1  #MySQL