tag:blogger.com,1999:blog-48365106652355662342024-03-15T18:09:19.351-07:00Cloud, Big Data and MobileHarish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.comBlogger199125tag:blogger.com,1999:blog-4836510665235566234.post-14455727509492635072016-01-31T20:50:00.000-08:002016-02-01T06:55:03.726-08:00CloudWatch + Lambda Case 4: Control launch of Specific “C” type EC2 instances post office hours to save costsWe have a customer who has predictable load volatility between 9 am to 6 pm and uses specific large EC2 instances during office hours for analysis, they use “c4.8xlarge” for that purpose. Their IT wanted to control launch of such large instance class post office hours and during nights to control costs, currently there is no way to restrict or control this action using Amazon IAM. In short we cannot create complex IAM policy with conditions that user A belonging to group A cannot launch instance type C every day between X and Y.<br />
Some stop gap followed is to have a job running which removes the policy from an IAM user when certain time conditions are met. So basically what we would do is, to have a job that calls an API that removes the policy which restricts an IAM user or group from launching instances. This will make the IAM policy management complex and tough to assess/govern drifts between versions.<br />
After the introduction of the CloudWatch events our Cloud operations started controlling it with lambda functions. Whenever a Instance type is launched it will trigger a lambda function, the function will filter whether it is a specific “C” type and check for the current time, if the time falls after office hours, it will terminate the EC2 instance launched immediately.<br />
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDyi2PXpPJqRpveM22q0pM33f5a6HLwAbv3x72C1kYptbM5J8puKRo62kgnjL22KYq9af9paZdaGzZ3TM5RLdH7pi4K3oDiplqv44d6Xq8C-Rj1ckrdadb934FPfzhGhovTRrJo1bL-B3R/s1600/CloudWatch+Events+Lambda+EC2+Stop+1.png" imageanchor="1"><img border="0" height="326" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDyi2PXpPJqRpveM22q0pM33f5a6HLwAbv3x72C1kYptbM5J8puKRo62kgnjL22KYq9af9paZdaGzZ3TM5RLdH7pi4K3oDiplqv44d6Xq8C-Rj1ckrdadb934FPfzhGhovTRrJo1bL-B3R/s640/CloudWatch+Events+Lambda+EC2+Stop+1.png" width="640" /></a><br />
<br />
The next step would be configuring rule details with Rule definition<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcpLC0X8ZioEyfsLyZ7zYVxkQjUHn4q9E4UWT2PGzPSvIl623ok8IjmKnzWm4_6VrWzKYqyBwXT02S3HW_Fa7HkDjA1rd98gO22Y8AMOV3UtKprQZmWDPcVZb9rF4rZFtn0lf3gYRxCd4O/s1600/CloudWatch+Events+Lambda+EC2+Stop+2.png" imageanchor="1"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcpLC0X8ZioEyfsLyZ7zYVxkQjUHn4q9E4UWT2PGzPSvIl623ok8IjmKnzWm4_6VrWzKYqyBwXT02S3HW_Fa7HkDjA1rd98gO22Y8AMOV3UtKprQZmWDPcVZb9rF4rZFtn0lf3gYRxCd4O/s640/CloudWatch+Events+Lambda+EC2+Stop+2.png" width="640" /></a><br />
<br />
<br />
Finally we will review the Rules Summary<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHDHCAouJG0LTf1qiZZelMbr-Rp6R9nTOljmFBRFoH_bbjrvivuaBM1n5nJdtaDXZcSb6acsE7Jez7ipZWxTeEpEwpTgZLr6tL672csqp4Cqcp-HXgbYCUrciq2aA6m1SAySaE0SSVhm8I/s1600/CloudWatch+Events+Lambda+EC2+Stop+3.png" imageanchor="1"><img border="0" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHDHCAouJG0LTf1qiZZelMbr-Rp6R9nTOljmFBRFoH_bbjrvivuaBM1n5nJdtaDXZcSb6acsE7Jez7ipZWxTeEpEwpTgZLr6tL672csqp4Cqcp-HXgbYCUrciq2aA6m1SAySaE0SSVhm8I/s640/CloudWatch+Events+Lambda+EC2+Stop+3.png" width="640" /></a><br />
<b><br /></b>
<b>Amazon Lambda Function Code Snippet (Python)</b><br />
import boto3<br />
<br />
def lambda_handler(event, context):<br />
#print ("Received event: " + json.dumps(event, indent=2))<br />
#print ("************************************************")<br />
<br />
ec2_client = boto3.client("ec2")<br />
<br />
<br />
print "Event Region :", event['region']<br />
<br />
event_time = event['detail']['eventTime']<br />
print "Event Time :", event_time<br />
<br />
time = event_time.split('T')<br />
t = time[1]<br />
t = t.split(':')<br />
hour = t[0]<br />
<br />
instance_type = event['detail']['requestParameters']['instanceType']<br />
print "Instance Type:", instance_type<br />
<br />
instance_id = event['detail']['responseElements']['instancesSet']['items'][0]['instanceId']<br />
print "Instance Id:",instance_id<br />
<br />
if( instance_type.startswith( 't' ) and hour > 18 or hour < 8 ):<br />
print ec2_client.terminate_instances( InstanceIds = [ instance_id ] )<br />
<br />
GitHub Gist URL: <a href="https://github.com/cloud-automaton/automaton/blob/master/aws/events/TerminateAWSEC2.py">https://github.com/cloud-automaton/automaton/blob/master/aws/events/TerminateAWSEC2.py</a><br />
<br />
This post was co authored with <a href="https://in.linkedin.com/in/priyadarshiniarasu">Priya</a> and <a href="https://in.linkedin.com/in/ramprasadguru">Ramprasad</a> of 8KMilesHarish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com1tag:blogger.com,1999:blog-4836510665235566234.post-48159177144373982016-01-31T20:40:00.000-08:002016-02-01T06:56:05.709-08:00CloudWatch + Lambda Case 3 -Controlling cross region EBS/RDS Snapshot copies for regulated industriesIf you are part of regulated industry like Pharmaceutical/ Life sciences/BFSI running mission critical applications on AWS, at times as part of the compliance requirements you will have to restrict/control data movement to a particular geographic region in the cloud. This is becomes complex to restrict some times. Let us explore in detail:<br />
We all know there are varieties of ways to move data from one AWS region to another, but one commonly used method is Snapshot copy across AWS regions. Usually you can restrict snapshot copy permission in IAM Policy, but what if you need the permission enabled for moving data between AWS accounts inside a region, but still want to control EBS/RDS snapshot copy action across regions. It can be only mitigated by automatically deleting the snapshot on destination AWS region in case snapshot copy activity is done.<br />
Our Cloud operations team used to altogether remove this permission in IAM or monitor this activity using polling scripts for customers with multiple accounts who need this permission and still need control. Now after the introduction of CloudWatch Events we have configured a rule that points to an AWS Lambda which gets triggered in near real time when snapshot is copied to destination AWS region. The lambda function will initiate a deletion process immediately. Though it is reactive it is incomparably faster than manual intervention.<br />
In this use case, Amazon CloudWatch Event will identify the EBS Snapshot copies across the regions and delete them.<br />
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPzedVrVqRZqOCcOgqQe0tqCOhTm1OvpHOlS7-oF_rGX9CnO3xYhluoTEoIPGtgKh0dbAdg_RLu7r3W-vuAL8Q9sJ1kTHHzBhUlKr9_tsTGTDOKuiCQmCylNjSJH71GGMMKDwSlewdl2VR/s1600/CloudWatch+Events+Lambda+Snapshot+copy+1.png" imageanchor="1"><img border="0" height="296" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPzedVrVqRZqOCcOgqQe0tqCOhTm1OvpHOlS7-oF_rGX9CnO3xYhluoTEoIPGtgKh0dbAdg_RLu7r3W-vuAL8Q9sJ1kTHHzBhUlKr9_tsTGTDOKuiCQmCylNjSJH71GGMMKDwSlewdl2VR/s640/CloudWatch+Events+Lambda+Snapshot+copy+1.png" width="640" /></a><br />
<br />
The next step would be configuring rule details with Rule definition<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAYGqjNHTu1p3ciE0DRleMUevDkFxsK3c1VDoJ59TuSQ1PtE_PKs_n2Igtlg86Q1ZJJB3Sgi_PFPgPSgzYPKzpmpLoukaT6KxcUuX7Mq8dnH_c_ms9lcUtsoQDG53azZOudk1gtGCi0uJy/s1600/CloudWatch+Events+Lambda+Snapshot+copy+2.png" imageanchor="1"><img border="0" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAYGqjNHTu1p3ciE0DRleMUevDkFxsK3c1VDoJ59TuSQ1PtE_PKs_n2Igtlg86Q1ZJJB3Sgi_PFPgPSgzYPKzpmpLoukaT6KxcUuX7Mq8dnH_c_ms9lcUtsoQDG53azZOudk1gtGCi0uJy/s640/CloudWatch+Events+Lambda+Snapshot+copy+2.png" width="640" /></a><br />
Finally we will review the Rules Summary<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIFO4EhhH4CfnW4nTi8glgJSiqbu3BEdf1W0ZcV7_uKvr7QCvTbLaQbObXYhKeRUSKBFCXlDLe7IvdzQzIRG3vITHtf1tqWdVwQK7-z6yIhtCXk8qb56L57VWCisz65v6xD0qOK5DbSsCH/s1600/CloudWatch+Events+Lambda+Snapshot+copy+3.png" imageanchor="1"><img border="0" height="326" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIFO4EhhH4CfnW4nTi8glgJSiqbu3BEdf1W0ZcV7_uKvr7QCvTbLaQbObXYhKeRUSKBFCXlDLe7IvdzQzIRG3vITHtf1tqWdVwQK7-z6yIhtCXk8qb56L57VWCisz65v6xD0qOK5DbSsCH/s640/CloudWatch+Events+Lambda+Snapshot+copy+3.png" width="640" /></a><br />
<br />
<br />
<b>Amazon Lambda Function Code Snippet (Python)</b><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCIbuD1BdQrstPZ2qc1RoAz3iq4PGnKuFxp7uXLhyvf8HxBGV6vhexpMpfUoc_0ADPJxN4ajX_UNY33t8ZWmzbTu_moH4-b_L-apycROujGl_DXtEiTevdcwkjpGD7_3gaWqCxFGBiqLaZ/s1600/CloudWatch+Events+Lambda+Snapshot+copy+4.png" imageanchor="1"><img border="0" height="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCIbuD1BdQrstPZ2qc1RoAz3iq4PGnKuFxp7uXLhyvf8HxBGV6vhexpMpfUoc_0ADPJxN4ajX_UNY33t8ZWmzbTu_moH4-b_L-apycROujGl_DXtEiTevdcwkjpGD7_3gaWqCxFGBiqLaZ/s640/CloudWatch+Events+Lambda+Snapshot+copy+4.png" width="640" /></a><br />
<br />
GitHub Gist URL:<a href="https://github.com/cloud-automaton/automaton/blob/master/aws/events/AWSSnapShotCopy.py"> https://github.com/cloud-automaton/automaton/blob/master/aws/events/AWSSnapShotCopy.py</a><br />
<div>
<br />
This post was co authored with <a href="https://in.linkedin.com/in/muthukumar-oman-b06b7718">Muthukumar</a> and <a href="https://in.linkedin.com/in/ramprasadguru">Ramprasad</a> of 8KMiles</div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-52911300652364722482016-01-31T20:17:00.000-08:002016-02-01T06:56:50.415-08:00CloudWatch + Lambda Case 2- Keeping watch on AWS ROOT user activity is normal or anomaly ?As a Best Practice you should never use your AWS root account credentials to access AWS. Instead, create individual (IAM) users for anyone who needs access to your AWS account. This allows you to give each IAM user a unique set of security credentials and grant different permissions to each user. Example: Create an IAM user for yourself as well, give that user administrative privilege, and use that IAM user for all your work and never share your credentials to anyone else.<br />
Usually Root has full access and it is not ideal to restrict the same in AWS IAM. Imagine you suddenly doubt some anomaly/suspicious activities done as Root user (using EC2 API’s etc) in your logs other than normal IAM user provisioning; this could be because Root user is compromised or forced, but ultimately it is a deviation from the best practice.<br />
In the past we used to poll the CloudTrail logs using programs and differentiate between “root” and “Root”, and our cloud operations used to react to these anomaly behaviors. Now we can inform the cloud operations and customer stake holders near real time using CloudWatch events. <br />
In this use case, Amazon CloudWatch Event will identify activities if any performed by an AWS ROOT user and notifications will be sent to SNS thru AWS Lambda.<br />
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target. The lambda function will detect if the event is triggered by root user and notifies through SNS.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_VqW3F2BW6ewSUxEgLqOrrC5Iddkbp7JUERBiOPzO96Km9ZgnG3TNHd5kCCC3pbxSsLL8uLcEAd_2ejwHm60ggPBELfADVpEtSaFbVr1Fo-AcaZdHY-CvoFG_BBcyXoa4N58AUZ-7KfNA/s1600/CloudWatch+Events+Lambda+Root+Activity+Tracking+1.png" imageanchor="1"><img border="0" height="296" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_VqW3F2BW6ewSUxEgLqOrrC5Iddkbp7JUERBiOPzO96Km9ZgnG3TNHd5kCCC3pbxSsLL8uLcEAd_2ejwHm60ggPBELfADVpEtSaFbVr1Fo-AcaZdHY-CvoFG_BBcyXoa4N58AUZ-7KfNA/s640/CloudWatch+Events+Lambda+Root+Activity+Tracking+1.png" width="640" /></a><br />
The next step would be configuring rule details with Rule definition<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5e2lvlyHuM9fBt63OFa9YQWURNFiQhlJ9dEKVdoX2WCFTk-0ZwKdvSk1D6NwzONf4CAHRd3qu6BP2cym7jy3XHTSp-c0ogBduJd-yD4p2zH1fBxFlle4uH8e0Ci6TSZiVFCYPiaxrNDpV/s1600/CloudWatch+Events+Lambda+Root+Activity+Tracking+2.png" imageanchor="1"><img border="0" height="324" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5e2lvlyHuM9fBt63OFa9YQWURNFiQhlJ9dEKVdoX2WCFTk-0ZwKdvSk1D6NwzONf4CAHRd3qu6BP2cym7jy3XHTSp-c0ogBduJd-yD4p2zH1fBxFlle4uH8e0Ci6TSZiVFCYPiaxrNDpV/s640/CloudWatch+Events+Lambda+Root+Activity+Tracking+2.png" width="640" /></a><br />
<br />
Finally we will review the Rules Summary<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcDWu6alsfIB8WBIfRHXwCDy6PB7OcsbSRFHbyirl36pONn8PCdQmSSGB7WBZcRmBj9mbTqvFSlpjLfD1CGWQ4qHB5kCXPw1if_SiztLZk6zmhCy47b8CpY7ISGbFkgYIVwOnZSHND47Np/s1600/CloudWatch+Events+Lambda+Root+Activity+Tracking+3.png" imageanchor="1"><img border="0" height="308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcDWu6alsfIB8WBIfRHXwCDy6PB7OcsbSRFHbyirl36pONn8PCdQmSSGB7WBZcRmBj9mbTqvFSlpjLfD1CGWQ4qHB5kCXPw1if_SiztLZk6zmhCy47b8CpY7ISGbFkgYIVwOnZSHND47Np/s640/CloudWatch+Events+Lambda+Root+Activity+Tracking+3.png" width="640" /></a><br />
<br />
<b>Amazon Lambda Function Code Snippet (Python)</b><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhViHwevIkrHOiXnlMnvuih3KXL_mKMs9bIFHzOnoMmQTOD-anam14ILjZFI32SHGi_rgU6aPqs92_Km7YfVTOXYXYGAgO_MR7DQ7fOYSKBJRewZJJxUeqNFbSCmdEFYGDVVwxsxatzzYg1/s1600/CloudWatch+Events+Lambda+Root+Activity+Tracking+4.png" imageanchor="1"><img border="0" height="216" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhViHwevIkrHOiXnlMnvuih3KXL_mKMs9bIFHzOnoMmQTOD-anam14ILjZFI32SHGi_rgU6aPqs92_Km7YfVTOXYXYGAgO_MR7DQ7fOYSKBJRewZJJxUeqNFbSCmdEFYGDVVwxsxatzzYg1/s640/CloudWatch+Events+Lambda+Root+Activity+Tracking+4.png" width="640" /></a><br />
<br />
GitHub Gist URL: <a href="https://github.com/cloud-automaton/automaton/blob/master/aws/events/TrackAWSRootActivity.py"> https://github.com/cloud-automaton/automaton/blob/master/aws/events/TrackAWSRootActivity.py</a><br />
<br />
This post was co authored with <a href="https://www.linkedin.com/in/saravanan444">Saravanan</a> and <a href="https://in.linkedin.com/in/ramprasadguru">Ramprasad</a> of 8KMilesHarish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-23909985531822952882016-01-31T19:41:00.004-08:002016-02-01T06:57:38.966-08:00 CloudWatch + Lambda Case 1- Avoid malicious CloudTrail action in your AWS Account<br />
As many of you know AWS CloudTrail provides visibility into API activity in your AWS account, Cloud Trail Logging lets you see which actions users have taken and which resources have been used, along with details such as the time and date of actions and the actions that have failed because of inadequate permissions. It enables you to answer important questions such as which user made an API call or which resources were acted upon in an API call. If a user disables CloudTrail logs accidentally or with malicious intent then audit logging events will not captured and hence you fail to have proper governance in place. The situation will get complex, If the user disables- enables back CloudTrail for a brief period of time where some important activities can go unlogged and unaudited. In short once CloudTrail logging is enabled it should not be disabled and this action needs to be defended in depth. <br />
Our Cloud operations team had earlier written a program that periodically scans the Cloud Trail logs entries, if any log activity was missing after an X period of time it alerts the operations. Overall reaction time on our cloud operations was >15-20 mins to mitigate this CloudTrail disable action.<br />
Now after the introduction of CloudWatch Events we have configured a rule that points to an AWS Lambda function as target. This function gets triggered in near real time when CloudWatch is disabled and automatically enables it back without any manual interaction from Cloud operations. The advanced version of the program triggers workflow which logs entries into ticket system as well. This event model has helped us reduce the mitigation to less than a minute.<br />
We have illustrated below the detailed steps on how to configure this event. Also we given the link for GIT with basic AWS Lambda Python code that can be used by your cloud operations.<br />
In this use case, Amazon CloudWatch Event will identify whether an AWS account has got CloudTrail enabled or not, if not enabled, Amazon CloudWatch Events will take corrective actions by enabling the same.<br />
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.<br />
<div>
<br /></div>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJv0_xSBqRJq94AmI-gkSkYbDe5Sdou3WeSDj68QofTwC_KmhSO1IkSAMqk04ob4TP-r_ygT7HTqC0dpK978J5bzwb0iFJUWh-CJQyRKZGe9jGnCR4i-o6tFn6hN9U8CzbXm-f7EEGTJ3i/s1600/CloudWatch+Events+Lambda+CloudTrail+1.png" imageanchor="1"><img border="0" height="309" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJv0_xSBqRJq94AmI-gkSkYbDe5Sdou3WeSDj68QofTwC_KmhSO1IkSAMqk04ob4TP-r_ygT7HTqC0dpK978J5bzwb0iFJUWh-CJQyRKZGe9jGnCR4i-o6tFn6hN9U8CzbXm-f7EEGTJ3i/s640/CloudWatch+Events+Lambda+CloudTrail+1.png" width="640" /></a></div>
<div>
<br /></div>
<div>
The next step would be configuring rule details with Rule definition</div>
<div>
<br /></div>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU5x8CcBwLtFYQtM9TLBHa30tAlwlC_kIYblqOk-ybzDHhNuNOyoUzmBZaDAQnHIkrNjvlJWeBuKdQgv9hut4xkDNe32VOaXGqBYXXq7hWdI2nIMrqXt6-43tQld_cOVpc35QxjTZOC4ri/s1600/CloudWatch+Events+Lambda+CloudTrail+2.png" imageanchor="1"><img border="0" height="322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU5x8CcBwLtFYQtM9TLBHa30tAlwlC_kIYblqOk-ybzDHhNuNOyoUzmBZaDAQnHIkrNjvlJWeBuKdQgv9hut4xkDNe32VOaXGqBYXXq7hWdI2nIMrqXt6-43tQld_cOVpc35QxjTZOC4ri/s640/CloudWatch+Events+Lambda+CloudTrail+2.png" width="640" /></a></div>
<div>
<br /></div>
<div>
Finally we will review the Rules Summary</div>
<div>
<br /></div>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVc2fqWk7lo5qioly85OnzrTphzJCP80T1gbMw0Udb1uC8uudLqDdzbpcDmc4LAiyhMJwSMaY8QO_7wX2Mj8MHf4Ho7gHmeUv9L1pvtb0hSzgo8rOjmgWlTpHwjARhqDVh9cRKC2BFr_pp/s1600/CloudWatch+Events+Lambda+CloudTrail+3.png" imageanchor="1"><img border="0" height="308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVc2fqWk7lo5qioly85OnzrTphzJCP80T1gbMw0Udb1uC8uudLqDdzbpcDmc4LAiyhMJwSMaY8QO_7wX2Mj8MHf4Ho7gHmeUv9L1pvtb0hSzgo8rOjmgWlTpHwjARhqDVh9cRKC2BFr_pp/s640/CloudWatch+Events+Lambda+CloudTrail+3.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<div>
<b>Amazon Lambda Function Code Snippet (Python)</b></div>
<div>
<br /></div>
<div>
import json</div>
<div>
import boto3</div>
<div>
<br /></div>
<div>
print('Loading function')</div>
<div>
""" Function to define Lambda Handler """</div>
<div>
def lambda_handler(event, context):</div>
<div>
try:</div>
<div>
</div>
<div>
client = boto3.client('cloudtrail')</div>
<div>
if event['detail']['eventName'] == 'StopLogging':</div>
<div>
response = client.start_logging(Name=event['detail']['requestParameters']['name'])</div>
<div>
</div>
<div>
except Exception, e:</div>
<div>
sys.exit();</div>
<div>
<br />
GitHub Gist URL: <a href="https://github.com/cloud-automaton/automaton/blob/master/aws/events/EnableAWSCloudTrail.py"> https://github.com/cloud-automaton/automaton/blob/master/aws/events/EnableAWSCloudTrail.py</a><br />
<br />
This post was co authored with <a href="https://in.linkedin.com/in/iammohan83">Mohan</a> and <a href="https://in.linkedin.com/in/ramprasadguru">Ramprasad</a> of 8KMiles</div>
</div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-65994822861706406862015-07-05T23:24:00.002-07:002015-07-05T23:43:19.894-07:00Amazon CloudSearch vs ElasticSearch vs Apache Solr Comparison in detail<br />
<div class="MsoNormal" style="margin-bottom: 10.0pt;">
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>This detailed article is co authored with <a href="https://www.linkedin.com/in/rdwarak">Dwarak</a>.</b></span><br />
<span style="font-family: Verdana, sans-serif;"><b>Slide Share link : </b><u><a href="http://www.slideshare.net/harishganesan/amazon-cloud-searchvsapachesolrvselasticsearchcomparisonreportv11">http://www.slideshare.net/harishganesan/amazon-cloud-searchvsapachesolrvselasticsearchcomparisonreportv11</a></u></span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">It’s the information age, and getting the
information to the people who want it, when they want it, and in the most
efficient way is one of the foremost challenges. Therefore, effective search
tools are a primary concern. There are many options available to developers,
including open source, independent vendors, and cloud-based search engines, but
given the collaborative strengths of open source products and the rapid growth
of cloud-based, the Amazon CloudSearch Comparative Report is focused on a
comparison of three major contenders: Apache Solr, Elasticsearch, and Amazon
CloudSearch. This report aims to provide a detailed and objective assessment of
the strengths of each.</span></div>
<span style="font-family: Verdana, sans-serif;">There are many features and variables to take
into account when deciding which product to choose. This report provides a
straight comparison of all the aspects a developer will need to consider,
including such features as Getting Started, Operations, Indexing, Search and
Query, Scaling, Protocols, API support, Customization, Cost, and more.</span><br />
<div class="MsoNormal" style="margin-bottom: 10.0pt;">
<a href="https://www.blogger.com/null" name="_Toc422613327"><span style="font-family: Verdana, sans-serif; font-size: large;">Smack down</span></a></div>
<div class="Title0">
<o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 10.0pt;">
<span lang="EN-IN">
</span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" valign="top" width="140"><div align="left" class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Feature<o:p></o:p></span></b></div>
</td>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" valign="top" width="154"><div class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Apache Solr<o:p></o:p></span></b></div>
</td>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" valign="top" width="154"><div class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch<o:p></o:p></span></b></div>
</td>
<td style="background: #6DC037; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" valign="top" width="154"><div class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch<o:p></o:p></span></b></div>
</td>
</tr>
<tr>
<td colspan="4" style="background: white; mso-background-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div align="left" class="MsoNormal">
Admin Operations<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Backup<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Replication/Custom handler/Custom
scripts<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Snapshot API/Custom scripts<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Fully-managed<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Patch Management<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Manual/Automated via custom scripts<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Manual/Automated via custom scripts<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Fully-managed<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Re-indexing<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Manual<o:p></o:p></div>
<div align="left" class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Manual<o:p></o:p></div>
<div align="left" class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Fully-managed<o:p></o:p></div>
<div align="left" class="MsoNormal">
Manual option available from
management console<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Monitoring<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
If hosted on EC2, Amazon CloudWatch<o:p></o:p></div>
<div align="left" class="MsoNormal">
SaaS Monitoring tools like NewRelic, Stackdriver,
Datadog<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
If hosted on EC2, Amazon CloudWatch<o:p></o:p></div>
<div align="left" class="MsoNormal">
SaaS Monitoring tools like NewRelic, Stackdriver,
Datadog<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
CloudSearch default metrics <o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Maintenance <b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
External managed service<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
External managed service<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Fully-managed<o:p></o:p></div>
</td>
</tr>
<tr>
<td colspan="4" style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div align="left" class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">A<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial;">PI</span></span></b><o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Client Library<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Java, PHP, Ruby, Rails, AJAX,
Perl, Scala, Python, .NET, JavaScript<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Java, Groovy, JavaScript, .NET,
PHP, Perl, Python, and Ruby<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Amazon SDK<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
HTTP RESTful API<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
YES<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
YES<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
YES<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Request Format<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
XML, JSON, CSV<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
XML, JSON<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
XML, JSON<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Response Format<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
XML, JSON, CSV<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
XML, JSON<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
XML, JSON<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Third party Integrations<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Available for Commercial and Open
source <o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Available for Commercial and Open
source<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Amazon Web Services Integrations
available<o:p></o:p></div>
</td>
</tr>
<tr>
<td colspan="4" style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div align="left" class="MsoNormal">
<b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: white; font-size: 14pt;">Search</span></b> <b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: white; font-size: 14pt;">Functions</span></b><o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Schema <b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Schema and Schema-less<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Schema and Schema-less<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Schema<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Dynamic fields support<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Synonyms <b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Multiple indexes<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
No<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Faceting <b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Rich documents support<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
No<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Auto Suggest<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Highlighting<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Query parser<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Standard, DisMax, Extended DisMax,
Other parsers<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Standard, query_string, DisMax,
match, multi_match<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Simple, structured, Lucene, or
DisMax<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Geosearch<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Analyzers, Tokenizers and Token
filters<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Default/Custom<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Default/Custom<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Default<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Fuzzy Logic<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Did you mean <b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Default/Custom<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Default/Custom<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
No<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Stopwords<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Customization<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
No<o:p></o:p></div>
</td>
</tr>
<tr>
<td colspan="4" style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div align="left" class="MsoNormal">
<b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: white; font-size: 14pt;">Advanced</span></b><o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Cluster management<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
ZooKeeper<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
in-built<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Fully-managed<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Scaling<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Vertical scaling/ Horizontal scaling<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Vertical scaling/ Horizontal scaling<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Fully-managed horizontal scaling<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Replication<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Sharding<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Failover<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes, if set up in Cluster Replica
mode<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes, if set up in Cluster Replica
mode<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Fully-managed<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Fault tolerant<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Yes, if set up in Cluster mode<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Yes, if set up in Cluster mode<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Fully-managed<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td colspan="4" style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div align="left" class="MsoNormal">
<b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: white; font-size: 14pt;">Import and Export</span></b><o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Data import<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Default import handlers, custom
import handlers<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Rivers modules, Logstash input
plugins, custom programs<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Batch upload<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Data export<b><o:p></o:p></b></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Default export handlers, custom export handlers<o:p></o:p></div>
</td>
<td style="background: white; border-right: solid white 1.0pt; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Snapshot API<o:p></o:p></div>
</td>
<td style="background: white; border: none; mso-background-themecolor: background1; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Custom program<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td colspan="4" style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div align="left" class="MsoNormal">
<b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: white; font-size: 14pt;">Others</span></b><o:p></o:p></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 104.7pt;" width="140"><div align="left" class="MsoNormal">
Web Interface<b><o:p></o:p></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.5pt;" width="154"><div align="left" class="MsoNormal">
Solr Admin<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
Sense<o:p></o:p></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div align="left" class="MsoNormal">
AWS Management Console<o:p></o:p></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: 10.0pt;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div>
<div class="WordSection1">
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In today's world of vast and available information, a good
search experience is central to a good user experience. Hence, delivering
effective search tools has become the key goal of all software products, market
places, e-commerce websites, and content management systems. Developers looking
to deliver a premium search experience to their users should be aware of some broad
trends:<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">1) Open source and platform-based search engines are
replacing proprietary search engines because of better licensing models and
community support.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">2) The cloud delivery model is succeeding over the
on-premise delivery model because of scalability, high availability and
operating expense.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In light of the above trends, the choice of leading
candidates for search technology boils down to three: Apache Solr,
Elasticsearch, and Amazon CloudSearch. Our clients often ask us how
these three choices compare relative to each other. This report aims to make it
easy for developers to pick the right technology for their application by
presenting a comprehensive framework for evaluation of the three options. We
have also applied our framework to top feature sets that are critical to any
search workload. We then broke them down further into granular features and
compared each of the three search engines.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In this report, we summarize our conclusions and present
them in a smack down style summary card. We encourage our readers to run a more
in-depth evaluation for their specific use cases.<o:p></o:p></span></div>
</div>
<div class="MsoNormal" style="margin-bottom: 10.0pt;">
<span lang="EN-IN">
<span style="font-family: "Calibri","sans-serif"; font-size: 11.0pt; line-height: 115%; mso-ansi-language: EN-US; mso-ascii-theme-font: minor-latin; mso-bidi-font-family: "Times New Roman"; mso-bidi-language: AR-SA; mso-bidi-theme-font: minor-bidi; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><br clear="all" style="mso-break-type: section-break; page-break-before: always;" /></span></span></div>
<div class="WordSection1">
<div class="Title0">
<a href="https://www.blogger.com/null" name="_Toc422613330"><span style="font-family: Verdana, sans-serif; font-size: large;"><b>Feature 1: Getting Started</b></span></a></div>
</div>
<h1>
<o:p></o:p></h1>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">‘Getting Started’ is
the first step by an engineer to understand the basics of the major features of
a product. In this section, we will see how the search engines discussed above
facilitate ‘Getting Started’.<o:p></o:p></span></i></div>
<div class="MsoNormal" style="margin-bottom: 10.0pt;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">
</span></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="height: 16.85pt; mso-yfti-firstrow: yes; mso-yfti-irow: -1;">
<td style="background: #6DC037; height: 16.85pt; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div align="left" class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background: white; mso-background-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch require
the end users to spend quality time in understanding and setting up the
respective search engines. The “Getting Started” manuals of Apache Solr and
Elasticsearch assume the end user to have minimal knowledge of search
engines, their related functions and architecture. <b><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The installation processes for
Apache Solr and Elasticsearch include tasks such as:<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Server setup<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Search
engine download<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Dependent
software installations<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Setup
of environmental requirements<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Understanding
of basic server commands<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 2pt 0in 12pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Administrative
access<b><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch are
shipped with test examples which allow users to do “warm up” search and
indexing operations. While the default test schema in Apache Solr is
sufficient for the user to get started, the Elasticsearch’s schema-less
design allows the user to send document request without any schema.<b><o:p></o:p></b></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: white; mso-background-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">If you already have an Amazon Web
Services (AWS) account set up, you can create a CloudSearch domain in a few
clicks using the AWS Management console. The AWS CloudSearch Management
console guides administrators through a step-by-step process, requesting the
user input:<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Instance type<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->High
availability options<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Replication
options<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Schema
definitions<b><o:p></o:p></b></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 2pt 0in 12pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Access
policies<b><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Among these options, it is important
to note that Amazon CloudSearch does not mandatorily prompt for all
information. The CloudSearch domain name and engine type are adequate to
create a CloudSearch instance.<b><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The other configurations such as
schema, instance type, access policies, and high availability options can be
modified at a later time based on the application requirements.<b><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The users are abstracted from hardware
provisioning, software installation, configuration, cluster setup and other
administration activities. <b><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Users receive two administrative regional
endpoints: a search endpoint and document endpoint. Both endpoints can be
accessed using RESTful API or AWS Software Development kit (SDK) with
Identity and Access Management (IAM) credentials.<b><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Another important note is that
CloudSearch default access policies for the document service and search
service endpoints are configured to block all IP addresses. Developers should
configure the authorized IP addresses to access the CloudSearch’s endpoints. <b><o:p></o:p></b></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;">CloudSearch
also provides a sample dataset, the IMDB movies, which can be used to test
drive the CloudSearch service. The CloudSearch developer documentation walks
through the steps to launch a test domain using the sample IMDB dataset.<b><o:p></o:p></b></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: white; mso-background-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch expect
users to have basic practical knowledge of the search engine and also
complete a few significant tasks to accomplish the first step ‘Getting
Started’. <b><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In Amazon CloudSearch, the ‘Getting
Started’ activities are easier and end users can have the CloudSearch
instance up and running with few clicks in a few minutes.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<h1>
<a href="https://www.blogger.com/null" name="_Toc422613331"><span style="font-family: Verdana, sans-serif;">Feature 2: Operations and Management</span></a><o:p></o:p></h1>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In this section, we’ll discuss some important administrative
operations such as <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin-left: .75in; mso-add-space: auto; mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt; line-height: 115%;">•<span style="font-size: 7pt; font-stretch: normal; line-height: normal;">
</span></span><!--[endif]-->Index backup<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .75in; mso-add-space: auto; mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt; line-height: 115%;">•<span style="font-size: 7pt; font-stretch: normal; line-height: normal;">
</span></span><!--[endif]-->Patch management<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-left: .75in; mso-add-space: auto; mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt; line-height: 115%;">•<span style="font-size: 7pt; font-stretch: normal; line-height: normal;">
</span></span><!--[endif]-->Re-indexing and recovery<o:p></o:p></span></div>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613332"></a><a href="https://www.blogger.com/null" name="_Toc421213423">2.1 Backup</a><b><i><o:p></o:p></i></b></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Data backup is a
routine operation, carried out within a defined period of time. Data backup is
an essential task for recovering data responsively from failures such as
hardware crash, data corruption or related events.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; line-height: 115%; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Apache Solr</span></b><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Apache Solr provides a feature called
‘ReplicationHandler’. The main objective of ReplicationHandler is to
replicate index data to slave servers, but it can also be used as a backup
copy server. A replication slave node can be configured with Solr Master, which
can be solely identified as a backup server, with no other operations taking
place on the slave node. <o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Solr‘s implicit support for replication allows
ReplicationHandler to be used as an API. The API has optional parameters like
location, name of snapshot, and number of backups. The backup API is a
bound-to-store snapshot on a local disk, but for any other storage options
the backup API requires customization.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">If you are required to store the backups in a
different store location like Amazon’s Simple Storage Service (S3), a local
storage server, or in a remote data center, ReplicationHandler has to be
customized. Solr core libraries are available in open source that allows for
any customization.<o:p></o:p></span></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Elasticsearch provides an advanced option called
‘Snapshot API’ for backing up the entire cluster. The API will back up the current cluster
state and related data and save it to a shared repository. <o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">The first or initial backup process will be a
complete copy of the data. The subsequent backup processes will snapshot the
delta between the backup of fresh data with previous snapshots. Elasticsearch
prompts end users to create a repository type, which can be chosen from a
shared file system:<o:p></o:p></span></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin-left: 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span lang="EN-IN" style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Amazon S3<o:p></o:p></span></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span lang="EN-IN" style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Hadoop Distributed File System (HDFS)<o:p></o:p></span></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-left: 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span lang="EN-IN" style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Azure Cloud<o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">This integration gives a greater flexibility for
developers to manage their backups.<o:p></o:p></span></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Backup Process</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">The backup options present in Apache Solr and
Elasticsearch can be executed manually or can be automated. To automate the
entire backup process, one has to write custom scripts that calls the
relevant API or handler. Most engineering companies follow this model of
writing custom scripts for backup automation.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">The backup also involves maintenance of the latest
snapshots and archives. The management tasks involve key tasks like snapshot
retrieval, archival, and expiration.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">In an alternate approach, if the Solr or
Elasticsearch cluster is set up in a cluster replication mode, any one of the
slave nodes is identified as backup server. The automation of the slave node
backup server needs a script written by the developer.<o:p></o:p></span></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch inherently takes cares of the
data that is stored and indexed, leaving a lighter load for engineering and
operations teams. Amazon CloudSearch self-manages all the data backup and its
management. The backups are internally maintained behind the scenes. In the
event of any hardware failure or other problem, Amazon CloudSearch restores
the backup automatically, and this process is not revealed to end users.<o:p></o:p></span></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">The default option in Apache Solr is only to back
up to a ‘local disk’ store; it does not offer any other storage options as
Elasticsearch does. However, the engineers can write their own handlers to
manage the backup process.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Elasticsearch is packaged with multiple storage
options plugins which gives added advantage for engineers.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch relieves the users of the
intricacies of the backup and its management process. The IT operations or
managed service team have a lesser role in the backup process as the entire
operations are managed behind the scenes by CloudSearch.<o:p></o:p></span></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<h2>
<span style="font-family: Verdana, sans-serif; font-size: small;">2.2 System upgrades and patch management</span><o:p></o:p></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Patch management and
system upgrades like OS patches and fixes are inevitable in operations and
administration. For any system, there is always a version upgrade, or
maintenance on the OS, and hardware or software changes.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1;">
<b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Rolling Restarts<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch both recommend using
‘Rolling Restarts’ for patch management, operating system upgrades and other
fixes. Rolling Restarts involve stopping and starting each cluster node in
the cluster sequentially. This allows the cluster to continue its operations
while each node is updated with the latest code, fixes, or patches while continuing
to serve search requests. Rolling Restarts is adopted when high availability
is mandatory and downtime is not allowable.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Sometimes, the Rolling Restarts require some
intelligent decision making based on cluster topology. If a cluster consists
of shards and replicas, the order of restarting each node has to be done
decisively. <o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Apache Solr</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Apache’s <b>ZooKeeper </b>service acts as a
stand-alone application and does not get upgraded automatically when Apache
Solr is upgraded, but it should be done manually at the same time.<o:p></o:p></span></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Elasticsearch recommends disabling the ‘shard
allocation’ configuration during node restart. This informs Elasticsearch to
stop re-balancing missing shards because the cluster will immediately start
working on node loss. <o:p></o:p></span></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch internally manages all patches
and upgrades related to its operating system. The managed search service
offering from Amazon CloudSearch monitors for when new features are rolled
out; upgrades are self-managed and immediately available to all customers
without any action on their part. <o:p></o:p></span></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">The patch management in Apache Solr and
Elasticsearch has to be carried out manually using the Rolling Restarts
feature. Customers automate this process by developing custom scripts to do
system upgrades and patch management.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Patch management in Amazon CloudSearch is
transparent to the customers. The upgrades and patches done on Amazon
CloudSearch are regularly updated in the ‘What’s New’ section of the
CloudSearch documentation.<o:p></o:p></span></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<a href="https://www.blogger.com/null" name="_Toc422613334"><span lang="EN-IN"><span style="font-family: Verdana, sans-serif; font-size: small;"><br /></span></span></a></div>
<div class="MsoNormal">
<a href="https://www.blogger.com/null" name="_Toc422613334"><span lang="EN-IN"><span style="font-family: Verdana, sans-serif; font-size: small;"><b>2.3
Re-indexing</b></span></span></a></div>
<div class="MsoNormal">
<i><span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Any business application changes over its lifetime,
as the business running it changes. The business change has a direct effect on
the data structure of the system’s persistent information store. The search
engine, which is seen as a secondary or alternate store, will eventually have
to change its data structure when required.
Any changes to the search engine data structure will require a
re-indexing of the data. <o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Example: A product company started collecting
‘feedback’ from their customer for a given product. The text string from the
new field ‘feedback’ needs to be added into the search schema, and may require
re-indexing.<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">If the search data is not re-indexed after a
structural change, the data that has already been indexed could become inaccurate
and the search results may behave differently than expected.<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Re-indexing becomes a necessary process over a
period of time as the application grows. It is also identified as a common and
mandatory admin operation executed periodically based on application
requirements.<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
</div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 1;">
<b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: 2.0pt; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Apache Solr recommends re-indexing if there
is a change in your schema definitions. The options below are widely used by
the Apache Solr user community.<o:p></o:p></span></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin-bottom: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Create a fresh index with new settings. Copy all
of the documents from the old index to the new one. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-bottom: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Configure Data import handler with
‘SolrEntityProcessor’. The SolrEntityProcessor imports data from Solr
instances or cores for a given search query.
The SolrEntityProcessor has a limitation where it can only copy fields
that are stored in the source index. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-bottom: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Configure Data import handler with the source or
origination data source. Push the data freshly to the new index<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: 2.0pt; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Elasticsearch proposes several approaches
for data re-indexing. The following approaches are usually combined:<o:p></o:p></span></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin-bottom: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt; font-stretch: normal;">
</span><!--[endif]-->Use Elasticsearch’s Scan and Scroll and Bulk APIs
to fetch and push data into the new index. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-bottom: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt; font-stretch: normal;">
</span><!--[endif]-->Update or create an index alias with the old index
name and delete the old index.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-bottom: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt; font-stretch: normal;">
</span><!--[endif]-->Use open source Elasticsearch plugins that can
extract all data from the cluster and re-index the data. Most of these
plugins internally use the Scan and Scroll and Bulk API (as mentioned above)
which reduces development time.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 6pt;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch recommends data
rebuilding when index fields are added or modified. Amazon CloudSearch
expects to issue an indexing request after a configuration change. Whenever
there is a configuration change, the CloudSearch domain status changes to
‘NEEDS INDEXING’. During the index rebuilding, the domain's status changes to
‘PROCESSING’, and upon completion the status is changed to ‘ACTIVE’. <o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch can continue to serve
search requests during the indexing process, but the configuration changes
are not reflected in the search results. The re-indexing process can take
some time for the changes to take effect. It is directly proportional to the
amount of data volume in your index.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch also allows document
uploads while indexing is in progress, but the updates can become slower, if
there are is large volume of document updates. During such a scenario, the uploads
or updates can be throttled or paused until the Amazon CloudSearch domain
returns to an ‘ACTIVE’ state.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Customers can initiate re-indexing by
issuing the index-documents command using RESTful API, AWS command line
interface (CLI), or AWS SDK. They can also initiate re-indexing from the
CloudSearch management console. <o:p></o:p></span></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; margin-bottom: .0001pt; margin-bottom: 0in; mso-yfti-cnfc: 128;">
<span style="font-family: Verdana, sans-serif;"><b><span lang="EN-IN" style="color: white; font-size: 14.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><b><span lang="EN-IN"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Re-indexing in Apache Solr and Elasticsearch is
mostly a manual process because it requires a decision that factors data
size, current request size, and offline hours. <o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch manages the re-indexing process
inherently and leaves much less to administrators. The re-indexing time
period is abstracted and not disclosed to administrators but Amazon
CloudSearch runs the re-indexing process based on the best practices
mentioned above.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><br /></span></div>
<h1>
<a href="https://www.blogger.com/null" name="_Toc422613335"><span lang="EN-IN"><span style="font-family: Verdana, sans-serif; font-size: large;">Feature
3: Monitoring</span></span></a><span lang="EN-IN"><o:p></o:p></span></h1>
<div class="MsoNormal">
<i><span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">Monitoring server health is an essential daily
task for operations and administration. In this section, we will describe the
built-in monitoring capabilities for all three search engines.<o:p></o:p></span></span></i></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">
</span></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Apache Solr has a built-in
web console for monitoring indexes, performance metrics, information about
index distribution and replication, and information on all threads running in
the Java Virtual Machine (JVM) at the time.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">For more detailed monitoring, Java
Management Extensions (JMX) can be configured with Solr that share runtime
statistics as MBeans. The Apache Solr JVM container has built-in
instrumentation that enables monitoring using JMX.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch has a management and
monitoring plugin called ‘Marvel’. Marvel has an interactive console called
‘Sense’ that helps users to interact easily with Elasticsearch nodes.
Elasticsearch has in-built diversified APIs that emit heap usage, garbage
collection stats, file descriptions, and more. Marvel is strongly integrated
with these APIs, and it periodically executes polling, collects statistics
and stores the data back in Elasticsearch. Marvel’s interactive graph report
dashboard allows administrators to query and aggregate historical stats data.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch recently
introduced Amazon CloudWatch integration. The Amazon CloudSearch metrics can
be used to make scaling decisions, troubleshoot issues, and manage clusters. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon
CloudSearch publishes four metrics into Amazon CloudWatch:
SuccessfulRequests, Searchable Documents, Index Utilization, and Partition Count.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">The CloudWatch metrics can be
configured to set alarms, which can notify administrators through Amazon
Simple Notification Service.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch have
integrations with in-built and external plugins. They can also support SaaS based
monitoring plugins or custom plugins developed by the customers.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">CloudSearch’s integration with
CloudWatch shares some good metrics and it is expected to offer newer ones in
the future.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<h1 style="margin-bottom: 12.0pt;">
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613336">Feature 4: Schema,
Data types, Dynamic Fields and Data Import/Export</a><o:p></o:p></span></h1>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613337">4.1 Schema management</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Schema: A schema is a
definition of fields and field types used by the search system to organize data
within the document files it indexes.<o:p></o:p></span></i></div>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Schema definition is
the foremost task in the search data structure design. It is important that the
schema definition caters to all business requirements and is designed to suit
the application.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Both Elasticsearch and Apache Solr
can run the search application in ‘Schema-less’ and ‘Schema’ mode. Schema
mode is suitable for application development or any production environments.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Schema-less is a very good option
for entrants to get started. After server setup, users can start the
application without a schema structure and create the field definitions on
the search indexing. However, to have a production-grade application running,
a proper schema structure becomes mandatory and the schema definition is a
necessity.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch also allows users
to set up search domains without any index fields. The index fields can be
added anytime, but before any valid document indexing or any search request.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In addition, the CloudSearch
management console has integration with Amazon Web Services like S3,
DynamoDB, or can access a local machine from where the schema can be imported
directly to CloudSearch domain. After the schema import, CloudSearch allows
the user to edit the fields or add new fields. This is a convenient feature
for a pre-built schema that is to be migrated to a CloudSearch domain.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr
and Elasticsearch can be started without any schema but they cannot be put
into production use. Amazon CloudSearch allows creating domains without any
index fields, but to have any index and search requests served the schema
should be created.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The general
best practice in schema management is to rehearse and design the schema
suiting application requirements before finalizing the search structure. The
underlying schema concept of all three search engines is consistent with this
practice.<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613338">4.2 Dynamic fields</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Dynamic fields are
like regular field definitions which support wildcard matching. They allow the
indexing of documents without knowing the type of fields they contain. A
dynamic field is defined using a wildcard pattern (*) for first, last, or only
character. All undefined fields go through dynamic field rules which validate
the pattern match configured with the dynamic field's indexing options.<o:p></o:p></span></i></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch allow end
users to set up dynamic fields and rules using RESTful API and schema
configuration.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In Amazon CloudSearch, dynamic
fields can be configured using indexing options in the CloudSearch management
console or using CloudSearch, RESTful API, or AWS SDK API.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">If you are unsure about the schema structure or exact field names,
dynamic fields come in handy. Amazon CloudSearch, Apache Solr, and
Elasticsearch all allow the flexibility to configure dynamic fields. This
helps the application development
team to describe any omitted field definitions in the schema document.<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613339">4.3 Data types</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">There are a variety of data types supported by these search
engines. The table below illustrates the data field types supported by each
search engine.<o:p></o:p></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" valign="top" width="191"><div class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Data type<o:p></o:p></span></span></b></div>
</td>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" valign="top" width="123"><div class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Solr<o:p></o:p></span></span></b></div>
</td>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" valign="top" width="134"><div class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Elasticsearch<o:p></o:p></span></span></b></div>
</td>
<td style="background: #6DC037; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" valign="top" width="154"><div class="MsoNormal">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">CloudSearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">String / Text<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Number types<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">integer, double, float,
long<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">byte, short, integer,
long, float, double<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">integer, double<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Date types<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Enum fields<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">No<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Currency<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">No<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">No<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Geo location / Latitude – Longitude<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Boolean<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">No<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Array types<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Yes<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 143.45pt;" width="191"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 92.15pt;" width="123"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.15pt;" width="134"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 115.55pt;" width="154"><div class="MsoNormal">
<br /></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; margin-left: 5.4pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 6.3in;" valign="top" width="605"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><b><span style="color: #365f91; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: accent1; mso-themeshade: 191;"><o:p></o:p></span></b></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 6.3in;" width="605"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The most important data types like string, date, and number types are
supported by all three search engines. Geo location data type, which is now
regularly used by modern applications, is also supported by all search
engines.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Engineers and developers may use an alternate data type if a particular
data type is not supported for their chosen search engine. Example,
‘currency’ data type supported in Solr is not available in Elasticsearch and
CloudSearch. During such cases, engineers use number type as an alternative
data type for ‘Currency’.<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613340">4.4 Data import & export</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The most important task in a search application development
is data migration from origination source to the search engine. The origination
data can be of a data source like a database, or a file system or a persistent
store. To commence a search data set, it is required to migrate or import the
full data set from its origin to the search engine. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Likewise, extracting data from a search engine and exporting
it to a different destination source is also a crucial task but executed
occasionally.<o:p></o:p></span></div>
<div class="MsoNormal">
</div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr has in-built handler
called Data import handler (DIH). The DIH provides a tool for migrating
and/or importing data from the origin store. The DIH can index data from data
sources such as <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Relational Database Management
System (RDBMS)<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Email<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->HTTP
URL end point<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Feeds
like RSS and ATOM<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 2pt 0in 12pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Structured XML files<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The DIH has more advanced features
like <a href="https://tika.apache.org/">Apache Tika integration</a>, delta
import, and transformers to quickly migrate the data. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The Apache Solr export handler can
export the query result data to a Javascript Object Notification (JSON) or
comma-separated values (CSV) format. The export query expects to sort and
filter query parameters and returns only the stored fields. Users also have
the option of developing a custom export handler and incorporate it with Solr
core libraries.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch ‘Rivers’ is an elegant
pluggable service which runs inside the Elasticsearch cluster. This service
can be configured for pulling or pushing the data that is indexed into the
cluster. Some of the popular Elasticsearch Rivers modules are CouchDB,
Dropbox, DynamoDB, FileSystem, Java Database Connectivity (JDBC), Java
Messaging Service (JMS), MongoDB, neo4j, Redis, Solr, Twitter, and Wikipedia.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">However, ‘Rivers’ will be deprecated
in the newer release of Elasticsearch, which recommends using official client
libraries built for popular programming languages. Alternatively, the
Logstash input plugin is also one of the identified tools that can be used to
ship data into Elasticsearch. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">For data export, Elasticsearch
snapshot can be used for any individual indices or an entire cluster into a
remote repository. This is discussed in detail in the section <a href="file:///C:/Users/harish11g/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.Outlook/97QUVVX2/Amazon%20CloudSearch_Comparison_Report_V11.docx#_Feature_2:_Operations">‘Operations
and Management - Backup’</a>.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon
CloudSearch recommends sending the documents in batches to upload on
CloudSearch domain. A batch is a collection of add, update, and delete
operations which should be described in JSON or XML format. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon
CloudSearch limits a single batch upload to 5 MB per batch, but allows
running parallel upload batches to reduce the time frame for full data
upload. The number of parallel batch uploads is directly proportional to the
CloudSearch instance types. Larger instance types have a higher upload
capacity, while smaller instance types have lower. During such scenarios, the
batch upload programs should intelligently threshold the uploads based on
instance capacity. <o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr has good handlers to
export and import the data. In any case, if the options present are not
viable, Apache Solr allows one to develop a new custom handler or customize
an existing handler that can be used for data import and export.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch has integration with
popular data sources in the form of ‘River’ modules or plugins. However, the
future versions of Elasticsearch strongly recommend using Logstash input
plugins or developing and contributing new Logstash input, as customization
of a plugin is allowed in Elasticsearch.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch does not have
elaborate options like other two search engines. However by combining custom
programs with bulk upload recommendations in Amazon CloudSearch, customers
can successfully migrate data into CloudSearch.</span><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="WordSection1">
<h1>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613341">Feature 5: Search and Indexing features</a><o:p></o:p></span></h1>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In this section, we will evaluate ‘Search and Indexing’
features present in the search engines we are evaluating. This is a very
important feature set as they are widely used by search application engineers.<o:p></o:p></span></div>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613342">5.1 Analyzers, Tokenizers and Token filters</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Generally speaking,
the search engine prepares text strings for indexing and searching using
analyzers, tokenizers, and filters. These tools are frequently used by
libraries configured for indexing and searching the data. Most of the time, the
libraries are composed in a sequential series.<o:p></o:p></span></i></div>
<div class="MsoListParagraphCxSpFirst" style="margin-left: .75in; mso-add-space: auto; mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt; line-height: 115%;">•<span style="font-size: 7pt; font-stretch: normal; line-height: normal;">
</span></span><!--[endif]--><i>During
indexing and querying, analyzer assesses the field text and tokenizes each
block of text into individual terms. Each token is a sub-sequence of the
characters in the text.<o:p></o:p></i></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-left: .75in; mso-add-space: auto; mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt; line-height: 115%;">•<span style="font-size: 7pt; font-stretch: normal; line-height: normal;">
</span></span><!--[endif]--><i>The token
filter filters each token in the stream sequentially and applies its filter
functionality.<o:p></o:p></i></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td colspan="3" style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Apache Solr and Elasticsearch</span></b><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 12.0pt; mso-themecolor: background1;"><o:p></o:p></span></b></span></div>
</td>
<td style="border: none; mso-cell-special: placeholder; padding: 0in 0in 0in 0in;" width="12"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td colspan="3" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch have
multifaceted in-built libraries for analyzers, tokenizers and token filters.
These libraries are packaged with search engine installable that can be
configured during indexing and searching. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Although the analyzers can be
configured for indexing and querying, the same series of libraries doesn’t
need to be used for both operations. The indexing and searching operations
can be configured to have different tokenizers and filters, as their goals
can be different.<o:p></o:p></span></div>
</td>
<td style="border: none; mso-cell-special: placeholder; padding: 0in 0in 0in 0in;" width="12"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 85.65pt;" width="114"><div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;">Search Engine<o:p></o:p></span></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 187.45pt;" width="250"><div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;">Tokenizers<o:p></o:p></span></b></div>
</td>
<td colspan="2" style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 187.5pt;" width="250"><div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;">Filters<o:p></o:p></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 85.65pt;" width="114"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 187.45pt;" width="250"><div class="MsoNormal">
<br /></div>
</td>
<td colspan="2" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 187.5pt;" width="250"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 85.65pt;" width="114"><div class="MsoNormal">
<b><span style="font-size: 14.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 187.45pt;" valign="top" width="250"><div align="left" class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;"><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;">Standard,
Classic, Keyword, Letter, Lower Case, N-Gram, Edge N-Gram, ICU, Path
Hierarchy, Regular Expression Pattern, UAX29 URL Email, White Space</span><o:p></o:p></span></i></div>
</td>
<td colspan="2" style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 187.5pt;" valign="top" width="250"><div align="left" class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;"><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;">ASCII
Folding, Beider-Morse, Classic, Common Grams, Collation Key,
Daitch-MokotoffSoundex, Double Metaphone, Edge N-Gram, English Minimal Stem,
Hunspell Stem, Hyphenated Words, ICU Folding, ICU Normalizer 2, ICU
Transform, Keep Words, KStem, Length, Lower Case, Managed Stop, Managed
Synonym, N-Gram, Numeric Payload Token, Pattern Replace, Phonetic, Porter
Stem, Remove Duplicates Token, Reversed Wildcard, Shingle, Snowball Porter,
Stemmer, Standard, Stop, Suggest Stop, Synonym, Token Offset Payload, Trim,
Type As Payload, Type Token, Word Delimiter</span><o:p></o:p></span></i></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 85.65pt;" width="114"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 187.45pt;" valign="top" width="250"><div align="left" class="MsoNormal">
<br /></div>
</td>
<td colspan="2" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 187.5pt;" valign="top" width="250"><div align="left" class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 85.65pt;" width="114"><div class="MsoNormal">
<b><span style="font-size: 14.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Elasticsearch<o:p></o:p></span></span></b></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 187.45pt;" valign="top" width="250"><div align="left" class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;"><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;">Standard,
Edge NGram, Keyword, Letter, Lowercase, NGram, Whitespace, Pattern, UAX Email
URL, Path Hierarchy, Classic, Thai</span><o:p></o:p></span></i></div>
</td>
<td colspan="2" style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 187.5pt;" valign="top" width="250"><div align="left" class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;"><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;">Standard
Token, ASCII Folding Token, Length Token, Lowercase Token, Uppercase Token,
NGram Token, Edge NGram Token, Porter Stem Token, Shingle Token, Stop Token,
Word Delimiter Token, Stemmer Token, Stemmer Override Token, Keyword Marker
Token, Keyword Repeat Token, KStem Token, Snowball Token, Phonetic Token,
Synonym Token, Compound Word Token, Reverse Token, Elision Token, Truncate
Token, Unique Token, Pattern Capture Token, Pattern Replace Token, Trim
Token, Limit Token Count Token, Hunspell Token, Common Grams Token,
Normalization Token, CJK Width Token, CJK Bigram Token, Delimited Payload
Token, Keep Words Token, Keep Types Token, Classic Token, Apostrophe Token</span><o:p></o:p></span></i></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 85.65pt;" width="114"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 187.45pt;" width="250"><div class="MsoNormal">
<br /></div>
</td>
<td colspan="2" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 187.5pt;" width="250"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<!--[if !supportMisalignedColumns]-->
<tr height="0">
<td style="border: none;" width="117"></td>
<td style="border: none;" width="250"></td>
<td style="border: none;" width="238"></td>
<td style="border: none;" width="12"></td>
</tr>
<!--[endif]-->
</tbody></table>
<div class="MsoNormal">
<br /></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 459.0pt;" valign="top" width="612"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 459pt;" width="612"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch analysis scheme
configuration is used for analyzing text data during indexing. The analysis
schemes basically control:<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Text field content processing<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Stemming<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Inclusion
of stopwords and synonyms<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Tokenization
(Japanese language)<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 2pt 0in 12pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Bigrams (Chinese, Japanese, Korean
languages) <o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 2pt 0in 0.0001pt;">
<span style="font-family: Verdana, sans-serif;">The
following analysis options are executed when text fields are configured with
an analysis scheme<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">1.<span style="font-size: 7pt; font-stretch: normal;"> </span><!--[endif]-->Algorithmic stemming: Level of
algorithmic stemming (minimal, light, and heavy) to perform. The stemming
levels vary depending on the analysis scheme language.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">2.<span style="font-size: 7pt; font-stretch: normal;"> </span><!--[endif]-->Stemming dictionary: A dictionary to
override the results of the algorithmic stemming. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">3.<span style="font-size: 7pt; font-stretch: normal;"> </span><!--[endif]-->Japanese Tokenization Dictionary: A
dictionary which specifies how particular characters should be grouped into
words (only for Japanese language).<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">4.<span style="font-size: 7pt; font-stretch: normal;"> </span><!--[endif]-->Stopwords: A set of terms that
should be ignored both during indexing and at search. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">5.<span style="font-size: 7pt; font-stretch: normal;"> </span><!--[endif]-->Synonyms: A dictionary of words that
have the same meaning in the text data<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Before processing the analysis
scheme, Amazon CloudSearch tokenizes and normalizes the text data. During
tokenization, the text data is split into multiple tokens; this is common
behavior in all search engine text processing. During normalization, upper
case characters are converted to lower case, and more formatting is applied. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">After the
tokenization and normalization processes are completed, stemming, stopwords,
and synonyms are applied.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 459.0pt;" width="612"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 459pt;" width="612"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch are
packaged with varied libraries with distinct functions of analyzers,
tokenizers, and filters. Also, these libraries are allowed to be customized which
gives greater flexibility for the developers. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch doesn’t carry
sophisticated tokenizers or filter libraries like Apache Solr or
Elasticsearch, but it has simplified the configuration. Amazon CloudSearch
tokenizers and filters cover most common search requirements and use
cases. This is ideal for developers
who want to quickly integrate search functionality into their application
stack.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
</td>
</tr>
</tbody></table>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613343">5.2 Faceting</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Faceting is the
composition of search results into categories or groups, based on indexed
terms. Faceting allows for categorizing search results into more sub-groups,
which can be used as the basis for filters or other searches. Faceting is also
for efficient computation of search results by facets. For example, facets for
‘Laptop’ search results can be 'Price', ‘Operating System’, 'RAM' or 'Shipping
Method’.<o:p></o:p></span></i></div>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Faceting is a popular
function that helps consumers filter through search results easily and
effectively.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr has far advanced options
in faceting ranging from simple to very advanced faceting behavior.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The below table details the
parameters used during faceting. They can be grouped by field value, date,
range, pivot, multi-select, and interval.<o:p></o:p></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1152;">
<tbody>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.95pt;" width="135"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="font-family: Verdana, sans-serif;">Facet grouping<o:p></o:p></span></b></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 350.35pt;" width="467"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="font-family: Verdana, sans-serif;">Parameters <o:p></o:p></span></b></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.95pt;" width="135"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Field value parameters<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 350.35pt;" width="467"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">facet.field, facet.prefix,
facet.sort, facet.limit, facet.offset, facet.mincount, facet.missing,
facet.method, facet.enum.cache.minDffacet.threads<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.95pt;" width="135"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Date faceting parameters<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 350.35pt;" width="467"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">facet.date, facet.date.start, facet.date.end, facet.date.gap,
facet.date.hardend, facet.date.other, facet.date.include<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.95pt;" width="135"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Range faceting parameters<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 350.35pt;" width="467"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">facet.range, facet.range.start,
facet.range.end, facet.range.gap, facet.range.hardend, facet.range.other,
facet.range.include<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 100.95pt;" width="135"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Pivot<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 350.35pt;" width="467"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">facet.pivot, facet.pivot.mincount<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 100.95pt;" width="135"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Interval<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 350.35pt;" width="467"><div align="left" class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">facet.interval, facet.interval.set<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Elasticsearch has
deprecated facets and announced that they will be removed in a future
release. The Elasticsearch team felt that their facet implementation was not
designed from the ground up to support complex aggregations. Elasticsearch
will be replacing facets with aggregations in their next release. <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Elasticsearch says “<i>An aggregation can be seen as a
unit-of-work that builds analytic information over a set of documents. The
context of the execution defines what this document set is (for example, a
top-level aggregation executes within the context of the executed
query/filters of the search request).</i>”<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch strongly recommends
migrating from facets to aggregations. The aggregations are classified into
two main families, Bucketing and Metric. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The following table lists the
aggregations available in Elasticsearch.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1152;">
<tbody>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 64.45pt;" width="86"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 397.65pt;" width="530"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 64.45pt;" width="86"><div class="MsoNormal">
<b><i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Elasticsearch
Aggregators<o:p></o:p></span></span></i></b></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 397.65pt;" width="530"><div class="MsoNormal">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Min Aggregation, Max Aggregation, Sum
Aggregation, Avg Aggregation, Stats Aggregation, Extended Stats
Aggregation, Value Count Aggregation, Percentiles Aggregation, Percentile
Ranks Aggregation, Cardinality Aggregation, Geo Bounds Aggregation, Top
hits Aggregation, Scripted Metric Aggregation, Global Aggregation, Filter
Aggregation, Filters Aggregation, Missing Aggregation, Nested Aggregation,
Reverse nested Aggregation, Children Aggregation, Terms Aggregation,
Significant Terms Aggregation, Range Aggregation, Date Range Aggregation,
IPv4 Range Aggregation, Histogram Aggregation, Date Histogram Aggregation,
Geo Distance Aggregation, GeoHash grid Aggregation<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 64.45pt;" width="86"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 397.65pt;" width="530"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon
CloudSearch simplifies facet configuration when defining indexing options.
These facets are targeted at common use cases like e-commerce, online travel,
classifieds, etc. The facet can be of any field having data type as date,
literal, or numeric field. This is done during CloudSearch domain
configuration. Amazon CloudSearch also allows the buckets definition to
calculate facet counts for particular subsets of the facet values. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The facet
information can be retrieved in two ways:<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Sort:
returns facet information sorted either by facet counts or facet values.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Buckets:
returns facet information for particular facet values or ranges<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">During
searching, facet information can be fetched for any facet-enabled field by
specifying the “facet.FIELD” parameter in the search request (‘FIELD’ is the
name of a facet-enabled field). <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon
CloudSearch does allow multiple facets which help to refine search results
further. See the below example. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Example:
"q=poet&facet.genres={}&facet.rating={}&facet.year={}&return=_no_fields"<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">All three
search engines allow users to perform faceting with minimal effort. However,
in terms of an advanced complex implementation, the approaches are different
for each search engine.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
</td>
</tr>
</tbody></table>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613344">5.3 Auto Suggestion</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">When a user types a search
query, suggestions relevant to the query input are presented and as more
characters are typed by the user, refined suggestions are presented. This
feature is called auto-suggest. Auto-suggest is an appealing and useful
requirement and employed in many search user interfaces.<o:p></o:p></span></i></div>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">This feature can be
implemented at the Search Engine level or at the Search Application level.
Below are some options available in these three search engines.<o:p></o:p></span></i></div>
<div class="MsoNormal">
<br /></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr has native support for
the auto-suggest feature. It can be facilitated by using NGramFilterFactory,
EdgeNGramFilterFactory, or TermsComponent. Usually, this Apache Solr feature
is used in conjunction with jQuery or asynchronous client libraries for
creating powerful auto-suggestion and user experience in the front-end applications.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch also has many edge
n-grams, which are easy to set up, flexible, and fast. Elasticsearch introduced
a new data structure, Finite State Transducer (FST), which resembles big
graph data structure. This data structure is managed in memory, which makes
it much faster than a term-based query could be. Elasticsearch also
recommends using edge n-grams when query input and its word ordering are less
predictable.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch offers
‘Suggesters’ to achieve auto-suggest. CloudSearch Suggesters are configured
based on a particular text field. When Suggesters are used for querying with
a search string, CloudSearch lists all documents where the search string in the
Suggester field begins with that search string. Suggesters can be configured
to find matches for the exact query, or to perform a fuzzy matching process
to correct the query string. The ‘Fuzzy Matching’ can be defined with
fuzziness level Low, High or Default. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Suggesters also can be configured
with SortExpression, which computes a score for each one. It’s important to
do domain indexing when a new Suggester is configured. Suggestions will not be reflected until all
of the documents are indexed.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch provides simple
yet powerful ‘Suggest’ implementation, which is sufficient for most of the
applications. If you are looking for advanced options or any further
customizations on ‘Suggestions’, Apache Solr and Elasticsearch offer some
good options. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
</td>
</tr>
</tbody></table>
</div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><span style="color: #f79646; font-size: 12pt; letter-spacing: 0.75pt; line-height: 115%;"><br clear="all" style="mso-break-type: section-break; page-break-before: always;" />
</span>
</span></div>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613345">5.4 Highlighting</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Highlighting is a way
of giving formatting clues to end users in the search results. It is a valuable
feature, where the front-end search applications highlight search snippets of
text from each search result. This function conveys to end users why the result
document matched their query. In this section, we will describe the options
present in all three search engines.<o:p></o:p></span></i></div>
<div class="MsoNormal">
</div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr includes document text
fragments, which are matched in the query response. These text fragments are
included in the response as a highlighted section that is used as a cue by
search clients for representation. Apache Solr is packaged with good
highlighting collections which give control over the text fragments, fragment
size, fragment formatting, and so on. These highlighting collections can be
incorporated with Solr Query parsers and Request Handlers. <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Apache Solr comes with
three highlighting utilities <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin-bottom: 0.0001pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Standard
Highlighter<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.5in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->FastVector
Highlighter <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 2pt 0in 6pt 35.7pt; text-indent: -17.85pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Postings Highlighter <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Standard Highlighter is most
commonly used by search engineers because it is a good choice for a wide
variety of search use-cases. The FastVector Highlighter is ideal for large
documents and highlighting text in a variety of languages. The Postings
Highlighter works well for full-text keyword search.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Elasticsearch also allows
for highlighting search results on one or more fields. The implementation
uses a Lucene based highlighter, fast-vector-highlighter and
postings-highlighter. In Elasticsearch, the highlighter can be configured in
the query to force a specific highlighter type. This is a very flexible
option for developers to choose a specific highlighter to suit their
requirements. <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Like Apache Solr, the
three highlighters present in Elasticsearch emulate the same behavior which
is seen in Solr because these highlighters are inherited from the Lucene
family.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch
simplifies the highlighting by specifying the highlight.FIELD parameter in
the search request. Amazon CloudSearch returns excerpts with the search
results to show where the search terms occur within a particular field of a
matching document.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">For example: Search terms
‘Smart Phone’ is highlighted for the description field:<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Highlights":
{"description": "A *smartphone* is a mobile phone with an
advanced mobile operating system. They typically combine the features of a
cell phone with those of other popular mobile devices, such as personal
digital assistant (PDA), media player and GPS navigation unit. A *smartphone*
has a touchscreen user interface and can run third-party apps, and are camera
phones."}<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch also
provides controls like number of search term occurrences within an excerpt,
how they should be highlighted, plain text or HTML and so on.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">From a development perspective, all
three search engines provide easy and simple highlighting implementations. If
you are looking for different and more advanced highlighting options, Apache
Solr and Elasticsearch have some good features.<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<o:p><br /></o:p></div>
</td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>
<h1>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613346">Feature 6: Multilingual support</a><o:p></o:p></span></h1>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Multilingualism is a very
important feature for global applications which cater to non-English speaking
geographies. A leading information measurement company’s survey reveals that
search engines built with multilingual features are emerging and successful
because of native language support, and focus on the cultural background of the
users.<o:p></o:p></span></i></div>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;"><u>Business Impact:</u>
A multilingual search is an effective marketing strategy to get the attention
of consumers. In e-commerce, a platform to do more business is created when the
language is in the native tongue of the customer.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr is packaged with
multilingual support for most common languages. Apache Solr carries many
language-specific tokenizers, and filters libraries which can be configured
during indexing and querying. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr engineering forums
recommend using multi-core architecture where each core manages one language.
Solr also supports language detection using Tika and LangDetect detection
features. This helps to map the text data to language-specific fields during
indexing.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch has incorporated a
vast collection of language analyzers for most commonly spoken languages. The
primary role of the language analyzer is to split, stem, filter, and apply
required transformations specific to the language. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch also allows a user to
define a custom analyzer that can be a base extension of another analyzer.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch has strong
support for language-specific text processing. Amazon CloudSearch has
pre-defined default analysis schemes support to 34 languages. Amazon
CloudSearch processes the text and text-array fields based on the configured
language-specific analysis scheme. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch also allows a
user to define a new analysis scheme that can be an extension of the default
language analysis scheme.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">All three search engines have ample
and effective support features for widely spoken international languages. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Languages Support</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The table below lists the languages
supported by each search engine<o:p></o:p></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1152;">
<tbody>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;">Search engine<o:p></o:p></span></b></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;">Languages supported<o:p></o:p></span></b></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></b></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;"><span style="font-size: 10pt;">Arabic,
Brazilian, Portuguese, Bulgarian, Catalan, Chinese, Simplified Chinese, CJK,
Czech, Danish, Dutch, Finnish, French, Galician, German, Greek, Hebrew,
Lao, Myanmar Khmer, Hindi, Indonesian, Italian, Irish, Japanese, Latvian,
Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Scandinavian,
Serbian, Spanish, Swedish, Thai and Turkish</span><o:p></o:p></span></i></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;">Elasticsearch<o:p></o:p></span></b></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;"><span style="font-size: 10pt;">Arabic,
Armenian, Basque, Brazilian, Bulgarian, Catalan, Chinese, Czech, Danish,
Dutch, English, Finnish, French, Galician, German, Greek, Hindi, Hungarian,
Indonesian, Irish, Italian, Japanese, Korean, Kurdish, Norwegian, Persian,
Portuguese, Romanian, Russian, Spanish, Swedish, Thai and Turkish</span><o:p></o:p></span></i></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal">
<br /></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal">
<br /></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch<o:p></o:p></span></b></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;"><span style="font-size: 10pt;">Arabic,
Armenian, Basque, Bulgarian, Catalan, Chinese - Simplified, Chinese -
Traditional, Czech, Danish, Dutch, English, Finnish, French, Galician,
German, Greek, Hindi, Hebrew, Hungarian, Indonesian, Irish, Italian,
Japanese, Korean, Latvian, Norwegian, Persian, Portuguese, Romanian,
Russian, Spanish, Swedish, Turkish and Thai</span><o:p></o:p></span></i></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
</td>
</tr>
</tbody></table>
<h1 style="margin-bottom: 12.0pt;">
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613347">Feature 7: Protocol
& API Support</a><o:p></o:p></span></h1>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613348">7.1 Request and Response formats</a><o:p></o:p></span></h2>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" valign="top" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Search engine<o:p></o:p></span></span></b></div>
</td>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 134.65pt;" valign="top" width="180"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Request
formats<o:p></o:p></span></span></b></div>
</td>
<td style="background: #6DC037; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 198.45pt;" valign="top" width="265"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Response
formats<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 134.65pt;" width="180"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 198.45pt;" width="265"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 134.65pt;" width="180"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">XML, JSON, CSV<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 198.45pt;" width="265"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">JSON, XML, CSV<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 134.65pt;" width="180"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 198.45pt;" width="265"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Elasticsearch<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 134.65pt;" width="180"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">JSON <o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 198.45pt;" width="265"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">XML, JSON <o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 134.65pt;" width="180"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 198.45pt;" width="265"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 134.65pt;" width="180"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">XML, JSON<o:p></o:p></span></span></i></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 198.45pt;" width="265"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">XML, JSON<o:p></o:p></span></span></i></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613349">7.2 External Integrations</a><o:p></o:p></span></h2>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-table-layout-alt: fixed; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" valign="top" width="118"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Search
engine<o:p></o:p></span></span></b></div>
</td>
<td style="background: #6DC037; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 357.45pt;" valign="top" width="477"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Integrations
available<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Drupal, Magento, Django, ColdFusion, Wordpress,
OpenCMS, Plone, Typo3, ez Publish, Symfony2, Riak, DataStax Enterprise
Search, Cloudera Search, Hortonworks Data Platform, MapR<o:p></o:p></span></span></i></div>
<div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;"><a href="https://wiki.apache.org/solr/IntegratingSolr#Integrating_Solr_With_Other_.28Non_Search.29_Applications"><i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;">https://wiki.apache.org/solr/IntegratingSolr#Integrating_Solr_With_Other_.28Non_Search.29_Applications</span></i></a><i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><o:p></o:p></span></i></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Elasticsearch<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">Drupal, Django, Symfony2, Wordpress, CouchBase,
SearchBlox, Hortonworks Data Platform, MapR<o:p></o:p></span></span></i></div>
<div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;"><a href="http://www.elasticsearch.org/guide/en/elasticsearch/client/community/current/integrations.html"><i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;">http://www.Elasticsearch.org/guide/en/Elasticsearch/client/community/current/integrations.html</span></i></a><i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><o:p></o:p></span></i></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 88.45pt;" width="118"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 357.45pt;" width="477"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613350">7.3 Protocols Support</a><o:p></o:p></span></h2>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" valign="top" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Search
engine<o:p></o:p></span></span></b></div>
</td>
<td style="background: #6DC037; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 326.05pt;" valign="top" width="435"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Protocols
support<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 326.05pt;" width="435"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 326.05pt;" width="435"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">HTTP, HTTPS<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 326.05pt;" width="435"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Elasticsearch<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 326.05pt;" width="435"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">HTTP, HTTPS<o:p></o:p></span></span></i></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 326.05pt;" width="435"><div class="MsoNormal" style="margin-bottom: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 120.5pt;" width="161"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="color: white; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch<o:p></o:p></span></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 326.05pt;" width="435"><div class="MsoNormal" style="margin-bottom: 2pt;">
<i><span style="font-size: 10.0pt; mso-bidi-font-size: 11.0pt;"><span style="font-family: Verdana, sans-serif;">HTTP, HTTPS<o:p></o:p></span></span></i></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<h1>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613351"></a><a href="https://www.blogger.com/null" name="_Feature_8:_High"></a>Feature 8: High Availability<o:p></o:p></span></h1>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">All three search engines are architected for <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt; line-height: 115%;">•<span style="font-size: 7pt; font-stretch: normal; line-height: normal;">
</span></span><!--[endif]-->High availability (HA)<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt; line-height: 115%;">•<span style="font-size: 7pt; font-stretch: normal; line-height: normal;">
</span></span><!--[endif]-->Replication <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt; line-height: 115%;">•<span style="font-size: 7pt; font-stretch: normal; line-height: normal;">
</span></span><!--[endif]-->Scaling design principles<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In this next section, we will discuss high availability
options present in these three search engines.<o:p></o:p></span></div>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613352">8.1 Replication</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Replication is copying
or synchronizing the search index from master nodes to slave nodes for managing
the data efficiently. <o:p></o:p></span></i></div>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Replication is a key
design principle exercised in high availability searches and scaling. From a
High Availability perspective, replication can be effective for both HA and
failovers from master nodes (shards or leaders) to slave nodes (replicas).
Replication from a scaling perspective is used to scale the slave or replica
nodes when the requests traffic increases.<o:p></o:p></span></i></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">
</span></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 459.3pt;" valign="top" width="612"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 459.3pt;" width="612"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr supports two models of
replication, namely legacy mode and SolrCloud. In legacy mode, the
replication handler copies data from the master node index to slave nodes.
The master server manages all index updates and the slave nodes handle read queries.
This segmentation of master and slave allows scaling Solr clusters to deliver
heavy volume loads. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache SolrCloud is a distributed
advanced cluster setup using Solr nodes designed with high availability and
fault-tolerance. Unlike legacy mode, there is no explicit concept of
"master/slave" nodes. Instead, the search cluster is categorically
split into leaders and replicas. The leader has the responsibility to ensure
the replicas are updated with the same data stored in the leader. Apache Solr
has a configuration called ‘numShards’ which defines number of shards
(leaders). During start-up, the core index is split across the ‘numShards’
(number of shards) and the shards are represented as leaders. The nodes that
are attached in the Solr cluster after the initial ‘numShards’ will be
automatically assigned as replicas for the leaders. <o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 459.3pt;" width="612"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 459.3pt;" width="612"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch follows a similar
concept to SolrCloud. In brief, an Elasticsearch index can be split into
multiple shards and each shard can be replicated into any number of nodes (0,
1, 2 …n). When replication is completed, the index will have primary shards
and replica shards. During index creation, the number of shards and replicas
are defined. The number of replicas can be dynamically changed, but the shards
count cannot.</span><o:p></o:p></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 459.3pt;" valign="top" width="612"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 459.3pt;" width="612"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Both Apache Solr and Elasticsearch
support synchronous and asynchronous replication models. If the replication
is configured in ‘synchronous’ mode, the primary (leader) shard will wait for
successful responses from the replica shards before returning commit
transaction. If the model is ‘asynchronous’, the response is returned to the
client as soon as the request is executed on the primary or leader shard. The
request to the replicas is forwarded asynchronously.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The diagram below depicts the
replication concept which is followed in Solr and Elasticsearch.<o:p></o:p></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge4dHLObsofXfJnirLD2k1v0V6nGqjKN5p7E-T8GeU9gq_y2yDRtGChx7Oqd6NVaReE9dKLVf_SpC9G7ucjhCDv-yfu8pWRR2q003pSBRDjKErmw0Q2v4gw_ZHmWoamg1GU7WUmds9uWYa/s1600/comparison+SOlr+vs+CloudSearch+vs+elasticsearch+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge4dHLObsofXfJnirLD2k1v0V6nGqjKN5p7E-T8GeU9gq_y2yDRtGChx7Oqd6NVaReE9dKLVf_SpC9G7ucjhCDv-yfu8pWRR2q003pSBRDjKErmw0Q2v4gw_ZHmWoamg1GU7WUmds9uWYa/s640/comparison+SOlr+vs+CloudSearch+vs+elasticsearch+1.png" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f">
<v:stroke joinstyle="miter">
<v:formulas>
<v:f eqn="if lineDrawn pixelLineWidth 0">
<v:f eqn="sum @0 1 0">
<v:f eqn="sum 0 0 @1">
<v:f eqn="prod @2 1 2">
<v:f eqn="prod @3 21600 pixelWidth">
<v:f eqn="prod @3 21600 pixelHeight">
<v:f eqn="sum @0 0 1">
<v:f eqn="prod @6 1 2">
<v:f eqn="prod @7 21600 pixelWidth">
<v:f eqn="sum @8 21600 0">
<v:f eqn="prod @7 21600 pixelHeight">
<v:f eqn="sum @10 21600 0">
</v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:formulas>
<v:path gradientshapeok="t" o:connecttype="rect" o:extrusionok="f">
<o:lock aspectratio="t" v:ext="edit">
</o:lock></v:path></v:stroke></v:shapetype><v:shape id="Picture_x0020_74" o:spid="_x0000_i1026" style="height: 192.75pt; mso-wrap-style: square; visibility: visible; width: 448.5pt;" type="#_x0000_t75">
<v:imagedata o:title="" src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image001.png">
</v:imagedata></v:shape><o:p></o:p></span></div>
<div align="center" class="MsoNormal" style="text-align: center;">
<i><span style="font-family: Verdana, sans-serif;">Replication handling in Apache Solr
and Elasticsearch<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1152;">
<tbody>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 70.9pt;" width="95"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">S1<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 155.9pt;" width="208"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Node 1 <o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 229.9pt;" width="307"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Shard 1 of
the cluster<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 70.9pt;" width="95"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">S2<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 155.9pt;" width="208"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Node 2 <o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 229.9pt;" width="307"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Shard 1 of the cluster<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 70.9pt;" width="95"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">R1<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 155.9pt;" width="208"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Node 3 <o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 229.9pt;" width="307"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Replica 1
of Shard 1<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 70.9pt;" width="95"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">R2<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 155.9pt;" width="208"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Node 4<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 229.9pt;" width="307"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Replica 1 of Shard 2<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 70.9pt;" width="95"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">R3<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 155.9pt;" width="208"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Node 5<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 229.9pt;" width="307"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Replica 2
of Shard 1<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 70.9pt;" width="95"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">R4<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 155.9pt;" width="208"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Node 6 <o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 229.9pt;" width="307"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Replica 2 of Shard 2<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 459.3pt;" width="612"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 459.3pt;" width="612"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch is simple and
refined when it comes to handling replication and streamlines the job of
search engineers and administrators. During the configuration of scaling,
Amazon CloudSearch prompts for the desired replication count which should be
based on load requirements. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch will
automatically scale up and scale down the replicas for a domain based on the
requests traffic and data volume, but not below the desired replication
count. In Amazon CloudSearch, the replication scaling option can be changed
at any time. If the scale requirement is temporary, (for example, anticipated
spikes because of a seasonal sale) the desired replication count of the
domain can be pre-scaled, and then the changes reverted after the requests
volume returns to a steady state. Modifying the replication count does not
require any index rebuilding but the replica sync completion is dependent on
the size of search index.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The following describes the benefits
of Amazon CloudSearch replication model.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin-top: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->The
search instance capacity is automatically replicated and load is distributed,
the search layer is robust and highly available at all times. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-top: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Improved
fault tolerance. If any one of the replicas is down, the other replica(s)
will continue to handle requests while the failed replica is in recovery
mode.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-top: 2pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->The entire process of scaling and
distribution is automated and avoids manual intervention and support.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-top: 2pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 459.3pt;" width="612"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 459.3pt;" width="612"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">All three search engines have a good
base to support the ‘replication’ feature. Apache Solr and Elasticsearch
allow defining your own replication topology which can be configured for
synchronous and asynchronous replication. They can be manually or
automatically scaled based on application requirements and by writing custom
programs. However, substantial managed service operations are required if the
cluster replication is set up in enterprise scale.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch fully manages the
replication by managing scaling, load distribution, and fault tolerance. This
simplicity saves operations costs for the enterprises and companies.<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613353">8.2 Failover</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Failover is a back-end
operation that switches to secondary or standby nodes in the event of primary
server failure. Failover is identified as an important fault tolerance function
for systems with lower or zero downtime requirements.<o:p></o:p></span></i></div>
<div class="MsoNormal">
</div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">When an Apache Solr or Elasticsearch
cluster is built with shards and replicas, the cluster inherently becomes
fault-tolerant and mechanically supports failover. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">During any failure, a cluster is
expected to support the operations while the failed node is put into recovery
state. Both the Apache Solr and Elasticsearch documentation strongly
recommend a distributed cluster setup to protect user experience from
application or infrastructure failure.
<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In the event of all nodes storing
shards and replicas failing, then the client requests will also fail. If the
shards are set to tolerant configuration, partial results can be returned
from the available shards. This behavior is anticipated in both Apache Solr
and Elasticsearch.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The representation below depicts how
failover is handled in cluster. This flow is applicable for both Solr and
Elasticsearch.<o:p></o:p></span></div>
<div class="MsoNormal">
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEip7YhoR2gnAbixN-WUpDXc8fptizdkcL4Vdhej_NwUy3wn3uWXeSJ9JP89XbKn6a-cYgXs5uXXTjusneVKx25zPUK4cDCbawZ5MFb041Qf3fQuaqoqemVAOnOsv0HUqlI_yoKoVVSOFLEL/s1600/comparison+SOlr+vs+CloudSearch+vs+elasticsearch+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEip7YhoR2gnAbixN-WUpDXc8fptizdkcL4Vdhej_NwUy3wn3uWXeSJ9JP89XbKn6a-cYgXs5uXXTjusneVKx25zPUK4cDCbawZ5MFb041Qf3fQuaqoqemVAOnOsv0HUqlI_yoKoVVSOFLEL/s640/comparison+SOlr+vs+CloudSearch+vs+elasticsearch+2.png" width="640" /></a></div>
<br /></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><v:shape id="Picture_x0020_9" o:spid="_x0000_i1025" style="height: 181.5pt; mso-wrap-style: square; visibility: visible; width: 425.25pt;" type="#_x0000_t75">
<v:imagedata o:title="" src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image002.png">
</v:imagedata></v:shape><o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1152;">
<tbody>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 148.6pt;" width="198"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="font-family: Verdana, sans-serif;">Node <o:p></o:p></span></b></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 154.05pt;" width="205"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="font-family: Verdana, sans-serif;">Replica 1 <o:p></o:p></span></b></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 154.05pt;" width="205"><div class="MsoNormal" style="margin-bottom: 2pt;">
<b><span style="font-family: Verdana, sans-serif;">Replica 2<o:p></o:p></span></b></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 148.6pt;" width="198"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">SHARD 1 – Node Number 1<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 154.05pt;" width="205"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">SHARD 1 FIRST REPLICA – Node Number 3<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 154.05pt;" width="205"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">SHARD 1 SECOND REPLICA – Node Number 5<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 148.6pt;" width="198"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">SHARD 2 –
Node Number 2<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 154.05pt;" width="205"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">SHARD 2
FIRST REPLICA – Node Number 4<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 154.05pt;" width="205"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">SHARD 2
SECOND REPLICA – Node Number 6<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.25in;">
<span style="font-family: Verdana, sans-serif;">The below table
illustrates the failure scenarios in a Search cluster.<o:p></o:p></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1152;">
<tbody>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 69.2pt;" width="92"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Scenario A<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 392.9pt;" width="524"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">If SHARD1
fails, then one of its replica nodes, either Node number 3 or Node number 5
is chosen as leader. <o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 69.2pt;" width="92"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Scenario B<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 392.9pt;" width="524"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">If SHARD2 fails, then one of its replica nodes, either
Node number 4 or Node number 6 is chosen as leader. <o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-right-color: white; border-right-width: 1pt; border-style: none solid none none; padding: 0in 5.4pt; width: 69.2pt;" width="92"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Scenario C<o:p></o:p></span></div>
</td>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: none; padding: 0in 5.4pt; width: 392.9pt;" width="524"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">If SHARD 1
REPLICA1 fails, then Shard 1 Replica 2 continues to support replication and
as well serve the requests. <o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #F2FAEC; border-right: solid white 1.0pt; border: none; mso-border-right-alt: solid white .25pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 69.2pt;" width="92"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">Scenario D<o:p></o:p></span></div>
</td>
<td style="background: #F2FAEC; border: none; mso-border-left-alt: solid white .25pt; mso-border-left-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 392.9pt;" width="524"><div class="MsoNormal" style="margin-bottom: 2pt;">
<span style="font-family: Verdana, sans-serif;">If SHARD 2 REPLICA1 fails, then Shard 2 Replica 2
continues to support replication and as well serve the requests. <o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Elasticsearch uses
internal Zen Discovery to detect failures. If the node holding a primary
shard dies, then a replica is promoted to the role of primary. Apache Solr
uses Apache ZooKeeper for Co-ordination, failure detection, and leader
voting. ZooKeeper initiates leader election process between replicas during a
leader/primary shard failure.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch has built-in
failover support. Amazon CloudSearch recommends scaling options and
availability options to increase fault tolerance in the event of a service
disruption or node failures. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">When Multi-AZ is turned on, Amazon
CloudSearch provisions the same number of instances in your search domain in
the second availability zone within that region. The instances in the primary
and secondary zones are capable of handling a full load in the event of any
failure. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In the event of a service disruption
or failure in one availability zone, the traffic requests are automatically
redirected to the secondary availability zone. In parallel, Amazon
CloudSearch self-heals the cluster in failure, and Multi-AZ restores the
nodes without any administrative intervention. During this switch, the
inflight queries might fail, and they will need to be retried from the
front–end application side. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">By increasing the partitions and
replicas in the Amazon CloudSearch scaling options, failover support can be
improved. If there's a failure in one of the replicas or partitions, the
other nodes (replica or partition) will handle requests and support while it
is being recovered.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch is very
sophisticated in terms of handling failure, as the node health is continuously
monitored. In the event of infrastructure failures, the nodes are
automatically recovered or replaced.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Failover can be architected by
applying techniques like replication, sharding, service discovery, and
failure-detection services. Apache Solr and Elasticsearch advocate building your
search system in ‘Cluster mode’ to address failover. They undertake that
responsibility by employing service discovery which can detect unhealthy
nodes. The service discovery maintains the cluster information and balances
the search cluster when nodes are detected for failures.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch supports failover
for single node as well as for cluster mode. Behind the scenes, CloudSearch
continuously monitors the health of the search instances and they are
automatically managed during failures. <o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div>
<div class="WordSection1">
<h1>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613354">Feature 9: Scaling</a><o:p></o:p></span></h1>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">The ability to scale
in terms of computing power, memory, or data volume is essential in any data
and traffic bound applications. Scaling is a significant design principle
employed to improve performance, balancing and high availability. <o:p></o:p></span></i></div>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Over time, the search
cluster is expected to be scaled horizontally (scale out) or vertically (scale
up) depending upon the needs. <o:p></o:p></span></i></div>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Scale-up is the
process of moving from a small server to a large server. Scale-out is the
process of adding multiple servers to handle the load. The scaling strategy
should be selected based on application requirements.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Scaling an Apache Solr or
Elasticsearch application involves manual processes. These can include a
simple server addition task or advanced tasks like cluster topology changes,
storage changes, or infrastructure upgrades.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">If vertical scaling takes place, the
search cluster needs to follow processes like new setup and configuration,
downtime, node restarts, etc. If scaling is horizontal, the process may
involve re-sharding, rebalancing, or cache warming. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">While a search cluster system can
benefit from powerful hardware, vertical scaling has its own limitations.
Upgrading or increasing the infrastructure specifications on the same server
can involve tasks like:<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->New setup<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Backup
<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Down
time<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 2pt 0in 12pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Application re-testing<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The scaling out process is
identified as a relatively easier task compared to scaling up.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">An expert search administrator
(Apache Solr or Elasticsearch) is usually posted to keep a close watch on the
performance of the search servers. Infrastructure and search metrics play a
key role in administrator decision making. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">When these metrics increase beyond
the threshold of a particular server and start affecting overall performance,
the new server(s) have to be manually spawned. Also, the scale up task can
expand to index partitioning, auto-warming, caching and
re-routing/distribution of the search queries to the new instances. It
requires a Solr expert on your team to identify and execute this activity
periodically.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Sharding and Replication</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Though scaling up, scaling out, and
scaling down involve manual work, technology-driven companies automate this
process by developing custom programs. These smart programs continuously
monitor the cluster group and make decisions to do elastic scaling. This
output is quite similar to AutoScaling’s offering. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In terms of administration
functionality, both Apache Solr and Elasticsearch offer scaling techniques
called Sharding and Replication. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Sharding (which means partitioning)
is a method in which a search index is split into multiple logical units
called "shards". If the indexed documents exceed the collection’s
physical size, then sharding is recommended by administrators. When sharding
is enabled the search requests are distributed to every shard in the
collection, results are individually collected and then merged.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Another scaling technique,
replication, (<a href="file:///C:/Users/harish11g/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.Outlook/97QUVVX2/Amazon%20CloudSearch_Comparison_Report_V11.docx#_Feature_8:_High">See 8.1 Replication - discussed in
detail</a>) allows adding new servers with redundant copies of your index
data to handle higher concurrent query loads by distributing the requests
around to multiple nodes. <o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch is a fully
managed search service; it scales up and down seamlessly as the amount of
data or query volume increases. CloudSearch can be scaled based on the data
or based on the requests traffic. When the search data volume increases, CloudSearch
can be scaled from a smaller instance type to a larger search instance type.
If the capacity of largest search instance type is also exceeded then CloudSearch
partitions the search index across multiple search instances (Sharding
technique). <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">When traffic and concurrency grows, Amazon
CloudSearch deploys additional (replicas) search instances to support traffic
load. This automation eases the complexity and manual labour required in the
scaling out process. Conversely, when the traffic drops, Amazon CloudSearch
scales down your search domain by removing the additional search instances in
order to minimize costs. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The Amazon CloudSearch management
console allows users to configure the desired partition count and the desired
replication count. The AWS console also allows changing of the instance type
(scaling up) anytime. This inherent behavior of elastic scaling makes one of
the most important points in favor of Amazon CloudSearch. <o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Scaling in search is implemented in
the form Sharding and Replication. All three search engines have a strong
scaling support for setting up their search tier in ‘cluster mode’. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Scaling in Apache Solr and
Elasticsearch often requires administration as there is no direct hard and
fast rule. Techniques like elastic scaling can implemented only up to a limit
and when cluster grows further, manual intervention and thought process is
required. Vertical scaling in Apache Solr and Elasticsearch is even more delicate.
It requires individual management of the nodes in the cluster and executed by
using techniques like ‘Rolling restarts’ and custom scripts.<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 2.0pt; mso-hyphenate: none;">
<span style="font-family: Verdana, sans-serif;">Amazon Cloud Search takes
away all the operation intricacies from the administrators. The desired partition
count and desired replication count option in CloudSearch will automatically
scale up and scale down based on the volume of data and requests traffic. This
saves lot of efforts and cost on operations and management.<o:p></o:p></span></div>
</div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">
<span style="font-size: 11pt; line-height: 115%;"><br clear="all" style="mso-break-type: section-break; page-break-before: always;" />
</span>
</span></span></div>
<div class="WordSection2">
<h1>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613355">Feature 10: Customization</a><o:p></o:p></span></h1>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">At times, the search
system or its software may not have support for a specific feature or built-in
integration with other systems. In such cases, most open source software allows
developers to customize and extend their desired features as plugins,
extensions or modules. Often, the developer community shares extension
libraries which are helpful for a practical cause. These libraries can be
customized and integrated with the system.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;">Apache
Solr and Elasticsearch both belong to the same source breed, allowing
customizations on: <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Analyzers<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Filters
<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Tokenizers<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Language
analysis <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Field
types<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Validators<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 2pt 0in 0.0001pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->Fall
back query analysis<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 2pt 0in 12pt 0.75in; text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->Alternate query custom handlers <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Since both products are
open source, the developers can customize or extend the libraries to fit the
required feature modifications through plugins and libraries. The build and
deployment becomes a developer’s responsibility after the extending the code
base.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<span style="font-family: Verdana, sans-serif;">Apache Solr and
Elasticsearch have many plugin extensions that will allow developers to add
custom functionality for a variety of purposes. These plugins are configured
as special libraries and refer to the application using configuration
mapping.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch does not allow
for any customizations. The search features in Amazon CloudSearch are offered
by AWS after much careful thought and collective feedback from the customers.
The Amazon CloudSearch team continually evaluates new features and rolls them
out proactively.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch has a highly
capable feature set to develop search systems. However, if you anticipate
strong customization on your search functionalities, Apache Solr or
Elasticsearch are better choices as their search core libraries are open sourced.
It is also important to note that any customization in the core libraries
leaves the build and deployment process responsibility to the developer. The
customization also needs to be maintained for every version upgrade or newer
release of your search engine.<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
</div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">
<span style="color: #f79646; font-size: 14pt; line-height: 115%;"><br clear="all" style="mso-break-type: section-break; page-break-before: always;" />
</span>
</span></span></div>
<div class="WordSection3">
<h1 style="margin-bottom: 12.0pt;">
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613356">Feature 11: More</a><o:p></o:p></span></h1>
<h2>
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613357">11.1 Client libraries</a><o:p></o:p></span></h2>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">Client libraries are
required for communicating with search engines. They are essential for
developers as they provide essential information to the connecting search
engine and allow applications to easily interact with high-level libraries.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><a href="https://wiki.apache.org/solr/IntegratingSolr#Solr_Client_Libraries_.2F_Language_Bindings">Apache Solr</a> has an open source
API client to interact with Solr using simple high-level methods. The client
libraries are available for PHP, Ruby, Rails, AJAX, Perl, Scala, Python,
.NET, and JavaScript.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Elasticsearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Elasticsearch provides official
clients for Groovy, JavaScript, .NET, PHP, Perl, Python, and Ruby. There are other community-provided client
libraries that can be integrated with Elasticsearch.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Open source</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Other than official and open source
client APIs, Elasticsearch and Apache Solr can be integrated using the
RESTful API. The REST client can use a typical web client developed in the
favored programming language or even called from a normal command line.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch exposes a RESTful
API for configuration, document service and search.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->The
configuration API can be used for CloudSearch domain creation, its
configuration and end to end management.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->The
document service API enables the user to add, replace, or delete documents in
your Amazon CloudSearch. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="text-indent: -0.25in;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span style="color: #9bbb59; font-size: 10pt;">•<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->The
search API is used for search or suggestion requests to your Amazon
CloudSearch domain.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Alternatively, AWS also shares a
downloadable SDK package, which simplifies coding. The SDK is available for
popular languages like Java, .NET, PHP, Python, and more. The SDK APIs are
built for most Amazon Web services, including Amazon S3, Amazon EC2,
CloudSearch, DynamoDB, and more. The SDK package includes the AWS library,
code samples, and documentation. <o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
</div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">
<span style="font-size: 11pt; line-height: 115%;"><br clear="all" style="mso-break-type: section-break; page-break-before: always;" />
</span>
</span></span></div>
<h1 style="margin-bottom: 12.0pt;">
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613358">Feature 12: Cost</a><o:p></o:p></span></h1>
<div class="MsoNormal">
<i><span style="font-family: Verdana, sans-serif;">From an overall
perspective, Cost is a very important factor and companies always endeavor ways
to reduce Total cost of Ownership (TCO). In this section, we will see the Cost components
in these three search engines.<o:p></o:p></span></i></div>
<table border="0" cellpadding="0" cellspacing="0" class="Stil2" style="border-collapse: collapse; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" valign="top" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;"><span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The cost factor in Apache Solr and
Elasticsearch includes infrastructure resources cost, managed services cost
and people resources cost. For any type of deployment, the servers cost and
engineers cost are essential. The commitment to continuous admin operations
depends on application requirements and its criticality.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Amazon CloudSearch</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Amazon CloudSearch cost component
includes server costs and engineers cost and they are essential for any
search deployment like the above two. Amazon CloudSearch being a
fully-managed service covers the managed services as part of the server
costs. Also, Amazon CloudSearch does not charge during the beginning of
service usage but charges at the end of the month based on CloudSearch usage.<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #6DC037; padding: 0in 5.4pt 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><b><span style="color: white; font-size: 14.0pt; mso-bidi-font-size: 11.0pt; mso-themecolor: background1;">Conclusion</span></b><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; padding: 0in 5.4pt; width: 451.3pt;" width="602"><div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">The net operating costs are
essentially the same across all three search engines, but people costs will
be 30% more for self-managed Apache Solr or Elasticsearch compared to Amazon
CloudSearch. <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">For Example, A highly important and
critical search application will require 24 * 7 support and managed services.
This cost incurred as part of Managed services which is an additional one in
Apache Solr and Elasticsearch deployments.</span></div>
</td></tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<div class="Title0">
<span style="font-family: Verdana, sans-serif;"><a href="https://www.blogger.com/null" name="_Toc422613359"><span style="font-size: large;"><b>Conclusion</b></span></a><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Search is an indispensable feature in most business
applications. <o:p></o:p></span></div>
<div class="MsoNormal" style="line-height: 115%; mso-yfti-cnfc: 64;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;">
</span></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Apache Solr and Elasticsearch are time proven solutions.
Many larger organizations have used Apache Solr and Elasticsearch for years,
but are now looking for greater operational efficiency and cost effectiveness.
On the other hand, companies looking for innovative ways grow their businesses
and provide value. In the recent years, a huge number technology companies have
started to employ the benefits of using cloud-based search services, mainly in
terms of getting started and then accommodating growth without the need to
switch vendors to do so. When
scalability, cost, and speed-to-market are primary concerns, we recommend using
some form of cloud service. And if you want to enjoy the benefits of a cloud
solution built on the architecture of Apache Solr, we recommend Amazon
CloudSearch.</span><o:p></o:p></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<b><br /></b></div>
<div class="MsoNormal">
<b><br /></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: 10.0pt;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div>
<div class="MsoNormal" style="margin-bottom: 10.0pt;">
<span lang="EN-IN"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-59299457969041634422015-06-16T19:46:00.001-07:002015-06-16T19:46:26.446-07:00Pointers for running a Regulated Pharma System in the Public Cloud<br />
<br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; font-family: Helvetica, Arial, sans-serif;"><tbody>
<tr><td style="border: 1pt solid windowtext; padding: 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<b><span style="font-family: Arial, sans-serif; font-size: 13.5pt;">Article Source:</span></b><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="font-family: Arial, sans-serif; font-size: 13.5pt;">https://www.linkedin.com/pulse/considerations-running-regulated-system-public-cloud-shah-nawaz</span><span style="font-size: 13.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td></tr>
</tbody></table>
<div style="color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; line-height: 24px;">
<strong style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-size: 20px; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"></strong></div>
<div class="MsoNormal" style="color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; line-height: 24px;">
<br /></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; font-stretch: inherit; line-height: 24px; margin-bottom: 30px; padding: 0px; vertical-align: baseline;">
As the Cloud becomes increasingly ubiquitous, the areas in which they can be adopted also becomes more widespread. Regulated industries are not immune to this. Soon we will all have to figure out how to run our workloads in the Cloud while maintaining our compliance and privacy posture. If we go back in time and look at each application that had a service regulated (GxP or 21 CFR Part 11 ) workload each piece of the software, platform, server, storage, network, even the data center was individually qualified and then individually validated to the appropriate regulation.<br />
<b><span style="color: #333333; font-family: Georgia, serif; font-style: italic;"> T</span><em style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-stretch: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">he Application should be validated; IT infrastructure should be qualified. (EU GMP Annex 11, 2011)</em></b><br />
Cloud is everywhere, so are we going to individually qualify and validate each building block? Certainly that seems like job security, but wouldn’t that make the cloud less compelling, and in some case unusable?<br />
<div class="MsoNormal">
<br /></div>
<strong style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-size: 20px; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">So who should do what for a GXp hosted application?</strong><br />
<strong style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-size: 20px; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><br /></strong></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; font-stretch: inherit; line-height: 24px; margin-bottom: 30px; padding: 0px; vertical-align: baseline;">
<strong style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><img alt="" class="center" data-loading-tracked="true" height="229" src="https://media.licdn.com/mpr/mpr/shrinknp_750_750/AAEAAQAAAAAAAAKaAAAAJDQyNTAxNDUzLWFhMDUtNGExZi05OGM3LTY4OTdmMWVlNWQ0YQ.png" style="border: 0px; box-sizing: border-box; display: block; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; height: auto; line-height: inherit; margin: 0px auto 15px; max-width: 100%; padding: 0px; text-align: center; vertical-align: baseline;" width="588" /></strong></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; font-stretch: inherit; line-height: 24px; margin-bottom: 30px; padding: 0px; vertical-align: baseline;">
If you have data privacy needs these should be tested as part of the validation test and formally documented and quality still needs to be addressed.<br />
<span style="color: #333333; font-family: Georgia, serif; font-style: italic;"><br /></span>
<span style="color: #333333; font-family: Georgia, serif; font-style: italic;">Platform Qualification documents are still needed when a regulated / validated application is hosted in a cloud environment “</span><br />
<br />
You also have to realize that the current concepts of computer system validation (CSV) do not work well, e.g how does one perform an installation qualification (IQ) in the cloud when one does not know the serial number of the machine on which the software will be installed, nor in some cases its location. <span class="underline" style="border: 0px; box-sizing: border-box; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: underline; vertical-align: baseline;"><strong style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">So we must pay attention to the purpose of the IQ , not to the implementation of the IQ </strong></span>and by extension, we must consider the purpose of CSV, not just its current practice. Any task carried out in the regulated domain should have at least the attributes i.e (Repeatability, Ability to Audit & Non-repudiation) whether paper-based or computer-based in house or in the cloud.<br />
<span style="color: #333333; font-family: Georgia, serif; font-style: italic;"><br /></span>
<span style="color: #333333; font-family: Georgia, serif; font-style: italic;">The idea is to controlling your data, and who can access it and what they do (and did to it…) upon accessed.</span><br />
<br />
The approach for compliance in the cloud needs to be different. If done correctly, compliance in the cloud can be far more efficient than any other means for providing complaint applications. Instead of worrying about qualifying each building block, the cloud vendor qualifies the platform once, to many standards and many certifications. Most of the Tier-1 cloud vendors provide these qualifications to any customer who needs to run a validated application on the cloud vendor’s platform. At least that is the approach many of the Cloud providers are taking. The provider qualifies the platform; the customer (or partner) validates the application.<br />
Whether you are considering Infrastructure Services i.e (AWS, Microsoft Azure, Google compute, Digital Ocean, Platform as a Service (AWS - Machine Learning MI, AWS RDS…), software as a service, thinking of putting your applications on Amazon Web Service, Azure or simply enabling your business with CRM (Veeva) , Office365 or google analytic, most providers offer detailed documentation and certifications across a wide range of standards.<br />
This enables their regulated customer and partners to run validated applications.<br />
<br />
Your situation may vary, as each customers QA has a different viewpoint of necessary qualifications and documentation to support a validated application. It is certainly no longer a question if one can qualify and/or validate their application to run in a cloud. It will come down to your Quality and Compliance process, and to what extend we can amend IQ/PQ/OQ to support distinct system categories running in cloud. For example, a clinical trail portal has different levels of risk then back-office HRMS applications and is thus validated to a different level. ONE-SIZE-FITS-ALL doesn’t works. One needs to understand the unique nature and risk profile of each application and then determine what is appropriate. It may have been convenient or perhaps a standard approach for traditional data-center delivered resources, but one needs to rethink some of those measures and practices that were adopted. Now that you are considering running them in a cloud each system would need to be evaluated for its validation posture.<br />
Various approached have been documented and talked about as it relates to running sensitive workloads in the Cloud, how life science organizations are using the Cloud across the value chain (Clinical Information System, Patient Data Archive, Patient Population Risk / Pattern Assessment, Genomic, and more…) and what levels of qualification documentation vendors must provide to customers in regulatory environments. This article considers some basic and fundamental approach to the Cloud.<br />
<strong style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-size: 20px; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><br /></strong>
<strong style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-size: 20px; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Security is Everyone Business:</strong><br />
Although security, by far remains the biggest concern, it can also be viewed as an enabler. However, one can’t build security services with the same develop and operate mindset that we have been using for decades. As we move into the cyber security era, machine-learning more pervasive techniques needs to be considered… (<em style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">There could be an entire article on this</em>…)<br />
<br />
<span style="color: #333333; font-family: Georgia, serif; font-style: italic;">"Design security in the context of the Cloud”.</span></div>
<h3 style="background-color: white; border: 0px; box-sizing: border-box; color: #333333; font-family: Helvetica, Arial, sans-serif; font-size: 20px; font-stretch: inherit; font-weight: normal; line-height: 26px; margin: 0px 0px 30px; padding: 0px; vertical-align: baseline;">
<strong style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Suggested Approach:</strong></h3>
<h3 style="background-color: white; border: 0px; box-sizing: border-box; color: #333333; font-family: Helvetica, Arial, sans-serif; font-size: 20px; font-stretch: inherit; font-weight: normal; line-height: 26px; margin: 0px 0px 30px; padding: 0px; vertical-align: baseline;">
<span style="color: #4d4f51; font-size: 16px; line-height: 24px;">Regulated or non-regulated, it is apparent that at the end of the day one needs assurance as to who has access and what they did with that state i.e ( Data/system). We also need to have traceability and repeatability by which you can audit/report and resolve issues that are prudent to any </span><strong style="border: 0px; box-sizing: border-box; color: #4d4f51; font-family: inherit; font-size: 16px; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">software development lifecycle</strong><span style="color: #4d4f51; font-size: 16px; line-height: 24px;">. ( </span><em style="border: 0px; box-sizing: border-box; color: #4d4f51; font-family: inherit; font-size: 16px; font-stretch: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Suggested Model below</em><span style="color: #4d4f51; font-size: 16px; line-height: 24px;"> )</span></h3>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; font-stretch: inherit; line-height: 24px; margin-bottom: 30px; padding: 0px; vertical-align: baseline;">
<em style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><img alt="" class="center" data-loading-tracked="true" height="361" src="https://media.licdn.com/mpr/mpr/shrinknp_750_750/AAEAAQAAAAAAAAIHAAAAJDRkYjY2ZjYyLTEzNzQtNDY1OS1hZTU1LWJiZmIxODU5YWIyNw.png" style="border: 0px; box-sizing: border-box; display: block; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; height: auto; line-height: inherit; margin: 0px auto 15px; max-width: 100%; padding: 0px; text-align: center; vertical-align: baseline;" width="588" /><em style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-stretch: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Regulatory considerations for the use of cloud computing will depend on the services you consume<em style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">. Not all services are created equally to satisfy compliance requirements</em></em></em><br />
<strong style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-size: 20px; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><br /></strong>
<strong style="border: 0px; box-sizing: border-box; color: #333333; font-family: inherit; font-size: 20px; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Rethink Operation Capabilities:</strong></div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; font-stretch: inherit; line-height: 24px; margin-bottom: 30px; padding: 0px; vertical-align: baseline;">
Your operation will be different, Tower Concept will not work nor can it scale to the ever-increasing demands. Consider adopting capabilities that can provide the appropriate assurance and governance to operate your Cloud services and at the same time promote agility and time to delivery service.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; font-stretch: inherit; line-height: 24px; margin-bottom: 30px; padding: 0px; vertical-align: baseline;">
<img alt="" class="center" data-loading-tracked="true" height="308" src="https://media.licdn.com/mpr/mpr/shrinknp_750_750/AAEAAQAAAAAAAAK-AAAAJGY2YWFlNWM4LTdiNDAtNDhhMy04Y2Q2LWVkMDE3ODIyNjRhYg.png" style="border: 0px; box-sizing: border-box; display: block; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; height: auto; line-height: inherit; margin: 0px auto 15px; max-width: 100%; padding: 0px; text-align: center; vertical-align: baseline;" width="588" />Image Source: 8kmiles.com</div>
<h3 style="background-color: white; border: 0px; box-sizing: border-box; color: #333333; font-family: Helvetica, Arial, sans-serif; font-size: 20px; font-stretch: inherit; font-weight: normal; line-height: 26px; margin: 0px 0px 30px; padding: 0px; vertical-align: baseline;">
Summary:</h3>
<ul style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; font-stretch: inherit; line-height: 24px; margin: 0px 0px 30px 40px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Codify your Infrastructure</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Pre-Qualify your environment prior to loading GxP workloads</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Build pre-validated images for Cloud usage</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Adopt a version control & release methodology</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Automate your stack as much as you can</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Automate your testing</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Test, test and test before you start to move workloads</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">To maximize effectiveness and minimize risk (and ultimately cost), security and privacy must be considered from the outset of any Cloud implementation not after implementation and deployment</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Cloud providers (Iaas and Paas) are generally not aware of a specific sectors security, privacy and regulatory needs of your sector, so design in the context of your organization</li>
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px 0px 5px; padding: 0px; vertical-align: baseline;">Adopt V-Model by Design (http://en.wikipedia.org/wiki/V-Model_(software_development))</li>
</ul>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-stretch: inherit; line-height: 24px; margin-bottom: 30px; padding: 0px; vertical-align: baseline;">
<div style="color: black; line-height: normal;">
<br /></div>
<div style="font-family: Helvetica, Arial, sans-serif; font-size: 16px;">
<b>References:</b></div>
<div style="font-family: Helvetica, Arial, sans-serif; font-size: 16px;">
<a href="http://en.wikipedia.org/wiki/Validation_%28drug_manufacture%29" saprocessedanchor="true" style="border: 0px; box-sizing: border-box; color: #96999c; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">http://en.wikipedia.org/wiki/Validation_%28drug_manufacture%29</a></div>
<div style="font-family: Helvetica, Arial, sans-serif; font-size: 16px;">
<a href="http://ec.europa.eu/health/files/eudralex/vol-4/annex11_01-2011_en.pdf" saprocessedanchor="true" style="border: 0px; box-sizing: border-box; color: #96999c; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">http://ec.europa.eu/health/files/eudralex/vol-4/annex11_01-2011_en.pdf</a></div>
<div style="font-family: Helvetica, Arial, sans-serif; font-size: 16px;">
<a href="http://www.ispe.org/gamp-5,http://www.prismforum.org" saprocessedanchor="true" style="border: 0px; box-sizing: border-box; color: #96999c; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">http://www.ispe.org/gamp-5</a></div>
<div style="font-family: Helvetica, Arial, sans-serif; font-size: 16px;">
<a href="http://www.ispe.org/gamp-5,http://www.prismforum.org" saprocessedanchor="true" style="border: 0px; box-sizing: border-box; color: #96999c; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">http://www.prismforum.org</a></div>
</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 16px; font-stretch: inherit; line-height: 24px; margin-bottom: 30px; padding: 0px; vertical-align: baseline;">
<br /></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-32170146959607756432015-06-16T10:36:00.001-07:002015-06-16T10:36:56.627-07:00Apache Solr to Amazon CloudSearch migration tool<br />
<div class="MsoNormal">
<span lang="EN-IN">In this post, we are introducing a new tool
called S2C – Apache Solr to Amazon CloudSearch Migration Tool. S2C is a Linux
console based utility that helps developers / engineers to migrate search index
from Apache Solr to Amazon CloudSearch.<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-IN">Very often customers initially build search
for their website or application on top of Solr, but later run into challenges
like elastic scaling and managing the Solr servers. This is a typical scenario
we have observed in our years of search implementation experience. For such use
cases, Amazon CloudSearch is a good choice. Amazon CloudSearch is a
fully-managed service in the cloud that makes it easy to set up, manage, and
scale a search solution for your website. To know more, please read the</span><span class="apple-converted-space"><span lang="EN-IN" style="color: #817d7d; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;"> </span></span><span lang="EN-IN"><a href="http://aws.amazon.com/documentation/cloudsearch/"><span style="color: #666666; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;">Amazon CloudSearch documentation</span></a><o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-IN">We are seeing growing trend every year,
organizations of various sizes are migrating their workloads to Amazon
CloudSearch and leveraging the benefits of fully managed service. For example,
Measured Search, an analytics and e-Commerce platform vendor, found it easier
to migrate to Amazon CloudSearch rather than scale Solr themselves (<a href="https://www.linkedin.com/pulse/3-reasons-why-amazon-cloudsearch-right-fit-our-magento-sameer-maggon"><span style="color: #666666; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;">see article</span></a></span><span class="apple-converted-space"><span lang="EN-IN" style="color: #817d7d; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;"> </span></span><span lang="EN-IN">for details).<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-IN">Since Amazon CloudSearch is built on top of
Solr, it exposes all the key features of Solr while providing the benefits of a
fully managed service in the cloud such as auto-scaling, self-healing clusters,
high availability, data durability, security and monitoring.<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-IN">In this post, we provide step-by-step
instructions on how to use the</span><span class="apple-converted-space"><span lang="EN-IN" style="color: #817d7d; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;"> </span></span><span lang="EN-IN"><a href="http://lucene.apache.org/solr/"><span style="color: #666666; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;">Apache Solr</span></a></span><span class="apple-converted-space"><span lang="EN-IN" style="color: #817d7d; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;"> </span></span><span lang="EN-IN">to</span><span class="apple-converted-space"><span lang="EN-IN" style="color: #817d7d; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;"> </span></span><span lang="EN-IN"><a href="http://aws.amazon.com/cloudsearch/"><span style="color: #666666; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;">Amazon
CloudSearch</span></a></span><span class="apple-converted-space"><span lang="EN-IN" style="color: #817d7d; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;"> </span></span><span lang="EN-IN">Migration (S2C) tool to
migrate from Apache Solr to Amazon CloudSearch.<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-IN">Before we get into detail, you can download
the S2C tool in the below link.<br />
</span><strong><span lang="EN-IN" style="color: #817d7d; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;">Download Link:</span></strong><span class="apple-converted-space"><b><span lang="EN-IN" style="color: #817d7d; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;"> </span></b></span><span lang="EN-IN"><a href="https://s3-us-west-2.amazonaws.com/s2c-tool/s2c-cli.zip"><span style="color: #666666; font-family: "Arial","sans-serif"; font-size: 10.5pt; line-height: 115%;">https://s3-us-west-2.amazonaws.com/s2c-tool/s2c-cli.zip</span></a><o:p></o:p></span></div>
<h2>
<u><span lang="X-NONE" style="color: windowtext; font-size: 12.0pt; line-height: 115%; mso-bidi-font-size: 14.0pt;">Pre-Requisites</span></u><u><span lang="EN-IN" style="color: windowtext; font-size: 12.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 14.0pt;"><o:p></o:p></span></u></h2>
<div class="MsoNormal">
<span lang="EN-IN">Before starting
the migration, the following pre-requisites have to be met. The pre-requisites
include installations and configuration on the migration server. The migration
server could be the same Solr server or independent server that sits between
your Solr server and Amazon CloudSearch instance. <o:p></o:p></span></div>
<div class="MsoNormal">
<i><span lang="EN-IN" style="line-height: 115%;">Note: We recommend running the migration from the Solr server instead
of independent server as it can save time and bandwidth. It is much better if
the Solr server is hosted on EC2 as the latency between EC2 and CloudSearch is
relatively less.<span style="font-size: 10pt;"><o:p></o:p></span></span></i></div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; margin-left: 5.4pt; mso-border-alt: solid black .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid black 1.0pt; mso-border-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 456.7pt;" valign="top" width="609"><div class="MsoListParagraphCxSpFirst" style="margin: 0in 0in 0.0001pt 0.25in;">
The following installations and configuration should be done on the
migration server (i.e. your Solr server or any new independent server that
connects between your Solr machine and Amazon CloudSearch). <o:p></o:p></div>
<div class="MsoListParagraphCxSpLast" style="margin: 0in 0in 0.0001pt 0.25in;">
<o:p> </o:p> </div>
<div class="MsoListParagraphCxSpFirst" style="margin: 0in 0in 0.0001pt 0.25in; text-indent: -0.25in;">
<!--[if !supportLists]--><span lang="EN-IN">1.<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]-->The application is developed using Java. <a href="http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html"><span style="color: #666666; font-family: "Arial","sans-serif"; font-size: 10.5pt;">Download
and Install Java 8 </span></a>.Validate the JDK path and ensure the
environment variables like JAVA_HOME, classpath, path is set correctly.<span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
<br /></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in; text-indent: -0.25in;">
<!--[if !supportLists]--><span lang="EN-IN">2.<span style="font-size: 7pt; font-stretch: normal;">
</span></span><!--[endif]--><span lang="EN-IN">We assume you
already have setup Amazon Web services IAM account. Please ensure the IAM
user has right permissions to access AWS services like CloudSearch.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
<i><span lang="EN-IN">Note: If you do not have an AWS IAM account with above mentioned
permissions, you cannot proceed further.<o:p></o:p></span></i></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
<br /></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in; text-indent: -0.25in;">
<!--[if !supportLists]-->3.<span style="font-size: 7pt; font-stretch: normal;">
</span><!--[endif]-->The IAM user should have AWS Access key and
Secret key. In the application hosting server, set up the Amazon environment
variables for access key and secret key. It is important that the application
runs using the AWS environment variables. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
To setup AWS environment variables, please read the below link.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
<a href="http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html"><span lang="EN-IN" style="font-size: 11.0pt; mso-ansi-language: EN-IN; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;">http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html</span></a><span lang="EN-IN" style="font-size: 11.0pt; mso-ansi-language: EN-IN; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;"> <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 0in 0in 0.0001pt 0.25in;">
<a href="http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html"><span lang="EN-IN" style="font-size: 11.0pt; mso-ansi-language: EN-IN; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;">http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html</span></a><span lang="EN-IN" style="font-size: 11.0pt; mso-ansi-language: EN-IN; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 0.25in;">
<span lang="EN-IN">Alternatively, you can set the following AWS
environment variables by running the commands below from Linux console.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 0.25in;">
<br /></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 0.25in;">
<i><span lang="EN-IN" style="background: lightgrey; font-family: Consolas; font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE; mso-highlight: lightgrey;">export
AWS_ACCESS_KEY=Access Key <o:p></o:p></span></i></div>
<div class="MsoListParagraphCxSpFirst" style="margin: 0in 0in 0.0001pt 0.25in;">
<i><span lang="EN-IN" style="background: lightgrey; font-family: Consolas; font-size: 10.0pt; mso-ansi-language: EN-IN; mso-fareast-language: X-NONE; mso-highlight: lightgrey;">export
AWS_SECRET_KEY=Secret Key</span></i><i><span lang="EN-IN" style="font-family: Consolas; font-size: 10.0pt; mso-ansi-language: EN-IN; mso-fareast-language: X-NONE;"> <o:p></o:p></span></i></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
<br /></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in; text-indent: -0.25in;">
<!--[if !supportLists]-->4.<span style="font-size: 7pt; font-stretch: normal;">
</span><!--[endif]--><i>Note: This step
is applicable only if the application is hosted on Amazon EC2.</i><o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
If you do not have an AWS Access key and Secret key, you can opt for
IAM role attached to an EC2 instance. A new IAM role can be created and
attached to EC2 during the instance launch. The IAM role should have access
to AWS resources like S3, DynamoDB and CloudSearch. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
For more information, read the below link<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
<a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html"><span style="font-size: 11.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;">http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html</span></a><span style="font-size: 11.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;">
<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
<br /></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in; text-indent: -0.25in;">
<!--[if !supportLists]-->5.<span style="font-size: 7pt; font-stretch: normal;">
</span><!--[endif]-->Download the migration utility ‘S2C’, unzip the
tool and copy it in your working directory.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 0.0001pt 0.25in;">
<o:p> </o:p>Download Link : <a href="https://s3-us-west-2.amazonaws.com/s2c-tool/s2c-cli.zip">https://s3-us-west-2.amazonaws.com/s2c-tool/s2c-cli.zip</a>
</div>
<div class="MsoListParagraphCxSpLast" style="margin: 0in 0in 0.0001pt 0.25in;">
<o:p></o:p></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<div class="MsoListParagraph" style="margin-bottom: .0001pt; margin: 0in; mso-add-space: auto;">
<b><u><span lang="EN-IN">S2C Utility File<o:p></o:p></span></u></b></div>
<div class="MsoNormal">
<span lang="EN-IN">The downloaded ‘S2C’
migration utility s</span><span lang="EN-IN">hould
have the following sub directories and files. <o:p></o:p></span></div>
<table border="0" cellpadding="0" cellspacing="0" class="MsoTableMediumShading2Accent3" style="border-collapse: collapse; border: none; margin-left: 5.4pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-IN" style="color: white; font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE; mso-themecolor: background1;">Folder / Files<o:p></o:p></span></b></div>
</td>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-IN" style="color: white; font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE; mso-themecolor: background1;">Description<o:p></o:p></span></b></div>
</td>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-IN" style="color: white; font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE; mso-themecolor: background1;">bin<o:p></o:p></span></b></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE;">Binaries of the migration tool<o:p></o:p></span></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-IN" style="color: white; font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE; mso-themecolor: background1;">lib<o:p></o:p></span></b></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE;">Libraries required for migration<o:p></o:p></span></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-IN" style="color: white; font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE; mso-themecolor: background1;">application.conf<o:p></o:p></span></b></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE;">Configuration file that allows end users to
input parameters<o:p></o:p></span></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE;">Require end-user’s input. <o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-IN" style="color: white; font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE; mso-themecolor: background1;">logback.xml<o:p></o:p></span></b></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE;">Log file configuration <o:p></o:p></span></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE;">Optional. Does not require end-user / developer input<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #D8D8D8; mso-background-themecolor: background1; mso-background-themeshade: 216; padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="background: #9BBB59; mso-background-themecolor: accent3; padding: 0in 5.4pt 0in 5.4pt; width: 111.05pt;" valign="top" width="148"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-IN" style="color: white; font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE; mso-themecolor: background1;">s2c<o:p></o:p></span></b></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 196.95pt;" valign="top" width="263"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 11.0pt; mso-fareast-language: X-NONE;">script file that executes the migration process<o:p></o:p></span></div>
</td>
<td style="padding: 0in 5.4pt 0in 5.4pt; width: 143.3pt;" valign="top" width="191"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b><u><span lang="EN-IN" style="background: lightgrey; mso-highlight: lightgrey;">Configure only
application.conf and logback.xml. Do not
modify any other file.
</span><span lang="EN-IN"> <o:p></o:p></span></u></b></div>
<div class="MsoNormal">
<b><u><span lang="EN-IN">application.conf<o:p></o:p></span></u></b></div>
<div class="MsoNormal">
<span lang="EN-IN">The application.conf
file has the configuration related to the new Amazon CloudSearch domain that
will be created. The parameters configured in the in the <b>application.conf </b>file are explained in the table below.<o:p></o:p></span></div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; margin-left: 5.4pt; mso-border-alt: solid black .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-table-layout-alt: fixed; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid black 1.0pt; mso-border-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 205.55pt;" width="274"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">s2c {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> api {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> SchemaParser =
"s2c.impl.solr.DefaultSchemaParser"<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> SchemaConverter =
"s2c.impl.cs.DefaultSchemaConverter"<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> DataFetcher =
"s2c.impl.solr.DefaultDataFetcher"<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> DataPusher =
"s2c.impl.cs.DefaultDataPusher"
}<o:p></o:p></span></div>
</td>
<td style="border-left: none; border: solid black 1.0pt; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">List of API that is executed
step by step during the migration. <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><i><span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Do not change this.<o:p></o:p></span></i></b></div>
</td>
</tr>
<tr style="height: 21.0pt; mso-yfti-irow: 1;">
<td style="background: #1F497D; border-top: none; border: solid black 1.0pt; height: 21.0pt; mso-background-themecolor: text2; mso-border-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 205.55pt;" width="274"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #1F497D; border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: 21.0pt; mso-background-themecolor: text2; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr style="height: 21.0pt; mso-yfti-irow: 2;">
<td rowspan="3" style="border-top: none; border: solid black 1.0pt; height: 21.0pt; mso-border-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 205.55pt;" width="274"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> solr {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> dir = "files"<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> server-url =
"http://localhost:8983/solr/collection1"<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> fetch-limit = 100<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> }<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: 21.0pt; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">dir – The base directory path
of Solr. <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Ensure the directory is
present and also its validity.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<i><span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Eg:/opt/solr/example/solr/collection1/conf<o:p></o:p></span></i></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr style="height: 21.0pt; mso-yfti-irow: 3;">
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: 21.0pt; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">server-url – Server host, port
and collection path<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">The endpoint which will be
used to fetch the data. <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">If the utility is run from a different
server, ensure the IP address and port has firewall access.<o:p></o:p></span></div>
</td>
</tr>
<tr style="height: 21.0pt; mso-yfti-irow: 4;">
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: 21.0pt; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">fetch-limit – number of solr
documents that can be fetched for each batch call<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">This configuration number
should be carefully set by the developer. <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">The fetch limit depends on the
following factors:</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<i style="text-indent: -0.25in;"><span style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Courier \(W1\)"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: "Courier \(W1\)"; mso-fareast-language: X-NONE;">1.<span style="font-family: 'Times New Roman'; font-size: 7pt; font-stretch: normal; font-style: normal;"> </span></span></i><i style="text-indent: -0.25in;"><span style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;">Record size
of a Solr record(1KB or 2KB)</span></i></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<i style="text-indent: -0.25in;"><span style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Courier \(W1\)"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: "Courier \(W1\)"; mso-fareast-language: X-NONE;">2.<span style="font-family: 'Times New Roman'; font-size: 7pt; font-stretch: normal; font-style: normal;"> </span></span></i><i style="text-indent: -0.25in;"><span style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;">Latency
between migration server and Amazon CloudSearch</span></i></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<i style="text-indent: -0.25in;"><span style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Courier \(W1\)"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: "Courier \(W1\)"; mso-fareast-language: X-NONE;">3.<span style="font-family: 'Times New Roman'; font-size: 7pt; font-stretch: normal; font-style: normal;"> </span></span></i><i style="text-indent: -0.25in;"><span style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;">Current
Request Load on the Solr Server</span></i></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<i><span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">E.g.: If the total Solr documents is 100000 and
fetch limit is 100, then it would take 100000 / 10 = 10000 batch calls to
complete the fetch. <o:p></o:p></span></i></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<i><span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">If size of each Solr record is 2KB, then 100000
* 2KB = 200MB data is migrated.<o:p></o:p></span></i></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr style="height: .25in; mso-yfti-irow: 5;">
<td style="background: #1F497D; border-top: none; border: solid black 1.0pt; height: .25in; mso-background-themecolor: text2; mso-border-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 205.55pt;" width="274"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #1F497D; border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: .25in; mso-background-themecolor: text2; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr style="height: .25in; mso-yfti-irow: 6;">
<td rowspan="5" style="border-top: none; border: solid black 1.0pt; height: .25in; mso-border-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 205.55pt;" width="274"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> cs {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> domain = "collection1"<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> region = "us-east-1"<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> instance-type = "</span><span lang="EN-IN"> </span><span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">search.m3.xlarge"<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> partition-count = 1<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> replication-count = 1<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;"> }<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: .25in; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">domain - CloudSearch domain
name <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Ensure that the domain name
does not already exist.<o:p></o:p></span></div>
</td>
</tr>
<tr style="height: .25in; mso-yfti-irow: 7;">
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: .25in; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Region – AWS region for the
new CloudSearch domain <o:p></o:p></span></div>
</td>
</tr>
<tr style="height: .25in; mso-yfti-irow: 8;">
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: .25in; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Instance type – Desired
instance type for CloudSearch nodes<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Choose the instance type based
on the volume of data and the expected query volume.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr style="height: .25in; mso-yfti-irow: 9;">
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: .25in; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Partition count – Number of partitions
required for CloudSearch <o:p></o:p></span></div>
</td>
</tr>
<tr style="height: .25in; mso-yfti-irow: 10;">
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: .25in; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">replication-count -
Replication count for CloudSearch <o:p></o:p></span></div>
</td>
</tr>
<tr style="height: .25in; mso-yfti-irow: 11;">
<td style="background: #1F497D; border-top: none; border: solid black 1.0pt; height: .25in; mso-background-themecolor: text2; mso-border-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 205.55pt;" width="274"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
<td style="background: #1F497D; border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; height: .25in; mso-background-themecolor: text2; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid black 1.0pt; mso-border-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 205.55pt;" width="274"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">wd = "/tmp"<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-alt: solid black .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 267.85pt;" width="357"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-IN" style="font-family: "Courier \(W1\)"; font-size: 9.0pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Temporary file path to store
intermediate data files and migration log files<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<h1>
<u><span lang="EN-IN" style="font-size: 12.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 16.0pt;">Running the migration</span></u><u><span lang="EN-IN" style="font-size: 12.0pt; line-height: 115%; mso-bidi-font-size: 16.0pt;">
</span></u><u><span lang="EN-IN" style="font-size: 12.0pt; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 16.0pt;"><o:p></o:p></span></u></h1>
<div class="MsoNormal" style="margin-bottom: 6.0pt;">
<span lang="EN-IN">Before launching the S2C migration tool,
verify the following:<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin: 0in 0in 6pt 17.85pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span lang="EN-IN">1.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Solr
directory path – Make sure that the Solr directory path is valid and available.
The tool cannot read the configuration if the path or directory is invalid. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 6pt 17.85pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span lang="EN-IN">2.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Solr
configuration contents - Validate that the Solr configuration contents are correctly
set inside the directory. Example: solrconfig.xml, schema.xml, stopwords.txt,
etc.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 6pt 17.85pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span lang="EN-IN">3.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Make
sure that the working directory is present in the file system and has write
permissions for the current user. It can be an existing directory or a new
directory. The working directory stores the fetched data from Solr and
migration logs.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 6pt 17.85pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span lang="EN-IN">4.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Validate
the disk size before starting the migration. If the available free disk space
is lesser than the size of the Solr index, the fetch operations will fail. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 6pt 17.85pt;">
<span lang="EN-IN">For example,
if the Solr index size is 7 GB, make sure that the disk has at least 10 GB to
20 GB of free space.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 6pt 17.85pt;">
<i><span lang="EN-IN">Note: The tool reads the data from Solr and
stores in a temporary directory (Please see configuration <b><u>wd = /tmp</u></b> in the above table).<o:p></o:p></span></i></div>
<div class="MsoListParagraphCxSpMiddle" style="margin: 0in 0in 6pt 17.85pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span lang="EN-IN">5.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Verify that
the AWS environment variables are set correctly. The AWS environment variables
are mentioned in the </span>pre-requisites
section above.<span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast" style="margin: 0in 0in 6pt 17.85pt; text-indent: -0.25in;">
<!--[if !supportLists]--><span lang="EN-IN">6.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Validate
the firewall rules for IP address and ports if the migration tool is run from a
different server or instance. Example: Solr default port 8983 should be opened
to the EC2 instance executing this tool.<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 6.0pt;">
<span lang="EN-IN">Run the following command from directory ‘<b>{S2C filepath}’ </b>example: /build/install/s2c-cli
<o:p></o:p></span></div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; margin-left: 5.4pt; mso-border-alt: solid black .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid black 1.0pt; mso-border-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 473.4pt;" valign="top" width="631"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<i><span lang="EN-IN" style="font-family: Consolas; font-size: 11.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">./s2c<o:p></o:p></span></i></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span lang="EN-IN" style="font-family: Consolas; font-size: 11.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">Or <o:p></o:p></span></b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<i><span lang="EN-IN" style="font-family: Consolas; font-size: 11.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">JVM_OPTS="-Xms2048m
-Xmx2048m" ./s2c </span></i><span lang="EN-IN" style="font-family: Consolas; font-size: 11.0pt; mso-bidi-font-size: 10.0pt; mso-fareast-font-family: Calibri; mso-fareast-language: X-NONE;">(<b>With
heap size</b>)</span><span lang="EN-IN"><o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: 6pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 6.0pt;">
<span lang="EN-IN">The above will invoke the shell ‘s2c’
script that starts the search migration process. </span><span lang="EN-IN">The migration
process is a series of steps that require user inputs as shown in the screen
shots below. <o:p></o:p></span></div>
<div class="MsoNormal">
<u><span lang="EN-IN">Step 1: Parse the Solr schema <o:p></o:p></span></u></div>
<div class="MsoNormal">
<span lang="EN-IN">The first step of migration prompts for a confirmation
to parse the Solr schema and Solr configuration file. During this step, the
application generates a ‘Run Id’ folder inside the working directory.<o:p></o:p></span></div>
<div class="MsoNormal">
<b><i><span lang="EN-IN" style="font-family: Consolas; line-height: 115%; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;">Example:
/tmp/s2c/m1416220194655<o:p></o:p></span></i></b><br />
<b><i><span lang="EN-IN" style="font-family: Consolas; line-height: 115%; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;"><br /></span></i></b>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3uwslH0zXgNvVrSbh2ZpVlYhi8cQ5Aj_BM6C1gTTHu6eLRcqQNV4SH6XudzAnkBElGwxLUfV45jsJql-Ih0OQg-MIYUwaPYElQEvwbBXAcXJ2XDbMGUGhd3xpfeOpsvyJHi4GeEXBiQLi/s1600/Solr+to+CloudSearch+Migration+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3uwslH0zXgNvVrSbh2ZpVlYhi8cQ5Aj_BM6C1gTTHu6eLRcqQNV4SH6XudzAnkBElGwxLUfV45jsJql-Ih0OQg-MIYUwaPYElQEvwbBXAcXJ2XDbMGUGhd3xpfeOpsvyJHi4GeEXBiQLi/s640/Solr+to+CloudSearch+Migration+1.png" width="640" /></a></div>
<b><i><span lang="EN-IN" style="font-family: Consolas; line-height: 115%; mso-bidi-font-size: 10.0pt; mso-fareast-language: X-NONE;"><br /></span></i></b></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shapetype
id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t"
path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f">
<v:stroke joinstyle="miter"/>
<v:formulas>
<v:f eqn="if lineDrawn pixelLineWidth 0"/>
<v:f eqn="sum @0 1 0"/>
<v:f eqn="sum 0 0 @1"/>
<v:f eqn="prod @2 1 2"/>
<v:f eqn="prod @3 21600 pixelWidth"/>
<v:f eqn="prod @3 21600 pixelHeight"/>
<v:f eqn="sum @0 0 1"/>
<v:f eqn="prod @6 1 2"/>
<v:f eqn="prod @7 21600 pixelWidth"/>
<v:f eqn="sum @8 21600 0"/>
<v:f eqn="prod @7 21600 pixelHeight"/>
<v:f eqn="sum @10 21600 0"/>
</v:formulas>
<v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/>
<o:lock v:ext="edit" aspectratio="t"/>
</v:shapetype><v:shape id="Picture_x0020_21" o:spid="_x0000_i1036" type="#_x0000_t75"
style='width:380.25pt;height:238.5pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image001.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-IN">The Run Id is a unique identifier for each
migration. Note down the Run Id as you will need it to resume the migration in
case of any failure. <o:p></o:p></span></div>
<div class="MsoNormal">
<br />
<br /></div>
<div class="MsoNormal">
<a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=4836510665235566234" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><u><span lang="EN-IN">Step 2: Schema conversion from Solr to
CloudSearch<o:p></o:p></span></u><br />
The second step prompts confirmation to convert
Solr schema to CloudSearch schema. Press any key to proceed further.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_dKqK2SqWxjq5mDmTftbZqIB9Oc6Bv1MchAdiWUgNz-WWzfaaBZ_uXlT0I_AHQjNw6cq4VeWUlJdCZqZo6Ozsmafj6xWt7BwxUXrmJjjeKoGfYDvHMyxIPDaniKoX6zJ2oCjdm3WvyCGh/s1600/Solr+to+CloudSearch+Migration+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_dKqK2SqWxjq5mDmTftbZqIB9Oc6Bv1MchAdiWUgNz-WWzfaaBZ_uXlT0I_AHQjNw6cq4VeWUlJdCZqZo6Ozsmafj6xWt7BwxUXrmJjjeKoGfYDvHMyxIPDaniKoX6zJ2oCjdm3WvyCGh/s640/Solr+to+CloudSearch+Migration+2.png" width="640" /></a></div>
<br /></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_29" o:spid="_x0000_i1035" type="#_x0000_t75" style='width:380.25pt;
height:238.5pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image003.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_30" o:spid="_x0000_i1034" type="#_x0000_t75" style='width:380.25pt;
height:238.5pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image005.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiii2Mzsw-nbWl0VJekWSu5Dj5CrfTBMFRyUJOhsUm51TlhqwaRErXBlb5dykNl6AiyvzCZo3yLhQvIPUOQpYTZDwtQ9l5jMUSgJm6eI19Ct-1ViJsN-586hMgGEEzIkjxx55020FtHFeAV/s1600/Solr+to+CloudSearch+Migration+3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiii2Mzsw-nbWl0VJekWSu5Dj5CrfTBMFRyUJOhsUm51TlhqwaRErXBlb5dykNl6AiyvzCZo3yLhQvIPUOQpYTZDwtQ9l5jMUSgJm6eI19Ct-1ViJsN-586hMgGEEzIkjxx55020FtHFeAV/s640/Solr+to+CloudSearch+Migration+3.png" width="640" /></a></div>
<span lang="EN-IN"><br /></span>
<span lang="EN-IN"><br /></span>
<span lang="EN-IN">The second step will also list all the converted
fields which are ready to be migrated from Solr to CloudSearch. If any fields
are left out, this step will allow you to correct the original schema. User can
abort the migration and identify the ignored fields, rectify the schema and
re-run the migration again.<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-IN">The below screen shot shows the fields ready for
CloudSearch migration. <o:p></o:p></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2Ay3GEz0Gb5qY4xeUTvkMP4jNC52Z_ri04RYE7SzBdtpnBG7ROHJn1se5bSrTb2fp-4_M8WakX4BmFafneTUNmAFVILDQ18NjVJeQ1TeeqMvrD71ibK92laBul4gPipQA_Gij6ZpbayUL/s1600/Solr+to+CloudSearch+Migration+4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2Ay3GEz0Gb5qY4xeUTvkMP4jNC52Z_ri04RYE7SzBdtpnBG7ROHJn1se5bSrTb2fp-4_M8WakX4BmFafneTUNmAFVILDQ18NjVJeQ1TeeqMvrD71ibK92laBul4gPipQA_Gij6ZpbayUL/s640/Solr+to+CloudSearch+Migration+4.png" width="640" /></a></div>
<span lang="EN-IN"><br /></span>
<span lang="EN-IN"><br /></span></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_22" o:spid="_x0000_i1033" type="#_x0000_t75" style='width:381pt;
height:239.25pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image007.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<u><span lang="EN-IN">Step 3: Data Fetch<o:p></o:p></span></u></div>
<div class="MsoNormal">
<span lang="EN-IN">The third step prompts for confirmation to fetch
the search index data from the Solr server. Press any key to proceed. </span><span lang="EN-IN">This step will generate a temporary file which will be stored in the
working directory. This temporary file will have all the fetched documents from
the Solr index.</span><span lang="EN-IN"><o:p></o:p></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_kl1xPik7UZWs9kLIUrxV4eldk8aa0JGsq4YGtFhBzIzgNQ-WMDWmCT6nhB9bRmH3GpjIDqXJLwAGuyKZ6M26GuXwhuu6Y0Fa-8YdKR13mb08l3O_itprAA1cRRt8u9XQA7Z5jwYqO-0w/s1600/Solr+to+CloudSearch+Migration+5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_kl1xPik7UZWs9kLIUrxV4eldk8aa0JGsq4YGtFhBzIzgNQ-WMDWmCT6nhB9bRmH3GpjIDqXJLwAGuyKZ6M26GuXwhuu6Y0Fa-8YdKR13mb08l3O_itprAA1cRRt8u9XQA7Z5jwYqO-0w/s640/Solr+to+CloudSearch+Migration+5.png" width="640" /></a></div>
<span lang="EN-IN"><br /></span></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_23" o:spid="_x0000_i1032" type="#_x0000_t75" style='width:381pt;
height:239.25pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image009.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_24" o:spid="_x0000_i1031" type="#_x0000_t75" style='width:381pt;
height:239.25pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image011.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN" style="font-family: "Courier \(W1\)"; mso-fareast-language: X-NONE;"><o:p></o:p></span></div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5DzC_jZZbVXJTJMdIyDrSzPz4LA491sMLH0jCkIp2EyMGFOSAQ67YZGu0H9ggS4N16ltkKBn1YJoVKTaJUJ-b8cF11bsKAygb0Ckhy5XfzMgpvekyFRINMBfEGBJ5cQ4UPOfstl2iqAB5/s1600/Solr+to+CloudSearch+Migration+6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5DzC_jZZbVXJTJMdIyDrSzPz4LA491sMLH0jCkIp2EyMGFOSAQ67YZGu0H9ggS4N16ltkKBn1YJoVKTaJUJ-b8cF11bsKAygb0Ckhy5XfzMgpvekyFRINMBfEGBJ5cQ4UPOfstl2iqAB5/s640/Solr+to+CloudSearch+Migration+6.png" width="640" /></a></div>
<span lang="EN-IN"><br /></span>
There is also option to skip the fetch
process if all the Solr data is already stored in the temporary file. If this
is the case, the prompt will look like the screenshot below.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi49rk6J6w93_VqzgH4ZwzGWI0CQmY8z95y0qIUN8ED_ERA2mJmBKmQb9szvIY-2nHtz4We9XP-GYd6swqxamrDicwaBQS10GmTWozDaGjmcpkNY1g8eodAXh2DL2GRLo3_xRVkfmcLg9pG/s1600/Solr+to+CloudSearch+Migration+7.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi49rk6J6w93_VqzgH4ZwzGWI0CQmY8z95y0qIUN8ED_ERA2mJmBKmQb9szvIY-2nHtz4We9XP-GYd6swqxamrDicwaBQS10GmTWozDaGjmcpkNY1g8eodAXh2DL2GRLo3_xRVkfmcLg9pG/s640/Solr+to+CloudSearch+Migration+7.png" width="640" /></a></div>
<br />
<br /></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_25" o:spid="_x0000_i1030" type="#_x0000_t75" style='width:381pt;
height:239.25pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image013.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<u><span lang="EN-IN">Step 4: Data push to CloudSearch<o:p></o:p></span></u></div>
<div class="MsoNormal">
<span lang="EN-IN">The
last and final step prompts for confirmation to push the search data from the temporary
file store to Amazon CloudSearch. This step also creates the CloudSearch domain
with the configuration specified in </span><span lang="EN-IN">application.conf</span><span lang="EN-IN"> including desired instance
type, replication count, and multi-AZ options.<o:p></o:p></span></div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikvCKD2_jKG2az7K7esocHS0b138NeW4DNDcZZ-6afw5Zg8B-NnWBh7znexXN1j2CR77Zp-GlF1jmf1rfyT-Cq7ejbaPQTXYFqEHmenJATpuZrgJYbQq0G0m21ioQhxzb-toV9pPmMFIoj/s1600/Solr+to+CloudSearch+Migration+8.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikvCKD2_jKG2az7K7esocHS0b138NeW4DNDcZZ-6afw5Zg8B-NnWBh7znexXN1j2CR77Zp-GlF1jmf1rfyT-Cq7ejbaPQTXYFqEHmenJATpuZrgJYbQq0G0m21ioQhxzb-toV9pPmMFIoj/s640/Solr+to+CloudSearch+Migration+8.png" width="640" /></a></div>
<br />
<br /></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_34" o:spid="_x0000_i1029" type="#_x0000_t75" style='width:381pt;
height:239.25pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image015.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-IN">If the domain is already created, the
utility will prompt to use the existing domain. If you do not wish to use an
existing domain, you can create a new CloudSearch domain using the same prompt.
<o:p></o:p></span></div>
<div class="MsoNormal">
<i><span lang="EN-IN">Note:
The console does not prompt for any ‘CloudSearch domain name’ but instead it
uses the domain name configured in the application.conf file.<o:p></o:p></span></i><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd9WhLOl4V8Ina-KnhEeryfkWkjcXOG932H6-bBg2Xx6-XCxk_kTrDyZ7-8CketCG6RR_XMz7SvLS_CGkQx0_GzGftKfUBpt3GAlBKdzcY2A9nZYvFK-K2R5oehnhltRyNNu5nf_j0jG5w/s1600/Solr+to+CloudSearch+Migration+9.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd9WhLOl4V8Ina-KnhEeryfkWkjcXOG932H6-bBg2Xx6-XCxk_kTrDyZ7-8CketCG6RR_XMz7SvLS_CGkQx0_GzGftKfUBpt3GAlBKdzcY2A9nZYvFK-K2R5oehnhltRyNNu5nf_j0jG5w/s640/Solr+to+CloudSearch+Migration+9.png" width="640" /></a></div>
<i><span lang="EN-IN"><br /></span></i></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_26" o:spid="_x0000_i1028" type="#_x0000_t75" style='width:381pt;
height:239.25pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image017.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<u><span lang="EN-IN">Step 5: Resume (Optional)<o:p></o:p></span></u></div>
<div class="MsoNormal">
<span lang="EN-IN">During
the migration steps, if there is any failure during the fetch operation, it can
be resumed. This is illustrated in the screen shot below. <o:p></o:p></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrIFSrVgXUc1XjvfeEYVdvtrDmOKZM2fpUSxV8vLNDPsnbZGQVSrINf1v_KGBtK_MnKExyM1YcFA4sDNvwg76SvOH9kTFIvvmUsrqUhqhkv6YSJ8nUQqtPYQxfBmb1OkcvO1WqqXvfjfPD/s1600/Solr+to+CloudSearch+Migration+10.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrIFSrVgXUc1XjvfeEYVdvtrDmOKZM2fpUSxV8vLNDPsnbZGQVSrINf1v_KGBtK_MnKExyM1YcFA4sDNvwg76SvOH9kTFIvvmUsrqUhqhkv6YSJ8nUQqtPYQxfBmb1OkcvO1WqqXvfjfPD/s640/Solr+to+CloudSearch+Migration+10.png" width="640" /></a></div>
<span lang="EN-IN"><br /></span>
<span lang="EN-IN"><br /></span></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_38" o:spid="_x0000_i1027" type="#_x0000_t75" style='width:381pt;
height:239.25pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image019.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<u><span lang="EN-IN">Step 6: Verification <o:p></o:p></span></u></div>
<div class="MsoNormal">
<span lang="EN-IN">Log
into AWS CloudSearch management console to verify that the domain and index
fields. <o:p></o:p></span></div>
<div class="MsoNormal">
<!--[if gte vml 1]><v:shape
id="Picture_x0020_37" o:spid="_x0000_i1026" type="#_x0000_t75" style='width:406.5pt;
height:228pt;visibility:visible;mso-wrap-style:square'>
<v:imagedata src="file:///C:\Users\HARISH~1\AppData\Local\Temp\msohtmlclip1\01\clip_image021.png"
o:title=""/>
</v:shape><![endif]--><!--[if !vml]--><!--[endif]--><span lang="EN-IN"><o:p></o:p></span></div>
<div class="MsoNormal">
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG6qCgumgz7y3WDZGhVj_I_iCeWFJumwyPdsRygSiRHQ6SAABhE82McKvzbqUywTyLtB3gXjO77bvijt9185lEm2McqiT7cU7pV6khzjL6RRVgQW1w3zyGmqcd0aG8KCWg7XRHx92DN3JT/s1600/Solr+to+CloudSearch+Migration+11.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG6qCgumgz7y3WDZGhVj_I_iCeWFJumwyPdsRygSiRHQ6SAABhE82McKvzbqUywTyLtB3gXjO77bvijt9185lEm2McqiT7cU7pV6khzjL6RRVgQW1w3zyGmqcd0aG8KCWg7XRHx92DN3JT/s640/Solr+to+CloudSearch+Migration+11.png" width="640" /></a></div>
<br />
<br /></div>
<div class="MsoNormal">
<span lang="EN-IN">Amazon
CloudSearch allows running test queries to validate the migration and as well
the functionality of your application.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<u><span lang="EN-IN" style="font-size: large; line-height: 115%; mso-ansi-language: EN-IN; mso-bidi-font-size: 16.0pt;"><b>Features supported</b></span></u></div>
<div class="MsoListParagraphCxSpFirst" style="margin-left: .25in; mso-add-space: auto; mso-list: l3 level1 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">1.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Support
for other non-Linux environments is not available for now. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; mso-list: l3 level1 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">2.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Support
for Solr Shards is not available for now. The Solr shard needs to be migrated
separately.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; mso-list: l3 level1 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">3.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">The
install commands may vary for different Linux flavors. Example installing
software, file editor command, permission set commands can be different for
every Linux flavors. It is left to engineering team to choose the right
commands during the installation and execution of this migration tool. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; mso-list: l3 level1 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">4.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Only
fields configured as ‘stored’ in Solr schema.xml are supported. The non-stored
fields are ignored during schema parsing. <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; mso-list: l3 level1 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">5.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">The
document id (unique key) is required to have following attributes:<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .75in; mso-add-space: auto; mso-list: l3 level2 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">a.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Document
ID should be 128 characters or less in size.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .75in; mso-add-space: auto; mso-list: l3 level2 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">b.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">Document
ID can contain any letter, any number, and any of the following characters: <span style="background: red; mso-highlight: red;">_ - = # ; : / ? @ &</span> <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .75in; mso-add-space: auto; mso-list: l3 level2 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">c.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]-->The
below link will help you to understand in data
preparation before migrating to CloudSearch <a href="http://docs.aws.amazon.com/cloudsearch/latest/developerguide/preparing-data.html"><span lang="EN-IN">http://docs.aws.amazon.com/cloudsearch/latest/developerguide/preparing-data.html</span></a><span lang="EN-IN"> <o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; mso-list: l3 level1 lfo2; text-indent: -.25in;">
<!--[if !supportLists]--><span lang="EN-IN">6.<span style="font-size: 7pt; font-stretch: normal;"> </span></span><!--[endif]--><span lang="EN-IN">If the conditions
are not met in a document, it will be skipped during migration. Skipped records
are shown in the log file.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; mso-list: l3 level1 lfo2; text-align: justify; text-indent: -.25in;">
<!--[if !supportLists]-->7.<span style="font-size: 7pt; font-stretch: normal;"> </span><!--[endif]--><span lang="EN-IN">If a
field type (mapped to fields) is not stored, the stopwords mapped to that particular
field type are ignored. </span><o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; text-align: justify;">
<i><span lang="EN-IN">Example
1: <o:p></o:p></span></i></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; text-align: justify;">
<i><field
name="description" type="text_general"
indexed="true" stored="true" /> <o:p></o:p></i></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; text-align: justify;">
<i>The above field
‘description’ will be considered for stopwords.<o:p></o:p></i></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; text-align: justify;">
<i><span lang="EN-IN">Example
2:<o:p></o:p></span></i></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; text-align: justify;">
<i><span lang="EN-IN"><field
name="fileName" type="string" /> <o:p></o:p></span></i></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: .25in; mso-add-space: auto; text-align: justify;">
<i>The above field
‘fileName’ will not be migrated and ignored in the stopwords.<o:p></o:p></i></div>
<br />
<div class="MsoListParagraphCxSpLast" style="margin-left: .25in; mso-add-space: auto; text-align: justify;">
<br /></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-79506111068890320232015-06-12T23:35:00.002-07:002015-06-15T02:17:16.379-07:0027 Best Practice Tips on Amazon Web Services Security GroupsAWS Security Groups are one of the most used and abused configurations inside an AWS environment if you are using them on cloud quite long. Since AWS security groups are simple to configure, users many times ignore the importance of it and do not follow best practices relating to it. In reality, operating on AWS security groups every day is much more intensive and complex than configuring them once. Actually, nobody talks about it! So in this article, I am going to share our experience in dealing with AWS Security groups since 2008 as a set of best practice pointers relating to configuration and day to day operations perspective.<br />
In the world of security, proactive and reactive speed determines the winner. So a lot of these best practices should be automated in reality. In case your organizations' Dev/Ops/Devops teams needs help with security group best practices automation, feel free to contact me.<br />
<br />
AWS released so many features in the last few years relating to Security, that we should not visualize Security groups in isolation, It just does not make sense anymore. The Security Group should always be seen in the overall security context, with this I start the pointers.<br />
<br />
<b>Practice 1:</b> Enable AWS VPC Flow Logs for your VPC or Subnet or ENI level. AWS VPC flow logs can be configured to capture both accept and reject entries flowing through the ENI and Security groups of the EC2, ELB + some more services. This VPC Flow log entries can be scanned to detect attack patterns,alert abnormal activities and information flow inside the VPC and provide valuable insights to the SOC/MS team operations.<br />
<br />
<b>Practice 2:</b> Use AWS Identity and Access Management (IAM) to control who in your organization has permission to create and manage security groups and network ACLs (NACL). Isolate the responsibilities and roles for better defense. For example, you can give only your network administrators or security admin the permission to manage the security groups and restrict other roles.<br />
<div>
<br /></div>
<div>
<b>Practice 3:</b> Enable AWS Cloud Trail logs for your account. The AWS Cloud Trail will log all the security group events and it is needed for management and operations of security groups. Event streams can be created from AWS Cloud Trail logs and it can be processed using AWS Lambda. For example : whenever a Security Group is deleted , this event will be captured with details on the AWS Cloud Trail logs. Events can be triggered in AWS Lamdba which can process this SG change and alert the MS/SOC on the dashboard or email as per your workflow. This is a very powerful way of reacting to events within span of <7 minutes. Alternatively, you can process the AWS Cloud Trail logs stored in your S3 every X frequency as a batch and achieve the above. But the Operation teams reaction time can vary depending on generation and polling frequency of the AWS Cloud Trail logs. This activity is a must for your operations team.</div>
<div>
<br />
<b>Practice 4:</b> Enable AWS App Config for your AWS account. App records all events related to your security group changes and can even send emails.<br />
<br /></div>
<b>Practice 5:</b> Have proper naming conventions for the Amazon Web Services security group. The naming convention should follow a enterprise standards. For example it can follow the notation: "AWS Region+ Environment Code+ OS Type+Tier+Application Code"<br />
Security Group Name - EU-P-LWA001<br />
AWS Region ( 2 char ) = EU, VA, CA etc<br />
Environment Code (1 Char) = P-Production , Q-QA, T-testing, D-Development etc<br />
OS Type (1 Char)= L -Linux, W-Windows etc<br />
Tier (1 Char)= W-Web, A-App, C-Cache, D-DB etc<br />
Application Code ( 4 Chars) = A001<br />
We have been using Amazon Web Services from 2008 and found over the years managing the security groups in multiple environments is itself a huge task. Proper naming conventions from beginning is a simple practice, but will make your AWS journey manageable.<br />
<br />
<b>Practice 6:</b> For security in depth, make sure your Amazon Web Services security groups naming convention is not self explanatory also make sure your naming standards stays internal. Example : AWS security group named UbuntuWebCRMProd is self explanatory for hackers that it is a Production CRM web tier running on ubuntu OS. Have an automated program detecting AWS security groups with Regex Pattern scanning of AWS SG assets periodically for information revealing names and alert the SOC/Managed service teams.<br />
<div>
<br /></div>
<b>Practice 7:</b> Periodically detect, alert or delete AWS Security groups not following the organization naming standards strictly. Also have an automated program doing this as part of your SOC/Managed service operations. Once you have this stricter control implemented then things will fall in line automatically.<br />
<br />
<b>Practice 8:</b> Have automation in place to detect all EC2,ELB and other AWS assets associated with Security groups. This automation will help us to periodically detect Amazon Web Services Security groups lying idle with no associations, alert the MS team and cleanse them. Unwanted security groups accumulated over time will create unwanted confusion.<br />
<br />
<b>Practice 9:</b> In your AWS account, when you create a VPC, AWS automatically creates a default security group for the VPC. If you don't specify a different security group when you launch an instance, the instance is automatically associated with the appropriate default security group. It will<br />
allow inbound traffic only from other instances associated with the "default" security group and allow all outbound traffic from the instance. The default security group specifies itself as a source security group in its inbound rules. This is what allows instances associated with the default security group to communicate with other instances associated with the default security group. This is not a good security practice. If you don't want all your instances to use the default security group, you can create your own security groups and specify them when you launch your instances. This is applicable to EC2 , RDS , ElastiCache and some more services in AWS. So detect "default" security groups periodically and alert to the SOC/MS.<br />
<br />
<b>Practice 10:</b> Alerts by email and cloud management dash board should be triggered whenever critical security groups or rules are added/modified/deleted in production. This is important for reactive action of your managed services/security operations team and audit purpose.<br />
<br />
<b>Practice 11 :</b> When you associate multiple security groups with an Amazon EC2 instance, the rules from each security group are effectively aggregated to create one set of rules. AWS uses this set of rules to determine whether to allow access or not. If there is more than one SG rule for a specific port, AWS applies the most permissive rule. For example, if you have a rule that allows access to TCP port 22 (SSH) from IP address 203.0.113.10 and another rule that allows access to TCP port 22 for everyone, then everyone will have access to TCP port 22 because permissive takes precedence.<br />
Practice X.1 : Have automated programs detecting EC2 associated with multiple SG/rules and alert the SOC/MS periodically. Condense the same manually to 1-3 rules max as part of your operations.<br />
<div>
Practice X.1 : Have automated programs detecting conflicting SG/rules like restrictive+permissive rules together and alert the SOC/MS periodically.</div>
<div>
<br /></div>
<b>Practice 12 :</b> Do not create least restrictive security groups like 0.0.0.0/0 which is open to every one.<br />
Since web servers can receive HTTP and HTTPS traffic open, only their SG can be permissive like<br />
0.0.0.0/0,TCP, 80, Allow inbound HTTP access from anywhere<br />
0.0.0.0/0,TCP, 443, Allow inbound HTTPS access from anywhere<br />
All least restrictive SG created in your account should be alerted to SOC/MS teams immediately.<br />
<br />
<b>Practice 13:</b> Have a security policy not to launch servers with default ports like 3306, 1630, 1433, 11211, 6379 etc. If the policy has to be accepted, then security groups also have to be created on the new hidden listening ports instead of the default ports. This provides a small layer of defense, since one cannot infer the information from the security group port on the EC2 service it is protecting. Automated detection and alerts should be created for SOC/MS, if security groups are created with default ports.<br />
<br />
<b>Practice 14:</b> Applications which require stricter compliance requirements like HIPAA, PCI etc to be met need end to end transport encryption to be implemented on server back end in AWS. The communication from ELB to Web->App->DB->Others tiers need to be encrypted using SSL or HTTPS. This means only secured ports like 443, 465, 22 are permitted in corresponding EC2 security groups. Automated detection and alerts should be created for SOC/MS if security groups are created on secure ports for regulated applications.<br />
<br />
<b>Practice 15:</b> Detection , alert and actions can be taken by parsing the AWS Cloud Trail logs based on usual patterns observed in your production environment<br />
Example:<br />
<b>15.1 :</b>If a port was opened and closed in <30 or X mins in production can be a candidate for suspicious activity if it is not normal pattern for your production<br />
<b>15.2 :</b>If a permissive Security Group was created and closed in <30 or X mins can be a candidate for suspicious activity if it is not the normal pattern for your production<br />
Detect anomalies on how long a change effected and reverted in security groups in production.<br />
<br />
<b>Practice 16:</b> In case ports have to be opened in Amazon Web Services security groups or a permissive AWS security group needs to be applied, Automate this entire process as part of your operations such that a security group is open for X agreed minutes and will be automatically closed aligning with your change management. Reducing manual intervention avoids operational errors and adds security.<br />
<br />
<b>Practice 17:</b> Make sure SSH/RDP connection is open in AWS Security Group only for jump box/bastion hosts for your VPC/subnets. Have stricter controls/policies avoid opening SSH/RDP to other instances of production environment. Periodically check , alert and close for this loop hole as part of your operations.<br />
<br />
<b>Practice 18:</b> It is a bad practice to have SSH open to the entire Internet for emergency or remote support. By allowing the entire Internet access to your SSH port there is nothing stopping an attacker from exploiting your EC2 instance. The best practice is to allow very specific IP address in your security groups, this restriction improves the protection. This could be your office or on premise or DC through which you connect your jump box.<br />
<br />
<b>Practice 19:</b> Too much or Too less: How many security groups for a usual multi tiered web app is preferred is a frequently asked question ?<br />
Option 1 : One security group cutting across multiple tiers is easy to configure, but it is not a recommended for secure production applications.<br />
Option 2: One Security group for every instance is too much protection and tough to manage operationally on longer term<br />
Option 3: Individual Security group for different tiers of the application, For example : Have separate security groups for ELB, Web , App, DB and Cache tiers of your application stack.<br />
Periodically check whether Option 1 type rule is being created in your production and alert the SOC/MS.<br />
<br />
<b>Practice 20:</b> Avoid allowing UDP or ICMP for private instances in Security groups. Not a good practice unless specifically needed.<br />
<br />
<b>Practice 21:</b> Open only specific ports, Opening range of
ports in a security group is not a good practice. In
the security group you can add many inbound ingress rules, While opening the
ports it is always advised to open for specific ports like 80,443, etc rather
than range of ports like 200-300.<br />
<div style="margin: 0in 0in 0.0001pt;">
<o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<br />
<div style="margin: 0in 0in 0.0001pt;">
<img alt="Add rules for communication between associated instances" src="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/add_rule_sg_source_gwt.png" id="_x0000_i1025" /><o:p></o:p></div>
<br />
<b>Practice 22: </b>Private Subnet instances can be
accessed only from the VPC CIDR IP range. Opening instances to the public IP
ranges is a possibility , but it does not make any sense. E.g., Opening HTTP to <a href="http://0.0.0.0/0">0.0.0.0/0</a>
in the SG of the private subnet instance does not make any sense. So detect and cleanse such rules.<br />
<div style="margin-bottom: .0001pt; margin: 0in;">
<o:p></o:p></div>
<br />
<div>
<table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse; width: 578px;">
<colgroup><col style="mso-width-alt: 21138; mso-width-source: userset; width: 434pt;" width="578"></col>
</colgroup><tbody>
<tr height="20" style="height: 15.0pt;">
<td class="xl63" height="20" style="height: 15.0pt; width: 434pt;" width="578"><b>Practice 23: </b>AWS CloudTrail log captures the events related security. AWS lambda events or automated programs should trigger alerts to operations when abnormal activities are detected. For example:<br />
<b>23.1:</b>Alert
when X number of SG were added/deleted at "Y" Hours or Day by IAM user / Account</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td class="xl63" height="20" style="height: 15.0pt; width: 434pt;" width="578"><b>23.2:</b>Alert
when X number of SG Rules were added/deleted at "Y" Hours or Day by IAM user / Account<br />
<br /></td>
</tr>
</tbody></table>
</div>
<b>Practice 24:</b> In case you are an enterprise make sure all security groups related activities of your production are part of your change management process. Security Group actions can be manual or automated with your change management in an enterprise.<br />
In case you are an agile Startup or SMB and do not have complicated Change management process, then automate most of the security group related tasks and events as illustrated above on various best practices. This will bring immense efficiency into your operations<br />
<br />
<b>Practice 25:</b> Use outbound/egress security groups wherever applicable within your VPC. Restrict FTP connection to any server on the Internet from your VPC. This way you can avoid data dumps and important files getting transferred out from your VPC. Defend harder and make it tougher !<br />
<br />
<b>Practice 26:</b> For some tiers of your application, use ELB in front your instance as a security proxy with restrictive security groups - restrictive ports and IP ranges. This doubles your defense but increases the latency.<br />
<br />
<b>Practice 27:</b> Some of the tools we use in conjunction to automate and meet above best practices are ServiceNow, Amazon CFT, AWS API'S, Rundeck, Puppet, Chef, Python , .Net and Java automated programs.<br />
<br />
<b><span style="font-size: large;">Note : In case your organizations Dev/Ops/Devops teams needs help on security group best practices automation on points listed above, feel free to contact me <u><a href="mailto:harish11g.aws@gmail.com">harish11g.aws@gmail.com</a></u></span></b>Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-90794385826547923462014-08-27T22:03:00.003-07:002014-08-27T23:05:24.541-07:00Billion Messages - Art of Architecting scalable ElastiCache Redis tier<br />
Whenever we are designing a highly scalable architectures on AWS running thousands of application servers and supporting millions of requests, usage of NoSQL solutions have become inevitable part. One such solution we often been using for years on AWS is Redis . <b>We love Redis. </b><br />
AWS introduced ElastiCache Redis on 2013 and we started using the same since it eased the management and operational efforts. In this article i am going to share my experience on designing large scale Redis tiers supporting billions of messages per day on AWS, step by step guide on how to deploy the same, what are the Implications you face at scale ? Best Practices to be adopted while designing sharded+replicated Redis Tiers etc.<br />
<br />
Since we need to support billions of message requests per day and it was growing:<br />
<br />
<ul>
<li>the ElastiCache Redis tier was designed with Partitions( shards) to scale out as the customer grows</li>
<li>the ElastiCache Redis tier was designed with Replica Slaves for HA and read scaling as the read volumes grow</li>
</ul>
<br />
When your application is growing at Rapid pace and lots of data are created every day, you cannot keep increasing (scaling up) the size of the ElastiCache Node. At one point you will hit the maximum memory capacity of your EC2 instance and you will be forced to partition. Partitioning is the process of splitting your Key Value data into multiple ElastiCache Redis instances, so that every instance will only contain a subset of your Key Value pair. It allows for much larger ElastiCache Redis data stores, using the sum of the memory of many ElastiCache Redis Nodes. It also allows to scale the computational power to multiple cores and multiple EC2, and the network bandwidth to multiple EC2 network adapters. There are two widely used partition/shard implementation techniques that are available for ElastiCache Redis Tier :<br />
<b>Technique 1)</b> Client side partitioning means that the Redis clients directly select the right ElastiCache Redis node where to write or read a given key. Many Redis clients implement client side partitioning, chose the right one wisely.<br />
<b>Technique 2)</b> Proxy assisted partitioning means that your clients send requests to a proxy that is able to speak the Redis protocol, which in turn sends requests directly to the right ElastiCache Redis instance. The proxy will make sure to forward our request to the right Redis instance accordingly to the configured partitioning schema. Currently the most widely used Proxy assisted partitioning tool is Twemproxy , written by Manju Raj of twitter. Git hub link <a href="https://github.com/twitter/twemproxy">https://github.com/twitter/twemproxy</a><u> . </u>Twemproxy is a proxy developed at Twitter for the Memcached ASCII and the Redis protocol. Twemproxy supports automatic partitioning among multiple Redis instances and currently it is the suggested way to handle partitioning with Redis.<br />
<div>
In this article we are going to explore in detail about Proxy assisted partitioning technique for highly scalable and available Redis tier.</div>
<br />
<b><span style="font-size: large;">Welcome to Twemproxy</span></b><br />
Twemproxy( nutcracker) is a fast single-threaded proxy supporting the Memcached ASCII protocol and more recently the Redis protocol.<br />
<br />
<b>Installing Twemproxy:</b><br />
Download the Twemproxy package.<br />
wget http://twemproxy.googlecode.com/files/nutcracker-0.3.0.tar.gz<br />
tar -xf nutcracker-0.3.0.tar.gz<br />
cd nutcracker-0.3.0<br />
./configure<br />
make<br />
make install<br />
<br />
<b>Configuration:</b><br />
Twemproxy (Nutcracker) can be configured through a YAML file specified by the -c or --conf-file command-line argument on process start. The configuration file is used to specify the server pools and the servers within each pool that nutcracker manages. The configuration files parses and understands the following keys:<br />
<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>listen: The listening address and port (name:port or ip:port) for this server pool.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>hash: The name of the hash function.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>hash_tag: A two character string that specifies the part of the key used for hashing. Eg "{}" or "$$". Hash tag enable mapping different keys to the same server as long as the part of the key within the tag is the same.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>distribution: The key distribution mode.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>timeout: The timeout value in msec that we wait for to establish a connection to the server or receive a response from a server. By default, we wait indefinitely.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>backlog: The TCP backlog argument. Defaults to 512.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>preconnect: A boolean value that controls if nutcracker should preconnect to all the servers in this pool on process start. Defaults to false.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>redis: A boolean value that controls if a server pool speaks redis or memcached protocol. Defaults to false.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>server_connections: The maximum number of connections that can be opened to each server. By default, we open at most 1 server connection.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>auto_eject_hosts: A boolean value that controls if server should be ejected temporarily when it fails consecutively server_failure_limit times. See liveness recommendations for information. Defaults to false.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>server_retry_timeout: The timeout value in msec to wait for before retrying on a temporarily ejected server, when auto_eject_host is set to true. Defaults to 30000 msec.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>server_failure_limit: The number of consecutive failures on a server that would lead to it being temporarily ejected when auto_eject_host is set to true. Defaults to 2.<br />
•<span class="Apple-tab-span" style="white-space: pre;"> </span>servers: A list of server address, port and weight (name:port:weight or ip:port:weight) for this server pool.<br />
<br />
For More details Refer: <a href="https://github.com/twitter/twemproxy">https://github.com/twitter/twemproxy</a><br />
<br />
<b>Running and Accessing Twemproxy </b><br />
To start the proxy just use the command “nutcracker” with the configuration file path specified or in its default path(conf/nutcracker.yml) .<br />
Based on the configuration the twemproxy will be running and listening. Configure your application to point to the port and address instead of the Redis cluster.<br />
<br />
<b><span style="font-size: large;">Twemproxy Deployment models:</span></b><br />
<br />
We usually deploy Twemproxy one one of the following models in AWS :<br />
<b>Model 1: Twemproxy as a separate Proxy Tier: </b>In this model Twemproxies are deployed in separate EC2 instances, The application tier is configured to point to Twemproxies . The Twemproxy tier in turn maintains the mappings to the ElastiCache redis nodes. It is better to use instances with very good IO bandwidth for twemproxy tier in AWS. In case you feel the instance CPU is underutilized, you can launch multiple Twemproxy instances inside the same single EC2 instance as well.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimjCjPyKa8z8iebmQ_P7eQkW9FjDq4gaFPLA85laKYF2oByHjE2iXHHGcTcDTQ6z_-ccA_yIZUusTs0CxDOrrq-gNJxeXML9UcHCIXteI4Xsg0byNY05XL2sE_-Pc5qlCDbEZ7oUKcdR5Z/s1600/ElastiCache+Redis+Proxy+Twemproxy+deployment+model+1+on+AWS.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimjCjPyKa8z8iebmQ_P7eQkW9FjDq4gaFPLA85laKYF2oByHjE2iXHHGcTcDTQ6z_-ccA_yIZUusTs0CxDOrrq-gNJxeXML9UcHCIXteI4Xsg0byNY05XL2sE_-Pc5qlCDbEZ7oUKcdR5Z/s1600/ElastiCache+Redis+Proxy+Twemproxy+deployment+model+1+on+AWS.jpg" height="470" width="640" /></a></div>
<br />
Though the above model looks clean and efficient there are optimizations that can be applied to this architecture :<br />
What happens when the twemproxy01 fails, how will the Application server instances know about it ?<br />
Why should i pay additional for twemproxy EC2 instances, Can it be minimized ?<br />
<br />
<b>Model 2 : Twemproxy bundled with application tier EC2's: </b>In this model twemproxies are bundled in the same box of the application server EC2 itself. Since two twemproxies are not aware of each others existence, it is easy to architect this model even in App->Auto Scaling mode. Every application server talks to the local twemproxy deployed in the same box this saves cost and avoids managing additional tier complexity as well.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIWhgL0J-uTa1eu3GUkc7oNNPgSPcxuvCpB81VHtphu2ivOwvFLDFisPvcYDE3fjZfcVOC6Q_lNjkc6NfGfJK6_LdYzZ1ONy8VDCL0i53riKP3jh3K6C9DRa8I1E6L7k62EDVOBs_ESTpU/s1600/ElastiCache+Redis+Proxy+Twemproxy+deployment+model+2+on+AWS.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIWhgL0J-uTa1eu3GUkc7oNNPgSPcxuvCpB81VHtphu2ivOwvFLDFisPvcYDE3fjZfcVOC6Q_lNjkc6NfGfJK6_LdYzZ1ONy8VDCL0i53riKP3jh3K6C9DRa8I1E6L7k62EDVOBs_ESTpU/s1600/ElastiCache+Redis+Proxy+Twemproxy+deployment+model+2+on+AWS.jpg" height="474" width="640" /></a></div>
<div>
<br /></div>
<span style="font-size: large;"><b>Reference </b><b>ElastiCache Redis + Twemproxy deployment:</b></span><br />
(This is a Reference deployment, the same can be scaled out to hundreds depending upon the need. It is a Redis Partitioned + replicated setup )<br />
1. Two ElastiCache Redis nodes in AWS (twem01 and twem02)<br />
2. Replication group for each ElastiCache redis nodes (twem01-rg and twem02-rg with one Read Replica each)<br />
3. Two twemproxy servers running in separate EC2. (twemproxy01 and twemproxy02)<br />
Once the above setup is done please note down the endpoints. We will be using the Replication group endpoint as the ElastiCache Redis endpoint for the twemproxy.<br />
<b><br /></b>
<b>ElastiCache Redis Endpoints:</b><br />
twem01-twem01.qcdze2.0001.usw2.cache.amazonaws.com:6379<br />
twem02-twem02.qcdze2.0001.usw2.cache.amazonaws.com:6379<br />
<b>ElastiCache Redis Replication endpoints:</b><br />
twem01-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379<br />
twem02-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMP15sE0SyorkVAJBZtIRslf7TQxW1LBEKbq1ZU0vn57UBVw1sdFCzWSwUk9Z4zo-Ia2ltVglyBRfRjzrOZy8h7lMawt7IEtux4shvbAw6ChmfZ-z1rcBCe_01et-500cDZKAmamNhKhUh/s1600/image+2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMP15sE0SyorkVAJBZtIRslf7TQxW1LBEKbq1ZU0vn57UBVw1sdFCzWSwUk9Z4zo-Ia2ltVglyBRfRjzrOZy8h7lMawt7IEtux4shvbAw6ChmfZ-z1rcBCe_01et-500cDZKAmamNhKhUh/s1600/image+2.jpg" height="238" width="640" /></a></div>
<br />
<br />
To test the Twemproxy we pumped following keys:<br />
Pump KV data through the Twemproxy01 (1-2000 keys)<br />
Pump KV data through the Twemproxy02(2001-4000 keys).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXQKHgljQR5HSC2wH8ugAj7r3gaGmewQA5oUPvznzkRV9Z9h-iPUctMpjGAlyC5b9IHXKYLNsdVnim5gBlqZxryrS8BViLVQu_vt4e0BpZFHsU3bN76y5EdGwF-9lAxHcH4H4GnWeH9eB6/s1600/image+3.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXQKHgljQR5HSC2wH8ugAj7r3gaGmewQA5oUPvznzkRV9Z9h-iPUctMpjGAlyC5b9IHXKYLNsdVnim5gBlqZxryrS8BViLVQu_vt4e0BpZFHsU3bN76y5EdGwF-9lAxHcH4H4GnWeH9eB6/s1600/image+3.jpg" height="350" width="640" /></a></div>
<br />
Configuration:<br />
beta:<br />
listen: 127.0.0.1:22122<br />
hash: fnv1a_64<br />
hash_tag: "{}"<br />
distribution: ketama #Consistent Hashing<br />
auto_eject_hosts: false<br />
timeout: 5000<br />
redis: true<br />
servers:<br />
- twem01-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379:1 server1<br />
- twem02-rg.qcdze2.ng.0001.usw2.cache.amazonaws.com:6379:1 server2<br />
<br />
<b>Test 1: Testing Key accessibility . </b><b>Testing "GET" operation across both the Twemproxy Instances for few sample keys. </b><br />
<b><br /></b>
<b>Fetch 4 Keys spread across 4000 KV data from Twemproxy01 EC2 instance:</b><br />
[root@twemproxy01 redish]# src/redis-cli -h 127.0.0.1 -p 22122<br />
redis 127.0.0.1:22122> get 1000<br />
"1000-data"<br />
redis 127.0.0.1:22122> get 2000<br />
"2000-data"<br />
redis 127.0.0.1:22122> get 3000<br />
"3000-data"<br />
redis 127.0.0.1:22122> get 4000<br />
"4000-data"<br />
<b>Fetch 4 Keys spread across 4000 KV data from Twemproxy02 EC2 instance:</b><br />
[root@twemproxy02 redish]# src/redis-cli -h 127.0.0.1 -p 22122<br />
redis 127.0.0.1:22122> get 1000<br />
"1000-data"<br />
redis 127.0.0.1:22122> get 2000<br />
"2000-data"<br />
redis 127.0.0.1:22122> get 3000<br />
"3000-data"<br />
redis 127.0.0.1:22122> get 4000<br />
"4000-data"<br />
<br />
From the above test it is evident that all 4000 KV data inserted using both Twemproxies are accessible from both Twemproxies( testing the sample) even though they are not aware among themselves. This is because of the same hashing and Key mapping translation done at Twemproxy level.<br />
<br />
<b>Test 2: Testing the ElastiCache Redis Availability and Fail over mechanism:</b><br />
<div>
<b><br /></b></div>
We are going to promote the twem01-rg replication group read replica to be the Primary Redis Node. After promotion we are going to test:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6osTLpNkobmL6XWEceHy4oiW3fem_rGV8iVUvtclIqdmzs2XzHhkIae89M5RiZ_IIot8SVvPkCfrFzd0QRDriRjjHoYmrZQ3CKypMzK7ZO_ZDYnV9rXk2joA05zjHYgY_C9zQpddl-MCO/s1600/image+7.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6osTLpNkobmL6XWEceHy4oiW3fem_rGV8iVUvtclIqdmzs2XzHhkIae89M5RiZ_IIot8SVvPkCfrFzd0QRDriRjjHoYmrZQ3CKypMzK7ZO_ZDYnV9rXk2joA05zjHYgY_C9zQpddl-MCO/s1600/image+7.jpg" height="166" width="640" /></a></div>
<br />
<ol>
<li>Whether the Twemproxy is able to recognize the newly promoted master</li>
<li>Whether the sample KV data is safely replicated and still accessible , to ensure failover is successful.</li>
</ol>
<br />
To promote ElastiCache Redis slave just click the promote Action and confirm or automate using API. During the promotion of Read Replica to master we observed that the transition happens very quickly and there is no timeout but the response time for the query is about 4-5 secs for about 3-4 minutes during the switch over. In the Twemproxy configuration we can set the timeout configuration, this value needs to be set accordingly so that during switch over there will be no connection refused. For the sample test we have set it as 5000<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_4f1TTE1GIco9ldnRA-dITIbR5axzJBU5YoyLkn2oCPlsabT1YkUTrT8XAO0bPekJoEVFPZYIowlINDy3SQaRT3qgxxrUmttpbYqrOoLKnopuA6zvKCIvKT5LJk0slPUrDwoN2oRmzXzW/s1600/image+8.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_4f1TTE1GIco9ldnRA-dITIbR5axzJBU5YoyLkn2oCPlsabT1YkUTrT8XAO0bPekJoEVFPZYIowlINDy3SQaRT3qgxxrUmttpbYqrOoLKnopuA6zvKCIvKT5LJk0slPUrDwoN2oRmzXzW/s1600/image+8.jpg" height="198" width="400" /></a></div>
<br />
<div>
<b><br /></b>
<b><br /></b></div>
<div>
<b>Repeat Test 1:</b></div>
[root@twemproxy01 redish]# src/redis-cli -h 127.0.0.1 -p 22122<br />
redis 127.0.0.1:22122> get 1000<br />
"1000-data"<br />
redis 127.0.0.1:22122> get 2000<br />
"2000-data"<br />
redis 127.0.0.1:22122> get 3000<br />
"3000-data"<br />
redis 127.0.0.1:22122> get 4000<br />
"4000-data"<br />
<b>Fetch 4 Keys spread across 4000 KV data from Twemproxy02 EC2 instance:</b><br />
[root@twemproxy02 redish]# src/redis-cli -h 127.0.0.1 -p 22122<br />
redis 127.0.0.1:22122> get 1000<br />
"1000-data"<br />
redis 127.0.0.1:22122> get 2000<br />
"2000-data"<br />
redis 127.0.0.1:22122> get 3000<br />
"3000-data"<br />
redis 127.0.0.1:22122> get 4000<br />
"4000-data"<br />
<div>
From the above test it is evident that all 4000 KV data are replicated properly between master and slaves nodes and the transition between slave to master happened successfully with all the data.</div>
<div>
<b>Reporting</b></div>
Nutcracker exposes stats at the granularity of server pool and servers per pool through the stats monitoring port. The stats are essentially JSON formatted key-value pairs, with the keys corresponding to counter names. By default stats are exposed on port 22222 and aggregated every 30 seconds.<br />
<br />
<b><span style="font-size: large;">Some best practices while designing highly scalable+available ElastiCache Redis Tier :</span></b><br />
<br />
<b>Practice 1 : Reduce the Number of Connections and pipeline messages:</b><br />
Whenever the application instance gets a request to get/put value to the ElastiCache redis node, the client makes a connection to the Redis Tier. Imagine it is a heavy traffic site, then thousands of requests hitting translates to thousands of connections from the application instance to Redis Tier. Now when you add Auto- scaling to your application tier and you have few hundred servers scaled out , then imagine the connection complexity and overhead this architecture brings to the ElastiCache Redis Tier.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1fVhoaHQKkWubWX69KVs5hccSAcIh9MXpB3gb4WX_4DOXqo0BThIWRqRsQYwDJ1hyphenhyphenbZbJVDrE7w3kx39izZbq3msHOBafy7MyQHdFtwqp1IQ-cDpr68YlcjJV7maQwgTg-bpJibRZgXCN/s1600/Minimize+Connections+to+Elasticache+redis+tier+-+Best+practice.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1fVhoaHQKkWubWX69KVs5hccSAcIh9MXpB3gb4WX_4DOXqo0BThIWRqRsQYwDJ1hyphenhyphenbZbJVDrE7w3kx39izZbq3msHOBafy7MyQHdFtwqp1IQ-cDpr68YlcjJV7maQwgTg-bpJibRZgXCN/s1600/Minimize+Connections+to+Elasticache+redis+tier+-+Best+practice.jpg" height="464" width="640" /></a></div>
Best practice is minimize the number of connections made from your application instance to your ElastiCache redis node. Use Twemproxy in bundled mode with Application EC2 instance, this keeps the process in close proximity and reduces the connection overhead. Secondly, Twemproxy internally uses minimal connections to ElastiCache Redis Instance by proxying multiple client connections onto one or few server connections.<br />
Redis also supports pipelines, where multiple requests can be pipelined and sent on a single connection. In a simple test using large Application & ElastiCache node we were able to process 125K message/sec in pipeline mode, now imagine what you could achieve on bigger instance types on AWS. The connection minimization architectural setup of twemproxy makes it ideal for pipelining requests and responses and hence saving on the round trip time. For example, if twemproxy is proxying three client connections onto a single server and we get requests - 'get key\r\n', 'set key 0 0 3\r\nval\r\n' and 'delete key\r\n' on these three connections respectively, twemproxy would try to batch these requests and send them as a single message onto the server connection.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifb5C70meVTKqJHBZlXPBjTAum9wt1jfJn7oyEHkQe7G5kOeiHKJT97V3R1NEGGrxAVlW62KY5Oaky8BcJ050WFJCdDZFBaE7EWnEL9zck0s_J2rgIJXidrPV9wIE_4eq9Q-a3XpkIxkQS/s1600/Minimize+Connections+to+Elasticache+redis+tier+-+Best+practice+solution.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifb5C70meVTKqJHBZlXPBjTAum9wt1jfJn7oyEHkQe7G5kOeiHKJT97V3R1NEGGrxAVlW62KY5Oaky8BcJ050WFJCdDZFBaE7EWnEL9zck0s_J2rgIJXidrPV9wIE_4eq9Q-a3XpkIxkQS/s1600/Minimize+Connections+to+Elasticache+redis+tier+-+Best+practice+solution.jpg" height="462" width="640" /></a></div>
<br />
Note : It is important to note that "read my last write" constraint doesn't necessarily hold true when twemproxy is configured with <b>server_connections: > 1</b>. Let us consider a scenario where twemproxy is configured with server_connections: 2. If a client makes pipelined requests with the first request in pipeline being set foo 0 0 3\r\nbar\r\n (write) and the second request being get foo\r\n (read), the expectation is that the read of key foo would return the value bar. However, with configuration of two server connections it is possible that write and read request are sent on different server connections which would mean that their completion could race with one another. In summary, if the client expects "read my last write" constraint, you either configure twemproxy to use server_connections:1 or use clients that only make synchronous requests to twemproxy.<br />
<div>
<br /></div>
<div>
<b>Practice 2: Configure Auto Ejection and Hashing combination properly</b><br />
Design for failure is the mantra of cloud architecture. Failures are commons when things are distributed on scale. Though partitioning when using ElastiCache Redis as a data store or cache is conceptually the same on broad lines, there is a huge difference operationally on large scale systems. When you are using ElastiCache Redis as a data store you need to be sure that a given key always maps to the same instance, Whereas if you are using ElastiCache Redis as cache if a given node is not available, then you can always start afresh using a different node in the hash ring with consistent hashing implementations.<br />
To be resilient against failures, it is recommended that you configure Auto eject hosts false when you treat redis as a Data Store and true in when you treat redis as a cache.<br />
resilient_pool:<br />
auto_eject_hosts: true<br />
server_retry_timeout: 30000<br />
server_failure_limit: 3<br />
Enabling auto_eject_hosts: This property ensures that a dead ElastiCache redis Node can be ejected out of the hash ring after server_failure_limit: consecutive failures have been encountered on that node. A non-zero server_retry_timeout: ensures that we don't incorrectly mark a node as dead forever especially when the failures were really transient. The combination of server_retry_timeout: and server_failure_limit: controls the tradeoff between resiliency to permanent and transient failures.<br />
Note that an ejected node will not be included in the hash ring for any requests until the retry timeout passes. This will lead to data partitioning as keys originally on the ejected node will now be written to another node still in the pool. If <b>ElastiCache Redis is used as a cache</b> (in memory) then in event of a Redis Node going down, the cache data will be lost. This cache miss can cascade performance problems to other tiers and altogether bring down your system on the cloud. To minimize KV cache miss, you can design your hash ring with Ketama hashing on the Redis Proxy. This will minimize the Cache miss in event of cache node failure, also it decreases the overall re-balancing needed in your Redis tier. In addition to helping hand on availability problems, Redis Proxy+Ketama can also help your Redis farm to Scale out and Scale down easily with minimal cache miss. To know more about Ketama on ElastiCache refer <a href="http://harish11g.blogspot.in/2013/01/amazon-elasticache-memcached-internals_8.html">http://harish11g.blogspot.com/2013/01/amazon-elasticache-memcached-internals_8.html </a> . The below diagram illustrates a ElastiCache Redis Cache Farm with Consistent Hash Ring.<br />
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<b>In short to minimize the cache miss when using auto eject with true it is recommended to use "Ketama Hashing ( Consistent Hashing Algorithm)" on your Twemproxy configuration. </b></div>
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6yGX_0tLJAOgLec6JbZH7bUWvgWiTuQ16pnNgqu-G3ZqSZBhIXgVR4UUHky0Jb4ITcs9a8LSWsQAB2CgzawAfrriUG9laba5UemsQL9qGBSgSXgDhyphenhyphen4yT2X71e1Oz9rZEzrO8R54AoVZl/s1600/ElastiCache+redis+as+a+Cache+with+Ketama+Consistent+hashing.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6yGX_0tLJAOgLec6JbZH7bUWvgWiTuQ16pnNgqu-G3ZqSZBhIXgVR4UUHky0Jb4ITcs9a8LSWsQAB2CgzawAfrriUG9laba5UemsQL9qGBSgSXgDhyphenhyphen4yT2X71e1Oz9rZEzrO8R54AoVZl/s1600/ElastiCache+redis+as+a+Cache+with+Ketama+Consistent+hashing.jpg" height="470" width="640" /></a></div>
<br /></div>
<div>
<b>ElastiCache Redis as a Data Store:</b><br />
What if the data stored in your Cache is important and needs to persisted across node failures and launch ? What if the date stored in your Cache cannot be lost and it needs to be replicated and promoted during failures?<br />
Welcome to ElastiCache Redis as Data store. ElastiCache Redis offers features to persist the in memory cache data to disk and also replicate it to a slave for high availability. If ElastiCache Redis is used as a store (persistent), you need to keep the map between keys and nodes fixed, and a fixed number of nodes. Since the data stored is important when you treat ElastiCache Redis as a data store, in event one Redis node goes down, you should have immediate standby up and running in minutes. You can architect ElastiCache Redis master with one or more replication Slave launched on different AZ from Master for High Availability in AWS. In event master node failure or master AZ failure, the slave Redis node can be promoted in minutes to act as master. This whole High availability design keeps the number of nodes on the hash ring stable and simple, Otherwise, you will end up building a system to re balance the keys (which is not easy) between nodes whenever there is a addition or removal of nodes during outages. In addition to above the ElastiCache Redis supports Partial Resynchronization with Slaves - If the connection between a master node and a slave node is momentarily broken, the master now accumulates data that is destined for the slave in a backlog buffer. If the connection is restored before the buffer becomes full, a quick partial resync will be done instead of a potentially longer full resync. This really saves network bottleneck during momentary failures.<br />
In large scale systems you will often find some partitions are heavily used than others , in event the usage is read heavy in nature you can add upto 5 Read replicas for the ElastiCache Redis Master partition. Since these replicas are used only for read they do not affect the Hash ring structure. But Twemproxy lacks the support for read scaling with Redis Replica's. So in event when you face this problem, you will have to Scale up the capacity(instance/node type) of the Master and Slave of that partition alone.<br />
<div>
If you are using <b>ElastiCache redis as a Data store</b> in the TwemProxy it is recommended to keep "auto_eject_hosts" property false so that in event of redis node failure it is not ejected from the hash ring. The hash ring can be built with both ketama or modula hash algorithms , since in event of Primary node failure, the Slave is going to be promoted and ring structure is going to be always maintained. But if you feel there is immense possibility for the number of primary node partitions to grow, or major failures to occu, it is better to choose ketama hash ring itself from beginning. The below diagram illustrates the architecture.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOfwof1SGd03Rt4a3ggtS69a0s92O3t4Pk0GgGcdIpkZ8eLbX_yqAS0Zyn6khdH3wLZ3ZiNMLh5B_h78POtxoX4x_bowh5rpu9UsSiwRPznDM66ss06IWg47_zv0heBm-zHbjEqub2yTEl/s1600/ElastiCache+redis+as+a+Data+store+with+Ketama+Consistent+hashing.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOfwof1SGd03Rt4a3ggtS69a0s92O3t4Pk0GgGcdIpkZ8eLbX_yqAS0Zyn6khdH3wLZ3ZiNMLh5B_h78POtxoX4x_bowh5rpu9UsSiwRPznDM66ss06IWg47_zv0heBm-zHbjEqub2yTEl/s1600/ElastiCache+redis+as+a+Data+store+with+Ketama+Consistent+hashing.jpg" height="470" width="640" /></a></div>
</div>
</div>
<div>
<br /></div>
<b>Practice 3: Configure the Buffer properly:</b><br />
All memory for incoming requests and outgoing responses is allocated in mbuf in Twemproxy. Mbuf enables zero copy for requests and responses flowing through the proxy. By default an mbuf is 16K bytes in size and this value can be tuned between 512 and 16M bytes using -m or --mbuf-size=N argument. Every connection has at least one mbuf allocated to it. This means that the number of concurrent connections twemproxy can support is dependent on the mbuf size. A small mbuf allows us to handle more connections, while a large mbuf allows us to read and write more data to and from kernel socket buffers. Large Scale web/mobile applications involving millions of hits might have small size request/response and lots of concurrent connections to handle in their backend. So at such scenarios, when Twemproxy is meant to handle a large number of concurrent client connections, you should set chunk size to a small value like 512 bytes to 1K bytes using the -m or --mbuf-size=N argument.<br />
<div>
<br /></div>
<b>Practice 4: Configure proper Timeouts</b><br />
It is always a good idea to configure Twemproxy timeout: for every server pool, rather than purely relying on client-side timeouts. Eg:<br />
<br />
resilient_pool_with_timeout:<br />
auto_eject_hosts: true<br />
server_retry_timeout: 30000<br />
server_failure_limit: 3<br />
timeout: 400<br />
Relying only on client-side timeouts has the adverse effect of the original request having timed out on the client to proxy connection, but still pending and outstanding on the proxy to server connection. This further gets exacerbated when client retries the original request.<br />
<br />
<br />
<b><span style="font-size: large;">Benefits of using Twemproxy for Redis Scaling</span></b><br />
<br />
<ul>
<li>Avoids re inventing the wheel. Thanks Manju Raj (twitter).</li>
<li>reduce the number of connections to your cache server by acting as a proxy</li>
<li>shard data automatically between multiple cache servers</li>
<li>support consistent hashing with different strategies and hashing functions</li>
<li>be configured to disable nodes on failure</li>
<li>run in multiple instances, allowing client to connect to the first available proxy server</li>
<li>Pipelining and batching of requests and hence saving of round-trips</li>
</ul>
<br />
<br />
<b><span style="font-size: large;">Disadvantages of Partitioning Model:</span></b><br />
<br />
<b>Point 1)</b> Operations involving multiple keys are usually not supported. For instance you can't perform the intersection between two sets if they are stored in keys that are mapped to different Redis instances (actually there are ways to do this, but not directly).Redis transactions involving multiple keys can not be used.<br />
<b>Point 2) </b>The partitioning granularity is the key, so it is not possible to shard a dataset with a single huge key like a very big sorted set. Ideally in such cases you should Scale UP the particular Redis Master-Slave to larger EC2 instance or pro grammatically stitch up the sorted set.<br />
<b>Point 3)</b>When partitioning is used, data handling is more complex, for instance you have to handle multiple RDB / AOF files, and to make a backup of your data you need to aggregate the persistence files/snapshots from multiple EC2 Redis slaves.<br />
<b>Point 4)</b> Architecting a partitioned + replicated ElastiCache Redis tier not complex. What is more complex is ? supporting transparent rebalancing of data with the ability to add and remove nodes at runtime. Systems like client side partitioning and proxies don't support this feature. However a technique called Presharding helps in this regard with limitations. Presharding technique ->Since Redis is lightweight, you can start with a lot of EC2 instances since the beginning itself. For example if you start with 32 or 64 EC2 instances (micro or small Cache Node instance type) as your node capacity , it will provide enough room to keep scaling up the capacity when your data storage needs increase. It is not a highly recommended technique. But still can be used in production if your growth pattern is very predictable.<br />
<div>
<div>
<br /></div>
<b><span style="font-size: large;">Future of highly scalable + available Redis tiers -> Redis Cluster</span></b></div>
<div>
Redis Cluster is the preferred way to get automatic sharding and high availability. It is currently not production ready. Once Redis Cluster / Client is available on Amazon ElastiCache, it will be the de facto standard for Redis partitioning. It uses a mix between query routing and client side partitioning.<br />
<div>
<br /></div>
<b>References:</b><br />
<a href="http://redis.io/documentation">http://redis.io/documentation</a><br />
<a href="https://github.com/twitter/twemproxy">https://github.com/twitter/twemproxy</a><br />
<br />
This article was co-authored with <a href="http://in.linkedin.com/in/sent2020">Senthil</a><br />
<br />
<b><span style="font-size: large;">Other Amazon ElastiCache Articles :</span></b><br />
<b><u><a href="http://harish11g.blogspot.in/2012/11/amazon-elasticache-memcached-ec2.html">Caching architectures using Memcached & Amazon ElastiCache</a></u></b><br />
<b><u><a href="http://harish11g.blogspot.in/2014/01/Creating-Elasticache-redis-replication-cluster-inside-amazon-vpc-guide-tutorial.html">Highly Available ElastiCache Redis replication cluster in AWS VPC</a></u></b><br />
<b><u><a href="http://harish11g.blogspot.in/2013/01/amazon-elasticache-memcached-internals_8.html">Deep dive into Amazon Elasticache</a></u></b><br />
<b><u><a href="http://harish11g.blogspot.in/2012/12/amazon-elasticache-distributed-caching.html">Distributed Cache On Steroids: Amazon ElastiCache</a></u></b><br />
<b><u><a href="http://harish11g.blogspot.in/2012/11/configuring-amazon-elasticache-launch.html">Launching Amazon ElastiCache in 3 Easy Steps</a></u></b><br />
<b><u><a href="http://harish11g.blogspot.in/2013/08/using-Amazon-ElastiCache-to-improve-performance-tips.html">AWS Performance Tip 3: Use Amazon ElastiCache to improve application performance</a></u></b><br />
<br />
<br />
<br />
<br /></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-91898365180380312122014-08-14T17:46:00.003-07:002014-10-12T09:54:31.877-07:0019 Best Practices for Creating Amazon CloudFormation Templates <div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-family: Verdana, sans-serif; font-size: 15px; line-height: 17.600000381469727px;">Amazon CloudFormation templates are widely used in the AWS cloud for environment creation by the IT and application teams. We have been helping enterprises with Amazon CFT based automation for years and following are some of the best practices to follow while creating Amazon CFT Templates. This article compiles the CFT practices /pointers used by us for provisioning complex large scale environments on AWS, not all of them will be applicable for all use cases.</span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.25pt;"><b>Practice #1:
Version your CloudFormation templates</b></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">CloudFormation template
should be commenced with a template format version and description such that it
could be artifact-able with version control tools like Git, SVN, CVS, etc.,<o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey;"><span style="font-family: Verdana, sans-serif;">Example:<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"AWSTemplateFormatVersion":
"2010-09-09",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Description": "AWS
CloudFormation Template for CustomerXYZ Version 5.0",<o:p></o:p></span></span></i></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-variant: small-caps; letter-spacing: .25pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-weight: bold;"><span style="font-family: Verdana, sans-serif;"><b>Practice #2: Use
input parameters </b><o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">Input parameters section
should be developed in scope with getting values from the end users of the CFT
for environments, SSH Key Pairs, EC2 instance types, RDS instance types,
ElastiCache node types, etc., and this makes the entire CloudFormation template
configurable and maintainable.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey;"><span style="font-family: Verdana, sans-serif;">Example:<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"WebTierKeyName":
{<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Description": "Name
of an existing EC2 KeyPair to enable SSH access to the WebTier",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Type":
"String",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "MinLength":
"1",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "MaxLength":
"64",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Default":
"testkey",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "AllowedPattern":
"[-_ a-zA-Z0-9]*",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "ConstraintDescription":
"can contain only alphanumeric characters, spaces, dashes and
underscores."<o:p></o:p></span></span></i></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-variant: small-caps; letter-spacing: .25pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-weight: bold;"><span style="font-family: Verdana, sans-serif;"><b>Practice #3: AMI
ID should be a parameter in inputs section</b><o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">AMI ID should be asked as
an input parameter from the end users for launching EC2 instances. It is highly
recommended not to hard code it inside your template and make it configurable
dynamically.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey;"><span style="font-family: Verdana, sans-serif;">Example:<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"NatAMIID":
{<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Type":
"String",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Default":
"ami-XXXXXXXX",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Description": "The
AMIID of NAT Instance"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></i></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-variant: small-caps; letter-spacing: .25pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-weight: bold;"><span style="font-family: Verdana, sans-serif;"><b>Practice #4:
Mention the AMI mappings</b><o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">AMI Mappings should be
included in the CloudFormation template to validate the respective AMI(s) in
the region(s) specified else it will take default AMI for the region (which is
not advisable).<o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey;"><span style="font-family: Verdana, sans-serif;">Example:<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"Mappings":
{<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "RegionMap": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "us-east-1": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "AMI":
"ami-XXXXXXXX"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"us-west-2":
{<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "AMI":
"ami-XXXXXXXX"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></i></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-variant: small-caps; letter-spacing: .25pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-weight: bold;"><span style="font-family: Verdana, sans-serif;"><b>Practice #5: Use
WaitConditionHandle, WaitCondition, DependsOn wherever applicable<o:p></o:p></b></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">When creating a large
multi-tiered infrastructure using CFT on AWS, there are times when the order of
the resource launch is important. For Example WaitCondition that waits for the
desired number of instances in a web server group.<o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">Using the AWS::CloudFormation::WaitCondition
and AWS::CloudFormation::WaitConditionHandle resources, a wait condition
can be placed with in a template to make AWS CloudFormation pause the creation
of the stack and wait for a signal before it continues to create the stack.<o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">We can specify that the
creation of a specific resource to follow another using the
“DependsOn” attribute. On adding a DependsOn attribute to a
resource, it is created only after the creation of the resource specified in
the DependsOn attribute. <o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey;"><span style="font-family: Verdana, sans-serif;">Example:<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"WebServerGroup"
: {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Type" :
"AWS::AutoScaling::AutoScalingGroup",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Properties" : {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "AvailabilityZones" : {
"Fn::GetAZs" : "" },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "LaunchConfigurationName" : {
"Ref" : "LaunchConfig" },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "MinSize" : "1",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "MaxSize" : "5",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">
"DesiredCapacity" : { "Ref" :
"WebServerCapacity" },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "LoadBalancerNames" : [ {
"Ref" : "ElasticLoadBalancer" } ]<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">},<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"WaitHandle"
: {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Type" :
"AWS::CloudFormation::WaitConditionHandle"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">},<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"WaitCondition"
: {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Type" : "AWS::CloudFormation::WaitCondition",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "DependsOn" :
"WebServerGroup",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Properties" : {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Handle" : { "Ref" : "WaitHandle"
},<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Timeout" : "300",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Count" : { "Ref" :
"WebServerCapacity" }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">} <o:p></o:p></span></span></i></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">For example, an Amazon EC2
instance with a public IP address is dependent on the VPC-gateway attachment if
the VPC and Internet Gateway resources are also declared in
the same template. The following snippet shows a sample gateway attachment and
an Amazon EC2 instance that depends on that gateway attachment:<o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"GatewayToInternet"
: {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Type" :
"AWS::EC2::VPCGatewayAttachment",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Properties" : {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "VpcId" : { "Ref" :
"VPC" },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "InternetGatewayId" : {
"Ref" : "InternetGateway" }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">},<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"EC2Host"
: {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Type" :
"AWS::EC2::Instance",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "DependsOn" :
"GatewayToInternet",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Properties" : {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "InstanceType" : {
"Ref" : "EC2InstanceType" },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "KeyName" : { "Ref" : "KeyName" },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "ImageId" : { "Fn::FindInMap" : [
"AWSRegionArch2AMI", { "Ref" : "AWS::Region" },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> { "Fn::FindInMap" : [
"AWSInstanceType2Arch", { "Ref" :
"EC2InstanceType" }, "Arch" ] } ] },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "NetworkInterfaces" : [{<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "GroupSet" : [{ "Ref" :
"EC2SecurityGroup" }],<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "AssociatePublicIpAddress" :
"true",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "DeviceIndex" : "0",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "DeleteOnTermination" : "true",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "SubnetId" : { "Ref" :
"PublicSubnet" }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }]<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">}<o:p></o:p></span></span></i></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-variant: small-caps; letter-spacing: .25pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-weight: bold;"><span style="font-family: Verdana, sans-serif;"><b>Practice #6: Tag
the resources properly</b><o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">Tags should be used for the
creation of AWS resources. Tags can be environment, purpose, application
specification etc., In addition to the stack name tags that AWS CloudFormation
provides, custom tags can be added to the resources that support tagging. This
helps in easy grouping of the assets associated with the environment.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey;"><span style="font-family: Verdana, sans-serif;">Example:<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"Tags":
[<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Key":
"Environment",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Value": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Ref":
"Environment"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Key":
"Purpose",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Value":
"CustomerXYZ VPC"<o:p></o:p></span></span></i></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-variant: small-caps; letter-spacing: .25pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-weight: bold;"><span style="font-family: Verdana, sans-serif;"><b>Practice #7:
Understand your Resources </b><o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">When a CloudFormation
template is deleted all AWS resources assigned with the template will be
automatically deleted by AWS. For example a VPC template will have all the
related resources like Route tables, Subnets, Network ACLs, etc., will be
deleted.<o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">In real scenario, S3 buckets
logs, EBS snapshots etc are associated with an infrastructure but they will not
terminated when the CFT is deleted, hence have to be automated for deletion
using scripts.<o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-variant: small-caps; letter-spacing: .25pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-weight: bold;"><span style="font-family: Verdana, sans-serif;"><b>Practice #8:
Create your Security groups using CFT</b><o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">Security Groups can be
governed and controlled using CloudFormation template for various tiers like
Web, App, DB, etc.,<o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"GatewayTierSg":
{<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Type":
"AWS::EC2::SecurityGroup",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Properties": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "GroupDescription":
"Gateway Security Group",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "VpcId": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Ref":
"Vpc"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">
"SecurityGroupIngress": [<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "IpProtocol":
"6",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "FromPort":
"22",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "ToPort":
"22",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "CidrIp":
"0.0.0.0/0"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> ],<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">
"SecurityGroupEgress": [<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "IpProtocol":
"-1",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "CidrIp":
"10.0.0.0/16"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> ]<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></i></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="color: #006699; font-variant: small-caps; letter-spacing: .25pt; mso-bidi-font-family: "Times New Roman"; mso-bidi-font-weight: bold;"><span style="font-family: Verdana, sans-serif;"><b>Practice #9:
Include the Output Section.</b><o:p></o:p></span></span></div>
<div align="left" class="MsoNormal" style="margin: 12pt 0in 6pt;">
<span style="font-size: 11pt; line-height: 120%;"><span style="font-family: Verdana, sans-serif;">Output section should be
declared with AWS specific end points to trace back. The template Outputs
section enables in returning one or more values to the user in response to the
AWS CloudFormation describe-stacks command. Example your endpoints are shown on
the Output section.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<i><span style="color: grey;"><span style="font-family: Verdana, sans-serif;">Example:<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">"RDSDatabaseEndpointDetail":
{<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Description":
"RDSDatabaseEndpointDetails",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Value": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Fn::GetAtt": [<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "RdsTierService",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">
"Endpoint.Address"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> ]<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "ElastiCacheEndpointDetail":
{<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Description":
"ElastiCacheEndpointDetails",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Value": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Fn::GetAtt": [<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">
"ElatiCacheTierService",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "ConfigurationEndpoint.Address"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> ]<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">
"WebTierElbExternalEndpointDetail": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Description":
"WebTierElb External Endpoint Access Details",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Value": {<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "Fn::GetAtt": [<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;">
"WebTierElbExternal",<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> "DNSName"<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> ]<o:p></o:p></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></i></div>
<br />
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"> },</span></span></i><o:p></o:p></div>
<div class="MsoNormal">
<b style="text-indent: -0.25in;"><span style="font-size: 7pt; font-weight: normal;"><br /></span></b>
<b style="text-indent: -0.25in;"><span style="font-size: 7pt; font-weight: normal;"><br /></span></b>
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #10: Automate Application Installation using cloud-init</b><br />
<br />
<div class="MsoListParagraphCxSpLast">
<span style="font-family: Verdana, sans-serif;">With the help of cloud-init the installation
of various packages can be done during the boot itself just by passing the
appropriate commands. In the cloud-init the scripts also can be passed through
the automation of various operations can be done.<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpLast">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>"Resources" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Ec2Instance" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Type" :
"AWS::EC2::Instance",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Properties" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "KeyName" : {
"Ref" : "KeyName" },<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "SecurityGroups"
: [ { "Ref" : "InstanceSecurityGroup" } ],<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "ImageId" : {
"Fn::FindInMap" : [ "RegionMap", { "Ref" :
"AWS::Region" }, "AMI" ]},<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "UserData" : {
"Fn::Base64" : { "Fn::Join" : ["",[<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "#!/bin/bash
-ex","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "yum -y install
gcc-c++ make","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "yum -y install mysql-devel
sqlite-devel","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "yum -y install
ruby-rdoc rubygems ruby-mysql ruby-devel","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "gem install
--no-ri --no-rdoc rails","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "gem install
--no-ri --no-rdoc mysql","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "gem install --no-ri --no-rdoc
sqlite3","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "rails new
myapp","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "cd
myapp","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "rails server
-d","\n",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "curl -X PUT -H
'Content-Type:' --data-binary '{\"Status\" :
\"SUCCESS\",",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"\"Reason\" : \"The application myapp is
ready\",",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"\"UniqueId\" : \"myapp\",",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"\"Data\" :
\"Done\"}' ",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"\"", {"Ref" :
"WaitForInstanceWaitHandle"},"\"\n" ]]}}<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoListParagraphCxSpFirst">
<br /></div>
<div class="MsoListParagraphCxSpFirst">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #11: Use helper scripts to start/stop services etc</b></div>
<div class="MsoListParagraphCxSpFirst">
<b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><br /></b></div>
<div class="MsoListParagraphCxSpFirst">
<span style="font-family: Verdana, sans-serif;">AWS CloudFormation provides a set of Python
helper scripts that you can use to install software and start services on an
Amazon EC2 instance that you create as part of your stack. You can call the
helper scripts directly from your template. The scripts work in conjunction
with resource metadata that you define in the same template. The helper scripts
run on the Amazon EC2 instance as part of the stack creation process.</span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: Verdana, sans-serif;"><span style="font-size: 10pt; line-height: 115%;"><br /><i><span style="color: #666666;">
"Resources" : {</span></i></span><i><span style="color: #666666;"> <br />
<span style="font-size: 10pt; line-height: 115%;"><br />
"WebServer": { <br />
"Type": "AWS::EC2::Instance",<br />
"Metadata" : {<br />
"Comment1" : "Configure the bootstrap helpers to install the
Apache Web Server and PHP",<br />
"Comment2" : "The website content is downloaded from the
CloudFormationPHPSample.zip file",<br />
<br />
"AWS::CloudFormation::Init" : {<br />
"config" : {<br />
"packages" : {<br />
"yum" : {<br />
"mysql" : [],<br />
"mysql-server" : [],<br />
"mysql-libs" : [],<br />
"httpd" : [],<br />
"php" : [],<br />
"php-mysql" : []<br />
}<br />
},<br />
<br />
"sources" : {<br />
"/var/www/html" : "</span><a href="https://s3.amazonaws.com/cloudformation-examples/CloudFormationPHPSample.zip"><span style="font-size: 10pt; line-height: 115%; text-decoration: none;">https://s3.amazonaws.com/cloudformation-examples/CloudFormationPHPSample.zip</span></a><span style="font-size: 10pt; line-height: 115%;">"<br />
},<br />
<br />
"files" : {<br />
"/tmp/setup.mysql" : {<br />
"content" : { "Fn::Join" : ["", [<br />
"CREATE DATABASE ", { "Ref" : "DBName" },
";\n",<br />
"GRANT ALL ON ", { "Ref" : "DBName" }, ".*
TO '", { "Ref" : "DBUsername" }, "'@localhost
IDENTIFIED BY '", { "Ref" : "DBPassword" },
"';\n"<br />
]]},<br />
"mode" : "000644",<br />
"owner" : "root",<br />
"group" : "root"<br />
}<br />
},</span></span></i>
<!--[if !supportLineBreakNewLine]--><br />
<!--[endif]--><o:p></o:p></span></div>
<div class="MsoListParagraphCxSpMiddle">
<b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><br /></b></div>
<div class="MsoListParagraphCxSpMiddle">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #12: Make use of parameter section constraints (wherever applicable)</b></div>
<div class="MsoListParagraphCxSpMiddle">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;"><br /></b></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: Verdana, sans-serif;">The parameter sections provide lot of
constraints like MinValue, Max Value, Allowed values, Max length, etc. By
effective usage of the of the parameter section you can validate the input
parameters from the user. Example: We can restrict TCP port value to be between 1 to 65535, we can restrict the webserver port to 80,8888 etc. This feature is particularly useful when you are developing complex cloudformation template for automation.</span></div>
<div class="MsoListParagraphCxSpLast">
<span style="font-family: Verdana, sans-serif;">
<br />
<i><br />
<span style="font-size: 10pt; line-height: 115%;"><span style="color: #666666;">"Parameters"
: {<br />
"UserAccount": {<br />
"Default": "guest",<br />
"NoEcho": "true",<br />
"Description" : "The guest
account user name",<br />
"Type": "String",<br />
"MinLength": "1",<br />
"MaxLength": "16",<br />
"AllowedPattern" :
"[a-zA-Z][a-zA-Z0-9]*"<br />
}<br />
}<br />
<br />
"Parameters" : {<br />
"WebServerPort": {<br />
"Default": "80",<br />
"Description" : "TCP/IP port for
the web server",<br />
"Type": "Number",<br />
"MinValue": "1",<br />
"MaxValue": "65535"<br />
}<br />
}<br />
<br />
"Parameters" : {<br />
"WebServerPort": {<br />
"Default": "80",<br />
"Description" : "Port for the web
server",<br />
"Type": "Number",<br />
"AllowedValues" : ["80",
"8888"]<br />
}<br />
}</span></span></i><span style="color: #666666;"><o:p></o:p></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<o:p><b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #13: Use NoEcho property for sensitive information</b></o:p></div>
<div class="MsoNormal">
<o:p><b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;"><br /></b></o:p></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">W</span><span style="font-family: Verdana, sans-serif; text-indent: 0.25in;">hen the cloudformation is in progress
we can use describe-stacks to check the progress of the stack, in this case we
can see the parameters values being returned. If we use the NoEcho Property like below, the sensitive information will not be returned.</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="font-family: Verdana, sans-serif;"><span style="color: rgba(0, 0, 0, 0);"> </span><i><span style="color: #666666;">"AdminPassword": {<o:p></o:p></span></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "NoEcho":
"true",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Description" :
"The Joomla! admin account password",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Type":
"String",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "MinLength":
"1",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "MaxLength":
"41",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "AllowedPattern" :
"[a-zA-Z0-9]*",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"ConstraintDescription" : "must contain only alphanumeric
characters."<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: windowtext; font-size: 10pt;"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #14: Use Select Function and reduce the parameters in the template</b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b style="font-family: Verdana, sans-serif; text-indent: 0.25in;">Fn::Select intrinsic function:</b><span style="font-family: Verdana, sans-serif;">Using the above function in the in the
Resources section of your template will help you to combine related parameters
which can reduce the total number of parameters in your template.</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>"Parameters" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "DbSubnetBlocks": {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Description":
"Three CIDR blocks",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Type":
"CommaDelimitedList",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Default":
"10.0.58.0/24, 10.0.59.0/24, 10.0.60.0/24"<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>}<o:p></o:p></i></span></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">To specify one of the three CIDR blocks, use Fn::Select in
the Resources section of the same template, as shown in the following sample
snippet:<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>"Subnet1": {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Type":
"AWS::EC2::Subnet",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Properties": {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "VpcId": {
"Ref": "VPC" },<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "CidrBlock": {
"Fn::Select" : [ "0", {"Ref":
"DbSubnetBlocks"} ] }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="font-family: Verdana, sans-serif;"><i><span style="color: #666666;">}</span></i><span style="color: rgba(0, 0, 0, 0);"><o:p></o:p></span></span></span></div>
<div class="Textbody" style="margin-bottom: 14.15pt;">
<br /></div>
<div class="Textbody" style="margin-bottom: 14.15pt;">
<b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><span style="font-size: 7pt; font-weight: normal;"><br /></span></b></div>
<div class="Textbody" style="margin-bottom: 14.15pt;">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #15: Use intrinsic Functions to manage automation</b></div>
<div class="Textbody" style="margin-bottom: 14.15pt;">
<b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><span style="font-size: 7pt; font-weight: normal;"> </span></b><b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><span style="line-height: 115%; mso-bidi-font-family: Mangal; mso-bidi-font-size: 10.5pt;">Fn::FindInMap </span>intrinsic
function</b></div>
<div class="Textbody" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; text-indent: 0.25in;">The function </span><span class="SourceText" style="font-family: Verdana, sans-serif; text-indent: 0.25in;">Fn::FindInMap</span><span style="font-family: Verdana, sans-serif; text-indent: 0.25in;"> returns the value corresponding to keys
in a two-level map that is declared in the </span><span class="SourceText" style="font-family: Verdana, sans-serif; text-indent: 0.25in;">Mappings</span><span style="font-family: Verdana, sans-serif; text-indent: 0.25in;">
section.</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>{<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> ...<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Mappings" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "RegionMap" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "us-east-1" : {
"32" : "ami-6451e20d", "64" :
"ami-7a11e213" },<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "us-west-1" : {
"32" : "ami-d5gc7978c", "64" :
"ami-cfc7978a" }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> },<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Resources" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "myEC2Instance" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Type" :
"AWS::EC2::Instance",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Properties" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "ImageId" : {
"Fn::FindInMap" : [ "RegionMap", { "Ref" :
"AWS::Region" }, "32"]},<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"InstanceType" : "m3.large"<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>}<o:p></o:p></i></span></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">You can use intrinsic functions, such as <span class="SourceText">Fn::If</span>, <span class="SourceText">Fn::Equals</span>, and <span class="SourceText">Fn::Not</span>, to conditionally create stack resources. These
conditions are evaluated based on input parameters that you declare when you
create or update a stack. After you define all your conditions, you can
associate them with resources or resource properties in the Resources and
Outputs sections of a template<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #16: Properties section</b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">If a resource does not require any properties to be
declared, you can omit the Properties section of that resource. This will keep the template code simple and manageable<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>"Resources" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "WebInstance" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Type" :
"AWS::EC2::Instance",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Properties" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "UserData" :
{<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"Fn::Base64" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"Fn::Join" : [ "", [ "Queue=", {
"Ref" : "MyQueue" } ] ]<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> } },<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "ImageId"
: "ami-20b65349"<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> },<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "MyQueue" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Type" :
"AWS::SQS::Queue",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Properties" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>} <o:p></o:p></i></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><span style="font-size: 7pt; font-weight: normal;"><br /></span></b></div>
<div class="MsoNormal">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #17: Planning Windows Bootstrapping using Amazon CFT</b></div>
<div class="MsoNormal">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;"><br /></b></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif; text-indent: 0.25in;">In order for the Windows bootstrapping to work the
CloudFormation the windows instance should be setup with the EC2ConfigService. </span><span style="font-family: Verdana, sans-serif; text-indent: 0.25in;">The AWS CloudFormation helper script cfn-init is used to
perform each of these actions, based on information in the </span><a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html" style="font-family: Verdana, sans-serif; text-indent: 0.25in;" title="AWS::CloudFormation::Init"><span style="color: #00000a; text-decoration: none; text-underline: none;">AWS::CloudFormation::Init</span></a><span style="font-family: Verdana, sans-serif; text-indent: 0.25in;"> resource.</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">"SharePointFoundation": {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "Type" :
"AWS::EC2::Instance",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "Metadata" : {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">
"AWS::CloudFormation::Init" : {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "config" : {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">"files" : {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">
"c:\\cfn\\cfn-hup.conf" : {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "content" : {
"Fn::Join" : ["", [<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "[main]\n",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "stack=", {
"Ref" : "AWS::StackName" }, "\n",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "region=", {
"Ref" : "AWS::Region" }, "\n"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> ]]}<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">
"c:\\cfn\\hooks.d\\cfn-auto-reloader.conf" : {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "content": {
"Fn::Join" : ["", [<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">
"[cfn-auto-reloader-hook]\n",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">
"triggers=post.update\n",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "path=Resources.SharePointFoundation.Metadata.AWS::CloudFormation::Init\n",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> "action=cfn-init.exe -v
-s ", { "Ref" : "AWS::StackName" },<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">
" -r SharePointFoundation",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">
" --region ", { "Ref" : "AWS::Region" },
"\n"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> ]]}<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> },<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">
"C:\\SharePoint\\SharePointFoundation2010.exe" : {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: #666666; font-family: Verdana, sans-serif;"><span style="font-size: 10pt;"> "source" : "</span><a href="http://d3adzpja92utk0.cloudfront.net/SharePointFoundation.exe"><span style="font-size: 10pt; text-decoration: none;">http://d3adzpja92utk0.cloudfront.net/SharePointFoundation.exe</span></a><span style="font-size: 10pt;">"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"> }<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;">},</span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><span style="font-size: 7pt; font-weight: normal;"><br /></span></b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><span style="font-size: 7pt; font-weight: normal;"><br /></span></b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #18: Use Nested Stacks </b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt;">You can declare only Maximum of 200 resources in your AWS
CloudFormation template. For complex environments and automation stacks you will need more than that, it is recommended to separate your
template into multiple templates using the nested stacks feature.</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt;"><br /></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>{<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"AWSTemplateFormatVersion" : "2010-09-09",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Resources" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "myStack" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Type" :
"AWS::CloudFormation::Stack",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Properties" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "TemplateURL" :
"https://s3.amazonaws.com/cloudformation-templates-us-east-1/S3_Bucket.template",<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "TimeoutInMinutes"
: "60"<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> },<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Outputs": {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "StackRef":
{"Value": { "Ref" : "myStack"}},<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>
"OutputFromNestedStack" : {<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> "Value" : {
"Fn::GetAtt" : [ "myStack", "Outputs.BucketName"
] }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i> }<o:p></o:p></i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-size: 10pt;"><span style="color: #666666; font-family: Verdana, sans-serif;"><i>}</i></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b style="font-family: Verdana, sans-serif; text-indent: -0.25in;"><br /></b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b style="color: #006699; font-family: Verdana, sans-serif; font-variant: small-caps; letter-spacing: 0.333333343267441px;">Practice #19: Integrate Amazon CFT with Amazon CloudTrail</b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt; text-indent: 0.25in;">It is recommended that AWS CloudFormation is integrated with the CloudTrail. The
actions related to CloudFormation like </span><b style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt; text-indent: 0.25in;">CreateStack</b><span style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt; text-indent: 0.25in;">, </span><b style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt; text-indent: 0.25in;">DeleteStack</b><span style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt; text-indent: 0.25in;">,
and </span><b style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt; text-indent: 0.25in;">ListStacks</b><span style="color: #00000a; font-family: Verdana, sans-serif; font-size: 12pt; text-indent: 0.25in;"> actions generate entries in CloudTrail log files. These logs can be processed for audit compliance purpose later.</span></div>
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"><br /></span></span></i>
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"><br /></span></span></i></div>
<div class="MsoNormal">
<i><span style="color: grey; font-size: 9.0pt; line-height: 120%; mso-bidi-font-size: 12.0pt;"><span style="font-family: Verdana, sans-serif;"><br /></span></span></i></div>
<div class="MsoNormal">
<br /></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com2tag:blogger.com,1999:blog-4836510665235566234.post-84147328138074409092014-04-22T09:12:00.000-07:002014-04-29T21:55:48.312-07:00Loading Big Index Data into newly launched Amazon CloudSearch engine<span style="font-family: Verdana, sans-serif;">Search tier is the most critical section of many online verticals like travel, e-commerce, classifieds etc. If users cannot search products efficiently they will not make their buying decisions properly, which in turn massively affects the revenues of these companies. Most of them are usually powered by Apache Solr, FAST , Autonomy, ElastiSearch etc. </span><span style="font-family: Verdana, sans-serif;">AWS also has a Search Service called CloudSearch which is a </span><span style="font-family: Verdana, sans-serif;">fully-managed service in the cloud that makes it easy to set up, manage, and scale a search solution for your website. Amazon CloudSearch relieves you from the worry of hardware provisioning, setup, and maintenance. As your volume of data and traffic fluctuates, Amazon CloudSearch automatically scales to meet your needs.</span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<span style="font-family: Verdana, sans-serif;">In AWS infrastructure Apache Solr has been the king and the software to beat till now, recently it has got heavy competitor in the form of Amazon CloudSearch -</span><span style="font-family: Verdana, sans-serif;"> API </span><span style="font-family: Verdana, sans-serif;">2013-01-01</span><span style="font-family: Verdana, sans-serif;">. </span><br />
<span style="font-family: Verdana, sans-serif;">API version 2013-01-01 of Amazon CloudSearch is internally powered by customized version of Apache Solr Engine, and it is specifically designed for running highly scalable and available search on Amazon Web Services Cloud. This 2013 CloudSearch API has lots of similarities with Apache Solr and customers can easily migrate to this version and leverage the benefits of Amazon Cloud Infrastructure. We are a</span><span style="font-family: Verdana, sans-serif;">lready hearing many AWS customers are planning their migration from FAST, Solr and A9 Engine into the Amazon CloudSearch - </span><span style="font-family: Verdana, sans-serif;">2013-01-01 API engine. </span><br />
<br />
<span style="font-family: Verdana, sans-serif;">My team is already migrating couple of customers into this Amazon CloudSearch </span><span style="font-family: Verdana, sans-serif;">2013-01-01 API and i have shared our experience on this process for the benefit of AWS community.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Reference Migration Architecture and requirements:</b></span><br />
<span style="font-family: Verdana, sans-serif;"><b><br /></b></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhK3hYhnW01snL-ckVgMSJBdh8EOlBy0xh1o7ULHn8KpZpFVRWlpQtdO9n_PGFzDyZrNaRixidO4kz56O3HsEFPC6_9OyaXCMa3UXdoTqqcZEsT5tl24f8eoSh6uAEn9bBLnau1PlFeFpUM/s1600/Highly+Scalable+Data+loading+architecture+into+Amazon+CloudSearch+using+Amazon+EMR.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhK3hYhnW01snL-ckVgMSJBdh8EOlBy0xh1o7ULHn8KpZpFVRWlpQtdO9n_PGFzDyZrNaRixidO4kz56O3HsEFPC6_9OyaXCMa3UXdoTqqcZEsT5tl24f8eoSh6uAEn9bBLnau1PlFeFpUM/s1600/Highly+Scalable+Data+loading+architecture+into+Amazon+CloudSearch+using+Amazon+EMR.jpg" height="470" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;"><b><br /></b></span>
<span style="font-family: Verdana, sans-serif;">In this article i am going to explore how to </span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">Migrate a 300+ GB index containing close to 247+ million records distributed in 105 searchable fields in a highly scalable /parallel manner in AWS infrastructure.</span></li>
<li><span style="font-family: Verdana, sans-serif;">300 + GB index file is stored in Amazon S3</span></li>
<li><span style="font-family: Verdana, sans-serif;">Custom Data loader program built on Amazon Elastic MapReduce is used for parallel loading</span></li>
<li><span style="font-family: Verdana, sans-serif;">Around ~6 Search.M2.2Xlarge are created with 2 partitions and 5 replication count</span></li>
<li><span style="font-family: Verdana, sans-serif;">Around 10+ M1.large EMR Core nodes are for Data loading. This loader can be increased to hundreds of nodes depending upon the volume and velocity of data pump required.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Amazon CloudSearch Infrastructure provisioning, Automated partitioning, replication count are handled by AWS. </span></li>
</ul>
<br />
<span style="font-family: Verdana, sans-serif;">Lets get into the details below:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 1)Create a new Amazon CloudSearch Domain:</b> We have named the search domain as "bigdatasearch" and chose the search instance type as search.m2.2xlarge. Since we are planning to pump and query a 300 GB index with millions of document, it did not make sense for us to chose a smaller instance type of Amazon CloudSearch. Usually the base instance type can be</span><span style="font-family: Verdana, sans-serif;"> selected based on the number and size of the documents you are planning to maintain in the Amazon CloudSearch. </span><br />
<span style="font-family: Verdana, sans-serif;"><b>Note:</b> Here we have chosen replication count as 5. This is little strange in a distributed architecture because usually more replication count for the master decreases the speed of document upload. But when we were playing with Amazon CloudSearch we observed that it is increasing the speed of uploads. In addition we also observed the following :</span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">If we keep the replication count 0 or less, use a smaller search instance type and pump documents in parallel from multiple nodes, either the Amazon CloudSearch Server is failing sometimes or error rates are high.</span></li>
<li><span style="font-family: Verdana, sans-serif;">If we keep the replication count 0 or less , use a larger search instance type and pump documents in parallel from multiple nodes, internally Amazon Cloud Search itself is creating 3-5 nodes and it shows in the replication count. </span><span style="font-family: Verdana, sans-serif;">Waiting to discuss with AWS SA folks on this behavior.</span></li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3u62bCT9mSgdAgxy9slhbd7ctZBdkFtO4nOxoBIB6IxTFtBfI2kLpXXEqlc0jXtJFQVltxs9myfkV8x9Lf75jzk59QAc_M7TjVzlTmB2mpLTUFxjpZ6th4gHDM8qCOhkk_ZTmSATe8-3e/s1600/1-Create+New+Amazon+CloudSearch+Domain+p1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3u62bCT9mSgdAgxy9slhbd7ctZBdkFtO4nOxoBIB6IxTFtBfI2kLpXXEqlc0jXtJFQVltxs9myfkV8x9Lf75jzk59QAc_M7TjVzlTmB2mpLTUFxjpZ6th4gHDM8qCOhkk_ZTmSATe8-3e/s1600/1-Create+New+Amazon+CloudSearch+Domain+p1.jpg" height="175" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<span style="font-family: Verdana, sans-serif;">We </span><span style="font-family: Verdana, sans-serif;">will be utilizing distributed uploading technique which we custom built using Amazon Elastic MapReduce to pump data to the Amazon CloudSearch server. This technique enables us to write more Index data in parallel. </span><br />
<div>
<br /></div>
<span style="font-family: Verdana, sans-serif;"><b>Step 2) Select how you would like to create the Amazon CloudSearch Schema: </b>Here we have chosen Manual setup, since we already have schema to be migrated to Amazon CloudSearch.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXrQZjOg_PDJDJmk-jF2V1-Yv-9U4FPAcOnS9NaLfCjLBQPRq0dazuWhqTeK2aGl8RJzWZ1YiXPbJxHDiz52YPrF7WLw2xAEtVLJvG3GYGnjvaoPPmlu97wpIsCAwUITlUeXEo0SR7lcK0/s1600/2-Create+New+Amazon+CloudSearch+-Configure+Index+p2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXrQZjOg_PDJDJmk-jF2V1-Yv-9U4FPAcOnS9NaLfCjLBQPRq0dazuWhqTeK2aGl8RJzWZ1YiXPbJxHDiz52YPrF7WLw2xAEtVLJvG3GYGnjvaoPPmlu97wpIsCAwUITlUeXEo0SR7lcK0/s1600/2-Create+New+Amazon+CloudSearch+-Configure+Index+p2.jpg" height="152" width="640" /></a></div>
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Verdana, sans-serif;">Next step is to Add index fields to create your Amazon CloudSearch Schema configuration.</span></div>
<br />
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 3)Adding Amazon CloudSearch Index Fields:</b> Once all the fields have been configured in the schema, click on continue button. In the schema file used we have 100+ fields to be indexed for this particular search domain.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYY0Tz-xFKmMUARA-wArmOxPatSALu0I2PWCUGefE79INr5_al_N3l-zVx-QGWAn2s1NBc9VTWHhyUffwuPOHhGWUJ3mo8bt5norqGrgEzGYBjKKWkIiyTxRHn4mlCVTJFn2iHSf60vxVI/s1600/4-Amazon+CloudSearch+Index+configuration+p4.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYY0Tz-xFKmMUARA-wArmOxPatSALu0I2PWCUGefE79INr5_al_N3l-zVx-QGWAn2s1NBc9VTWHhyUffwuPOHhGWUJ3mo8bt5norqGrgEzGYBjKKWkIiyTxRHn4mlCVTJFn2iHSf60vxVI/s1600/4-Amazon+CloudSearch+Index+configuration+p4.jpg" height="294" width="640" /></a></div>
<br /></div>
<div>
<br /></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 4) Review the setup configurations and launch:</b></span><br />
<span style="font-family: Verdana, sans-serif;">We have 100+ Index fields with scaling options instance type as m2.2xlarge and replication count 5 in the "bigdatasearch" domain.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO7kWKJGH_pk-Zxas8FjWyPCnK69yTJm-jghD6aV5VRf1EpIxoxovx4WAWn_PcXZkjUuadYuAeNGw57bYImV8c4m1mbJCgMz13lYHnO9TCkus5QCxxTFVZxQJJkg2gmKb3324wTIeYAvfp/s1600/6-Review+Amazon+CloudSearch+Domain+Setup+Configuration+6.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO7kWKJGH_pk-Zxas8FjWyPCnK69yTJm-jghD6aV5VRf1EpIxoxovx4WAWn_PcXZkjUuadYuAeNGw57bYImV8c4m1mbJCgMz13lYHnO9TCkus5QCxxTFVZxQJJkg2gmKb3324wTIeYAvfp/s1600/6-Review+Amazon+CloudSearch+Domain+Setup+Configuration+6.jpg" height="208" width="640" /></a></div>
<br /></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 5 ) Wait till the Amazon CloudSearch Infrastructure</b> is provisioned for you on the back. Usually it takes 10 minutes, it will also list if there is any error encountered when creating the index fields.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjL9oPfzc4bJfYRrma5sM-JNPWDkHXb1q4hqvJ3046a2BlZqSG5MKlmrpTDfMggm1ldU5bJ5xNpM2_Q60L8w_PZtzJJWHNBQEFPr5Xeu-BKdaPHFYmlQowGHV_8qTUy__2ABAv_uBjMKYTE/s1600/7-Launch+the+Amazon+CloudSearch+domain+p7.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjL9oPfzc4bJfYRrma5sM-JNPWDkHXb1q4hqvJ3046a2BlZqSG5MKlmrpTDfMggm1ldU5bJ5xNpM2_Q60L8w_PZtzJJWHNBQEFPr5Xeu-BKdaPHFYmlQowGHV_8qTUy__2ABAv_uBjMKYTE/s1600/7-Launch+the+Amazon+CloudSearch+domain+p7.jpg" height="98" width="640" /></a></div>
<br /></div>
<div>
<span style="font-family: Verdana, sans-serif;">Once the Amazon CloudSearch infrastructure is provisioned at the back end , you should notice the "bigdatasearch" domain is <b>"Active". </b></span><span style="font-family: Verdana, sans-serif;">The search and Document endpoints are published and currently no of searchable document is <b>"0". There is only 1 CloudSearch Index Partition (Shards) and 5 search.m2.2xlarge instances.</b></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFkhtGnenCDaW7J1YNf9VknICvo8cmAy9OYWIz7iS04bS7a6wtMMbUSorrmiUYh7K2ldFIor2qQEAXmdGeiRQvpmO-69yi36Pmym-TSjR7EevUgidJY6h-6yifVZuFllsCIeNUcotWDcl0/s1600/8-View+the+Amazon+CloudSearch+configuration+p8.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFkhtGnenCDaW7J1YNf9VknICvo8cmAy9OYWIz7iS04bS7a6wtMMbUSorrmiUYh7K2ldFIor2qQEAXmdGeiRQvpmO-69yi36Pmym-TSjR7EevUgidJY6h-6yifVZuFllsCIeNUcotWDcl0/s1600/8-View+the+Amazon+CloudSearch+configuration+p8.jpg" height="172" width="640" /></a></div>
<br /></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 6)Configuring Synonyms: </b>We have 2+ MB of Synonyms which needs to be configured into the Amazon CloudSearch domain. For this, we used Cloud Search cli-toolkit to upload synonyms to Cloud Search.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b><i>cs-configure-analysis-scheme -d bigdatasearch --name customanalysisscheme --lang en -e cloudsearch.ap-southeast-1.amazonaws.com --synonyms customsynonyms.txt</i></b></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<br />
<span style="font-family: Verdana, sans-serif;">Since the volume of index data is huge (300+ GB) we have created a Custom Data Loader built on Amazon Elastic MapReduce to pump the data in parallel into Amazon CloudSearch. Since it is built on Amazon Elastic MapReduce, we can use the same program without modification for scale to upload TB's of index into the search system with hundreds of Data loader EMR core/task nodes. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 7)<span class="Apple-tab-span" style="white-space: pre;"> </span>Create Amazon Elastic MapReduce Data Loader Cluster Configuration:</b></span></div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj51sw7LWJkpjWQH-Z60Zo_iFtkINJblD6poK1vhKDjhZcKP1oJU8U9XukDocLCdgLYMaKRqe2an1hy3T0rzccdmV5iXVD5mBDbH6I97za9CDvKmTkbd1wG_UYmedyqQaxincXm0cIH1qgC/s1600/9-Create+EMR+Cluster+Configuration+p9.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj51sw7LWJkpjWQH-Z60Zo_iFtkINJblD6poK1vhKDjhZcKP1oJU8U9XukDocLCdgLYMaKRqe2an1hy3T0rzccdmV5iXVD5mBDbH6I97za9CDvKmTkbd1wG_UYmedyqQaxincXm0cIH1qgC/s1600/9-Create+EMR+Cluster+Configuration+p9.jpg" height="330" width="640" /></a></div>
<br /></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 8) Configure the Elastic MapReduce (EMR) Capacity:</b> We are using 10 M1.Large core node instances for uploading the data from inside AWS VPC. Depending upon the Data size (GB->TB) and Upload hours we can increase the EMR core nodes capacity and number to speed up the data pump (upload) process.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJR5LfjUIOu6IhWxyKRWA82toi3eF5EfmRmR_yM2sAk4W8_GMwAJGlJ9WrG7naHiP2hg8btUwq22gL_K7t_39sz36oF3_m6eqnVMuNIAKT_sYJGg93Wj7aBNQxEO72laG-OXDTlm6QPUV9/s1600/10-Amazon+Elastic+MapReduce+Data+Loader+cluster+configuration+p10.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJR5LfjUIOu6IhWxyKRWA82toi3eF5EfmRmR_yM2sAk4W8_GMwAJGlJ9WrG7naHiP2hg8btUwq22gL_K7t_39sz36oF3_m6eqnVMuNIAKT_sYJGg93Wj7aBNQxEO72laG-OXDTlm6QPUV9/s1600/10-Amazon+Elastic+MapReduce+Data+Loader+cluster+configuration+p10.jpg" height="218" width="640" /></a></div>
<br />
<span style="font-family: Verdana, sans-serif;">To know more about How Spot instances can save cost on Amazon EMR ? refer URL </span><u style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-Adding-Spot-EC2-with-Amazon-Elastic-MapReduce-EMR.html">AWS Cost Saving Tip 12: Add Spot Instances with Amazon EMR</a></u><br />
<span style="font-family: Verdana, sans-serif;"><span style="color: red;"><br /></span></span>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 9)Add Custom data loader program Jar to EMR:</b> </span><br />
<span style="font-family: Verdana, sans-serif;">We have exported the data from a MSSQL server as flat UTF-8 dump file and stored it in Amazon S3. We are giving the 300+ GB Dump file as the input for the Amazon EMR CloudSearch Data Loader program to upload into Amazon CS in parallel. Buckets configurations of the Data Loader jar, Input, output and log files are configured in this screen</span><br />
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTnNIsBh5fzoXseDs9S2MFVPAc0ATllHeFLrG5dqAMN27JC_stnEdROBUV_qcZUgDcJlhdLlELE4H2MqyqYv_iC5DJsc1C5wNvqnXyv7v9_6ZUSckRefBr0EC4drdeMZQSLGTn-hM9ppR4/s1600/11-Amazon+S3+path+of+the+Input+Data+p11.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTnNIsBh5fzoXseDs9S2MFVPAc0ATllHeFLrG5dqAMN27JC_stnEdROBUV_qcZUgDcJlhdLlELE4H2MqyqYv_iC5DJsc1C5wNvqnXyv7v9_6ZUSckRefBr0EC4drdeMZQSLGTn-hM9ppR4/s1600/11-Amazon+S3+path+of+the+Input+Data+p11.jpg" height="260" width="640" /></a></div>
<br /></div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 10)</b><span class="Apple-tab-span" style="white-space: pre;"> </span><b>Configure Amazon CloudSearch Access Policies:</b> We need to open Cloud Search security group access policies to accept upload requests from EMR cluster inside VPC. </span><span style="font-family: Verdana, sans-serif;">Configure static IP’s of all the instances or IP range of the data loader clients</span></div>
<div>
<br /></div>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGZ8b8uNvzSrlRVjbKBLKPFB8Qq4Brx1EPLDETWqsYX64iwLKutuY5Z9cAZ4yLnbBAfvWtOQsutOKNQWRVU-CWPT9rTR8AjvRUt4majep7EAxDKBccSlmRXjvtzPssrgK7R8MOkGcmQhpa/s1600/12+-+CloudSearch+Access+Policies+to+EMR+p12.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGZ8b8uNvzSrlRVjbKBLKPFB8Qq4Brx1EPLDETWqsYX64iwLKutuY5Z9cAZ4yLnbBAfvWtOQsutOKNQWRVU-CWPT9rTR8AjvRUt4majep7EAxDKBccSlmRXjvtzPssrgK7R8MOkGcmQhpa/s1600/12+-+CloudSearch+Access+Policies+to+EMR+p12.jpg" height="318" width="640" /></a></div>
<br />
<span style="font-family: Verdana, sans-serif;"><b>Step 11)Run the Amazon Elastic MapReduce Data loader job :</b></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8dSYMfAa1mCsAEhZDro6wYb-ARHdIeAnFUzOCZ2sB_3rWpaSIzx1ryFYKhBzQlh274Vqk8y6afp-VFPsYli7nF-d6IKoj3jdRo7P5-0UOmApBgNaYwriMDBomrtX_HxtXG4S3IbYy2LCg/s1600/13-Amazon+CloudSearch+EMR+Data+Loader+Cluster+Details+p13.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8dSYMfAa1mCsAEhZDro6wYb-ARHdIeAnFUzOCZ2sB_3rWpaSIzx1ryFYKhBzQlh274Vqk8y6afp-VFPsYli7nF-d6IKoj3jdRo7P5-0UOmApBgNaYwriMDBomrtX_HxtXG4S3IbYy2LCg/s1600/13-Amazon+CloudSearch+EMR+Data+Loader+Cluster+Details+p13.jpg" height="268" width="640" /></a></div>
<br /></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 12)<span class="Apple-tab-span" style="white-space: pre;"> </span>Analyzing the Amazon EMR Data loader Job Output:</b></span><br />
<span style="font-family: Verdana, sans-serif;">Output of the JOB can be seen in the AWS EMR JOB logs. </span><span style="font-family: Verdana, sans-serif;">Here are few details:</span></div>
<div>
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">“Map output records” in the log tells how many records are inserted into the Amazon CloudSearch , we can observe close to </span><span style="font-family: Verdana, sans-serif;">247,681,520 documents(247+ million) are pumped.</span></li>
<li><span style="font-family: Verdana, sans-serif;">“Bytes Read” in the output tells what is size of data set which the JOB has read. We can observe</span><span style="font-family: Verdana, sans-serif;"> 322387978332 bytes which is equivalent to 300+ GB of index in the Amazon CloudSearch</span></li>
<li><span style="font-family: Verdana, sans-serif;">The entire pumping process took ~30 hours with 10 m1.large core nodes for us. We observed that increasing the number of Data loader EMR nodes or their capacity improves the upload speed drastically.</span></li>
</ul>
</div>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ0-weOkc2L-NCBPMhGCZ2SrdxLFRC5xeO9o9u_6jV__gGWceHoSK9C9her06W3ykhji621x8Hu6Rh7v-JBCcKUyx9ebvh7m31bCU3BfpIUl8AeNuGj-7vo1PvzkUmklGI2tWvuAU8rtBq/s1600/014-Amazon+CloudSearch+EMR+Data+Loader+Output+Console.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ0-weOkc2L-NCBPMhGCZ2SrdxLFRC5xeO9o9u_6jV__gGWceHoSK9C9her06W3ykhji621x8Hu6Rh7v-JBCcKUyx9ebvh7m31bCU3BfpIUl8AeNuGj-7vo1PvzkUmklGI2tWvuAU8rtBq/s1600/014-Amazon+CloudSearch+EMR+Data+Loader+Output+Console.jpg" height="364" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: Verdana, sans-serif;"><b>Step 13) Clean up :</b> </span><span style="font-family: Verdana, sans-serif;">Reset Replication Count to level of HA needed ideally 1-2 nodes.<span style="white-space: pre;"> </span></span><span style="font-family: Verdana, sans-serif;">Once the Job is completed, Revert back the Security Access Policies in Amazon cloud search.</span><span class="Apple-tab-span" style="font-family: Verdana, sans-serif; white-space: pre;"> </span><span style="font-family: Verdana, sans-serif;">Terminate the EMR Cluster and clean any leftover resources.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Step 14)</span><span class="Apple-tab-span" style="font-family: Verdana, sans-serif; white-space: pre;"> </span><span style="font-family: Verdana, sans-serif;">Analyzing the CloudSearch Dashboard :</span></b><br />
<span style="font-family: Verdana, sans-serif;">We observed that it takes some time for cloud search to reflect actual count of the indexed documents.</span></div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>After the pumping of 300 + GB index you can observe that currently 2 Amazon CloudSearch partitions ( shards) are used to distribute 247+ million documents with 100+ index fields. This is tremendous cost savings compared to A9 powered Amazon CloudSearch. </b>Amazon CloudSearch has automatically created shards based on the volume of data pumped in to the system. This is cool !!!, it reduces the maintenance headache of the infra admins. If the Amazon CloudSearch team can make this partition concept as configurable parameter in future it will be useful. </span></div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTRlWyqPHiQ6o06mcmWM7HGhVnHGRzCnUl0r-kHoPJyJE86-EEWbYxLBUGsCf5h2blIJtlP-RSAdB1OtG-I2a-K48iTM-TvOrO0TmaC3OIXWggp9rVk3chnteXpcLJCU2aqIq6BpRD9lws/s1600/15-Amazon+CloudSearch+Partitions+and+Number+of+Records+p15.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTRlWyqPHiQ6o06mcmWM7HGhVnHGRzCnUl0r-kHoPJyJE86-EEWbYxLBUGsCf5h2blIJtlP-RSAdB1OtG-I2a-K48iTM-TvOrO0TmaC3OIXWggp9rVk3chnteXpcLJCU2aqIq6BpRD9lws/s1600/15-Amazon+CloudSearch+Partitions+and+Number+of+Records+p15.jpg" height="180" width="640" /></a></div>
<br /></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Step 15) Executing a Sample Search queries: </b>We are executing a some sample product search queries on the "bigdatasearch" domain to check whether everything is fine. Distributed query was fired and Results came Sub Second from one of the partitions.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-2DDAfsxjhuXByl7F-k75gJlbwQlB3pctiTZcI_x4FtkLGw56t_KLFmv7jkHYMW1eor0ar5CodrgKCD45fv5l127KUO5uDDvuepuY7RwP0OGH3V0mNdpveNlhUNtIrt5O9RQiDlTW0JAG/s1600/16-Sample+Search+of+data+loaded+on+Amazon+CloudSearch++p16.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-2DDAfsxjhuXByl7F-k75gJlbwQlB3pctiTZcI_x4FtkLGw56t_KLFmv7jkHYMW1eor0ar5CodrgKCD45fv5l127KUO5uDDvuepuY7RwP0OGH3V0mNdpveNlhUNtIrt5O9RQiDlTW0JAG/s1600/16-Sample+Search+of+data+loaded+on+Amazon+CloudSearch++p16.jpg" height="332" width="640" /></a></div>
<br />
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div>
</div>
</div>
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 0px;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div style="margin: 0px;">
<span style="font-family: Verdana, sans-serif;">In short, It is cost effective compared to old A9 powered CloudSearch, Automated scaling of replication counts for request scalability, automated scaling of partitions for data scalability relieves the infra admin headaches, strong apache Solr pedigree and its long list of feature additions in coming months will make it more interesting. </span></div>
<div style="margin: 0px;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div style="margin: 0px;">
<span style="font-family: Verdana, sans-serif;">After working with this service few weeks, we feel it is going to become the major search service on AWS in coming years, giving tough fight for Apache Solr and ElastiSearch deployments on EC2. </span></div>
<div style="margin: 0px;">
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">This article was co authored with Ankit @8Kmiles.</span></div>
<div style="margin: 0px;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</div>
</div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-8540321622157530862014-02-06T09:40:00.001-08:002014-02-06T09:47:37.458-08:00 ElastiCache Redis : How to Backup & Launch from RDB Snapshots ?<span style="font-family: Verdana, sans-serif;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><br />
<span style="font-family: Verdana, sans-serif;">ElastiCache Redis is a powerful KV store offering both in-memory and persistent store for Key Values.When we persist data in ElastiCache Redis, it is obvious that we need to constantly backup Redis to recover the data during failures. By default there is no option to easily create the snapshot from the running ElastiCache Redis cluster. We need to install Redis on Self Managed EC2 node and point it to ElastiCache Redis Master for the snapshot process. </span><span style="font-family: Verdana, sans-serif;">This post explains about the step by step method for Redis cluster snapshot creation.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Redis Snapshot Architecture :</b></span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">ElastiCache cluster with master redis node of size cache.m1.large in AZ1 and </span><span style="font-family: Verdana, sans-serif;">Replication group with one read replica node of size cache.m1.large in AZ2</span></li>
<li><span style="font-family: Verdana, sans-serif;">Self managed Redis installed on EC2 for snapshots on AZ1</span></li>
</ul>
<br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEic0C4F2i2T35CAeFJlQZGLof4sjtuIg6ef2bVxaWkYVVAb0oqktjO4UVAO4a0KoIUo0Twc5vGjDWQStu_4yqts2qdA69smftxIZ3H-nYNU7jVIGrh43HBrp6-lM4glxBb8_oprLBa3dCSS/s1600/How+to+take+Snapshot+from+ElastiCache+Redis.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEic0C4F2i2T35CAeFJlQZGLof4sjtuIg6ef2bVxaWkYVVAb0oqktjO4UVAO4a0KoIUo0Twc5vGjDWQStu_4yqts2qdA69smftxIZ3H-nYNU7jVIGrh43HBrp6-lM4glxBb8_oprLBa3dCSS/s1600/How+to+take+Snapshot+from+ElastiCache+Redis.jpg" height="494" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<b style="font-family: Verdana, sans-serif; font-size: x-large;"><u>Stage 1: Creating a Snapshot for ElastiCache redis Persistent store.</u></b><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 1)</b> Launch an EC2 instance on the same Availability zone where the ElastiCache Redis master node runs. </span><br />
<span style="font-family: Verdana, sans-serif;">If the Redis Snapshot EC2 node is temporary (or) replication/Backup performance is very critical then keep the Redis Snapshot node in the same AZ as the ElastiCache Redis Master Node. </span><br />
<span style="font-family: Verdana, sans-serif;">If Redis Snapshot node is going to permanently run and high availability of the backup mechanism is very critical, run this on different AZ from master redis node. </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 2)</b>Download and install the Redis software on your newly launched EC2 instance. Make sure the Redis snapshot ec2 version and ElastiCache Redis Master version are compatible.</span><br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="background: black; border-collapse: collapse; border: none; mso-background-themecolor: text1; mso-border-alt: solid windowtext 1.0pt; mso-border-insideh: 1.0pt solid windowtext; mso-border-insidev: 1.0pt solid windowtext; mso-padding-alt: 0in 0in 0in 0in; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">$ wget
http://redis.googlecode.com/files/redis-2.6.4.tar.gz</span><span style="color: white; font-family: "Times New Roman","serif"; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">$ tar xzf redis-2.6.4.tar.gz</span><span style="color: white; font-family: "Times New Roman","serif"; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">$ cd redis-2.6.4</span><span style="color: white; font-family: "Times New Roman","serif"; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; mso-bidi-font-family: "Times New Roman"; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">$ make</span><span style="font-family: "Times New Roman","serif"; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<br />
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><b>Step 3)</b>Start the Redis Server on the newly launched EC2 instance. It will be listening on the port 6379</span></div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="background: black; border-collapse: collapse; border: none; mso-background-themecolor: text1; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div style="margin-bottom: .0001pt; margin: 0in;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">#
src/redis-server </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1513] 10 Jan 10:25:45.083 * Max number of open
files set to 10032 </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1513] 10 Jan 10:25:45.085 # Server started,
Redis version 2.6.4</span><span style="color: white; font-family: "Verdana","sans-serif"; font-size: 13.5pt; mso-themecolor: background1;"> </span><span style="color: white; mso-themecolor: background1;"><o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<br />
<div class="MsoNormal">
<br /></div>
<span style="font-family: Verdana, sans-serif;"><b>Step 4)</b>Open another terminal window and connect locally to the Redis Snapshot EC2 instance</span><span style="font-family: 'Times New Roman', serif; font-size: 12pt;"> </span><br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="background: black; border-collapse: collapse; border: none; mso-background-themecolor: text1; mso-border-alt: solid windowtext 1.0pt; mso-border-insideh: 1.0pt solid windowtext; mso-border-insidev: 1.0pt solid windowtext; mso-padding-alt: 0in 0in 0in 0in; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div style="margin-bottom: .0001pt; margin: 0in;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">#
src/redis-cli -h localhost -p 6379 </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">redis localhost:6379> </span><span style="font-size: 11.0pt;"><o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<span style="font-family: Verdana, sans-serif;"><b>Step 5)</b>To synchronize the data between the local Redis Snapshot EC2 instance and the ElastiCache Redis master use the following command in the Redis Snapshot EC2.</span><span style="font-family: 'Times New Roman', serif; font-size: 12pt;"> </span><br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="background: black; border-collapse: collapse; border: none; mso-background-themecolor: text1; mso-border-alt: solid windowtext 1.0pt; mso-border-insideh: 1.0pt solid windowtext; mso-border-insidev: 1.0pt solid windowtext; mso-padding-alt: 0in 0in 0in 0in; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div style="margin-bottom: .0001pt; margin: 0in;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">SLAVEOF
redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com 6379</span><span style="font-size: 11.0pt;"><o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;">where "redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com 6379" is the endpoint of the ElastiCache Master Redis node.</span></div>
<span style="font-family: Verdana, sans-serif;">Bringing a Additional Redis EC2 node for snapshots into the infrastructure is not ideal. It adds extra cost </span><span style="font-family: Verdana, sans-serif;">and manual labor effort to setup, automate and manage this Redis Snapshot EC2. </span><span style="font-family: Verdana, sans-serif;">I hope AWS ElastiCache team will release a mechanism where we can easily take the snapshots from the ElastiCache Cluster itself. </span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><b>Note: Before this step ensure that security group permissions are opened between the Redis Snapshot EC2 and ElastiCache Redis Cluster.</b></span></div>
<br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 6)</b>Once the synchronization is done between Redis Snapshot node and ElastiCache Redis Master, you can take snapshot using the command BGSAVE or SAVE in the snapshot EC2 Node. The snapshot file named <b>"dump.rdb"</b> is created in the disk.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 7)</b> You can optionally detach it from the ElastiCache cluster by using the command on redis-cli “SLAVEOF NO ONE”. This step is recommended if your redis cluster size is few GB's and you are backing up persistent data once/4 times a day, as you can save costs by not unnecessarily running your Redis snapshot EC2 instance for 24 hours and automatically bring it up whenever needed.</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">We conducted a small test to check whether all the above steps are successfully reflected. </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">We pumped the ElastiCache Redis cluster with 1.8GB of Data spanning few hundred millions of KV data with each record in few <KB size. </span><span style="font-family: Verdana, sans-serif;">We observed the time taken to sync the data is about ~5 minutes on m1.large capacity in same AZ deployment. </span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">If your Redis is heavily used it is better to use Higher EC2 capacities for the ElastiCache Cluster and Snapshot node. Bigger the nodes, better the NW/IO and lesser the replication lag. </span></li>
<li><span style="font-family: Verdana, sans-serif;">If you are running a small cluster with few GB's then creating the Redis Snapshot Node on demand will save costs. If your cluster is big, data is critical and backup frequencies are less, then run Snapshot EC2 node continuously for better performance.</span></li>
</ul>
<br />
<br />
<span style="font-family: Verdana, sans-serif;">The replication action with timing is detailed below:</span><br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="background: black; border-collapse: collapse; border: none; mso-background-themecolor: text1; mso-border-alt: solid windowtext 1.0pt; mso-border-insideh: 1.0pt solid windowtext; mso-border-insidev: 1.0pt solid windowtext; mso-padding-alt: 0in 0in 0in 0in; mso-yfti-tbllook: 1184;"><tbody>
<tr><td style="border: solid windowtext 1.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div style="margin-bottom: .0001pt; margin: 0in;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595]
10 Jan 11:03:06.676 # Server started, Redis version 2.6.4 </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:03:06.676 * The server is now
ready to accept connections on port 6379 </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:03:18.018 * SLAVE OF redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com:6379
enabled (user request) </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:03:18.779 * Connecting to
MASTER... </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:03:18.975 * MASTER <->
SLAVE sync started </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:03:18.975 * Non blocking connect
for SYNC fired the event. </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:03:18.977 * Master replied to
PING, replication can continue... </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:06:03.949 * MASTER <->
SLAVE sync: receiving 1943998490 bytes from master </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:08:13.348 * MASTER <->
SLAVE sync: Loading DB in memory </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:09:37.430 * MASTER <->
SLAVE sync: Finished with success </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:09:37.441 * 100 changes in 300
seconds. Saving... </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:09:39.970 * Background saving
started by pid 1635 </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:16:44.860 # User requested shutdown... </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:16:44.860 * Saving the final RDB
snapshot before exiting. </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:19:10.270 * DB saved on
disk </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">[1595] 10 Jan 11:19:10.270 # Redis is now ready
to exit, bye bye... </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<br />
<span style="font-family: Verdana, sans-serif;">Test whether the same key values are synced between Master Redis Node in ElastiCache Cluster and Redis Snapshot EC2. The below steps illustrate the same:</span><span style="font-family: 'Times New Roman', serif; font-size: 12pt;"> </span><br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="background: black; border-collapse: collapse; border: none; mso-background-themecolor: text1; mso-border-alt: solid windowtext 1.0pt; mso-border-insideh: 1.0pt solid windowtext; mso-border-insidev: 1.0pt solid windowtext; mso-padding-alt: 0in 0in 0in 0in; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div style="margin-bottom: .0001pt; margin: 0in;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">#
src/redis-cli -h redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com -p
6379 </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">redis
redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com:6379> get 100sent </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">"1001234567890123456789012345678901234567890abcdefghijklmopqrstuvwxyz"</span><br />
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;"><br /></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">redis
redissnapshot.qcdze2.0001.usw2.cache.amazonaws.com:6379> get 10sent </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">"101234567890123456789012345678901234567890abcdefghijklmopqrstuvwxyz"</span></div>
</td>
</tr>
</tbody></table>
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<span style="font-family: Verdana, sans-serif;"><b>Important Observation:</b></span><br />
<span style="font-family: Verdana, sans-serif;">Always sync the Redis Snapshot EC2 node with the ElastiCache Redis master node and not with the read slave in the redis replication group. Following error will be displayed when trying to sync with the slave.</span><span style="font-family: 'Times New Roman', serif; font-size: 12pt;"> </span><br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="background: black; border-collapse: collapse; border: none; mso-background-themecolor: text1; mso-border-alt: solid windowtext 1.0pt; mso-border-insideh: 1.0pt solid windowtext; mso-border-insidev: 1.0pt solid windowtext; mso-padding-alt: 0in 0in 0in 0in; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div style="margin-bottom: .0001pt; margin: 0in;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">#Connecting
to MASTER...</span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">#MASTER <-> SLAVE sync started</span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">#Non blocking connect for SYNC fired the event.</span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">#Master replied to PING, replication can
continue...</span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;"># MASTER aborted replication with an error: ERR
Can't SYNC with a slave</span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;">We tried the above approach to check whether we can reduce the workload on Redis Master node and take the backups from Slave redis Nodes. As AWS as rightly named it is " Redis Read Replica Nodes" and you cannot use it for snapshot replication. </span></div>
<br />
<div>
<b style="font-family: Verdana, sans-serif; font-size: x-large;"><u>Stage 2: Launching ElastiCache Redis Cluster from the existing Snapshot taken above.</u></b></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 1) </b>Import the data to the S3 bucket (redisaws) and assign open/download permission for the email id aws-scs-s3-readonly@amazon.com.<span style="color: red;"> </span></span><span style="font-family: Verdana, sans-serif;">Note : </span><span style="font-family: Verdana, sans-serif;">The aws-scs-s3-readonly account is used exclusively for customers uploading Redis snapshot data from Amazon S3.</span><br />
<br />
<span style="font-family: Verdana, sans-serif;"><b>Step 2)</b> Launch a new ElastiCache Redis cluster from the "dump.rdb" snapshot taken in the earlier stage 1 - step 6 from Redis Snapshot node. Lets understand what is RDB in detail.</span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">RDB is a very compact single file point in time representation of your Redis data and are perfect for backups. Being a single compact file it can be transferred to alternate AZ and AWS regions quite fast and is ideal during outages/DR. RDB also allows faster restarts of Redis Master with big datasets in new ElastiCache Clusters nodes. Both these points helps us achieve better RPO and RTO during DR scenario using RDB. </span></li>
<li><span style="font-family: Verdana, sans-serif;">RDB needs to fork() often in order to persist on disk using a child process. Fork() can be time consuming if the dataset is big, and may result in Redis Master to stop serving clients for some millisecond or even for one second if the dataset is very big and the CPU performance is not great. Also s</span><span style="font-family: Verdana, sans-serif;">ince the Redis Snapshot instance in this architecture is not used for accepting reads/writes and it is primarily used only for replication + snapshot it will add again minimal load on the master node during replication. Factor the ElastiCache Master Redis node capacity taking both these parameters into consideration.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Note : RDB is NOT good if you need to minimize the chance of data loss in case Redis Master stops working. It is suggested to use RDB model with ElastiCache Redis Replication group cluster for better Availability and integrity.</span></li>
</ul>
<br />
<div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYnuVKAUl0YexRAt4aTBRapoi11SfOehypxCPna1HoTVD1yhBPeGDttr0IxB-4B3WVC1zpQoZ2X-mGtJukRpZSM00KNBLL0Z491po3cQ7lx6RqWNH3khT6glBEui8r647IgP3Kq9jVfNUW/s1600/How+to+launch+ElastiCache+Redis+from+Snapshot.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYnuVKAUl0YexRAt4aTBRapoi11SfOehypxCPna1HoTVD1yhBPeGDttr0IxB-4B3WVC1zpQoZ2X-mGtJukRpZSM00KNBLL0Z491po3cQ7lx6RqWNH3khT6glBEui8r647IgP3Kq9jVfNUW/s1600/How+to+launch+ElastiCache+Redis+from+Snapshot.jpg" height="640" width="558" /></span></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 3)</b> Once the ElastiCache cluster is launched. Connect to the cluster endpoint and check for the existence of the same key values tested in the dump.</span><br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="background: black; border-collapse: collapse; border: none; mso-background-themecolor: text1; mso-border-alt: solid windowtext 1.0pt; mso-border-insideh: 1.0pt solid windowtext; mso-border-insidev: 1.0pt solid windowtext; mso-padding-alt: 0in 0in 0in 0in; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.65in;" valign="top" width="638"><div style="margin-bottom: .0001pt; margin: 0in;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">src/redis-cli
-h rds.qcdze2.0001.usw2.cache.amazonaws.com -p 6379 </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">redis
rds.qcdze2.0001.usw2.cache.amazonaws.com:6379> get 10sent </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">"101234567890123456789012345678901234567890abcdefghijklmopqrstuvwxyz" </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<br /></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">redis
rds.qcdze2.0001.usw2.cache.amazonaws.com:6379> get 100sent </span><span style="color: white; font-size: 11.0pt; mso-themecolor: background1;"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt;">
<span style="color: white; font-family: "Verdana","sans-serif"; font-size: 11.0pt; mso-themecolor: background1;">"1001234567890123456789012345678901234567890abcdefghijklmopqrstuvwxyz" </span><span style="font-size: 11.0pt;"><o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<span style="font-family: Verdana, sans-serif;">You can notice from the above that the key values are same and the new ElastiCache node was created properly with original data.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<span style="font-family: Verdana, sans-serif;">This post was co authored with Senthil 8K Miles</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<br />Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com1tag:blogger.com,1999:blog-4836510665235566234.post-12988432071799451742014-01-31T03:25:00.003-08:002015-06-23T10:29:12.669-07:0025 Best Practice Tips for architecting your Amazon VPC<div>
<div>
<span style="font-family: Verdana, sans-serif;">According to me Amazon VPC is one of the most important feature introduced by AWS. We have been using AWS from 2008 and Amazon VPC from the day it was introduced and i strongly feel the customer adoption towards AWS cloud gained real momentum only after the introduction of VPC into the market. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Amazon VPC comes with lots of advantages over the limitations faced in Amazon Classic cloud like: </span><span style="font-family: Verdana, sans-serif;">Static private IP address , Elastic Network Interfaces : possible to bind multiple Elastic Network Interfaces to a single instance, Internal Elastic Load Balancers, Advanced Network Access Control ,Setup a secure bastion host , DHCP options , Predictable internal IP ranges , Moving NICs and internal IPs between instances, VPN connectivity, </span><span style="font-family: Verdana, sans-serif;">Heightened security etc. Each and everything is a interesting topic on its own and i will be discussing them in detail in future. </span></div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Today i am sharing some of our implementation experience on working with hundreds of Amazon VPC deployments as best practice tips for the AWS user community. You can apply some of the relevant ones in your existing VPC or use these points as part of your migration approach to Amazon VPC.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif; font-size: large;"><b>Please also look into 27 Best practices on AWS Security Groups to understand the relation with Amazon VPC. </b></span><br />
<span style="color: blue; font-family: Verdana, sans-serif; font-size: large;"><a href="http://harish11g.blogspot.com/2015/06/best-practices-tips-on-amazon-web-services-security-groups-aws-security-managed-services.html"><b>http://harish11g.blogspot.com/2015/06/best-practices-tips-on-amazon-web-services-security-groups-aws-security-managed-services.html</b></a></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<span style="font-family: Verdana, sans-serif;"><b>Practice 1) Get your Amazon VPC combination right: </b>Select the right Amazon VPC architecture first. You need to decide the right Amazon VPC & VPN setup combination based on your current and future requirements. It is tough to modify/re-design the Amazon VPC at later stage, so it is better to design it taking into consideration your NW and expansion needs for next ~2 years. Currently different types of Amazon VPC setups are available; Like Public facing VPC, Public and Private setup VPC, Amazon VPC with Public and Private Subnets and Hardware VPN Access, Amazon VPC with Private Subnets and Hardware VPN Access, Software based VPN access etc. Choose the one which you feel you will be in next 1-2 years.</span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 2)</b> <b>Choose your CIDR Blocks: </b>While designing your Amazon VPC, the CIDR block should be chosen in consideration with the number of IP addresses needed and whether we are going to establish connectivity with our data center. The allowed block size is between a /28 netmask and /16 netmask. Amazon VPC can have contain from 16 to 65536 IP addresses. Currently Amazon VPC once created can't be modified, so it is best to choose the CIDR block which has more IP addresses usually. Also when you design the Amazon VPC architecture to communicate with the on premise/data center ensure your CIDR range used in Amazon VPC does not overlaps or conflicts with the CIDR blocks in your On premise/Data center. Note: If you are using same CIDR blocks while configuring the customer gateway it may conflict.</span><br />
<span style="font-family: Verdana, sans-serif;">E.g., Your VPC CIDR block is 10.0.0.0/16 and if you have 10.0.25.0/24 subnet in a data center the communication from instances in VPC to data center will not happen since the subnet is the part of the VPC CIDR. In order to avoid these consequences it is good to have the IP ranges in different class. Example., Amazon VPC is in 10.0.0.0/16 and data center is in 172.16.0.0/24 series.</span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 3)</b> <b>Isolate according to your Use case: </b>Create separate Amazon VPC for Development , Staging and Production environment (or) Create one Amazon VPC with Separate Subnets/Security/isolated NW groups for Production , Staging and development. We have observed 60% of the customer preferring the second choice. You chose the right one according to your use case. </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 4)</b> <b>Securing Amazon VPC : </b>If you are running a machine critical workload demanding complex security needs you can secure the Amazon VPC like your on-premise data center or more sometimes. Some of the tips to secure your VPC are: </span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">Secure your Amazon VPC using Firewall virtual appliance, Web application firewall available from Amazon Web Services Marketplace. You can use check point, Sophos etc for this</span></li>
<li><span style="font-family: Verdana, sans-serif;">You can configure Intrusion Prevention or Intrusion Detection virtual appliances and secure the protocols and take preventive/corrective actions in your VPC</span></li>
<li><span style="font-family: Verdana, sans-serif;">Configure VM encryption tools which encrypts your root and additional EBS volumes. The Key can be stored inside AWS (or) in your Data center outside Amazon Web Services depending on your compliance needs. </span><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/understanding-Amazon-Elastic-Block-Store-Securing-EBS-TrendMicro-SecureCloud.html">http://harish11g.blogspot.in/2013/04/understanding-Amazon-Elastic-Block-Store-Securing-EBS-TrendMicro-SecureCloud.html</a></span></li>
<li><span style="font-family: Verdana, sans-serif;">Configure Privileged Identity access management solutions on your Amazon VPC to monitor and audit the access of Administrators of your VPC.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Enable the cloud trail to audit in the VPC environments ACL policy's. Enable cloud trail : </span><a href="http://harish11g.blogspot.in/2014/01/Integrating-AWS-CloudTrail-with-Splunk-for-managed-services-monitoring-audit-compliance.html" style="font-family: Verdana, sans-serif;">http://harish11g.blogspot.in/2014/01/Integrating-AWS-CloudTrail-with-Splunk-for-managed-services-monitoring-audit-compliance.html</a></li>
<li><span style="font-family: Verdana, sans-serif;">Apply anti virus for cleansing specific EC2 instances inside VPC. Trend micro has very good product for this.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Configure Site to Site VPN for securely transferring information between Amazon VPC in different regions or between Amazon VPC to your On premise Data center</span></li>
<li><span style="font-family: Verdana, sans-serif;">Follow the Security Groups and NW ACL's best practices listed below</span></li>
</ul>
</div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 5) Understand Amazon VPC Limits:</b> Always design the VPC subnets in consideration with the expansion in the future. Also understand the Amazon VPC's limits before using the same. AWS has various limitations on the VPC components like Rules per security group, No of route tables and Subnets etc. Some of them may be increased after providing the request to the Amazon support team while few components cannot be increased. Ensure the limitations are not affecting your overall design. Refer URL:</span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html">http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html</a></span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 6)</b> <b>IAM your Amazon VPC: </b>When you are going to assign people to maintain your Amazon VPC you can create Amazon IAM account with the fine grained permissions (or) use Sophisticated Privileged identity Management solutions available on AWS marketplace to IAM your VPC.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 7) Disaster Recovery or Geo Distributed Amazon VPC Setup</b> : When you are designing a Disaster Recovery Setup plan using VPC or expanding to another Amazon VPC region you can follow these simple rules. Create your </span><span style="font-family: Verdana, sans-serif;">Production site VPC CIDR : 10.0.0.0/16 and your DR region VPC CIDR: 172.16.0.0/16. Make sure they do not conflict with on premises subnet CIDR block in event both needs to be integrated to on premise DC as well. After CIDR blocks creation , setup a VPC tunnel between regions and to your on premise DC. This will help to replicate your data using private IP’s. </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 8) Use security groups and Network ACLs wisely:</b> It is advisable to use security groups over Network ACLs inside Amazon VPC wherever applicable for better control. Security groups are applicable on EC2 instance level while network ACL is applicable on Subnet level. Security groups are used for White list mostly. To blacklist IPs, one can use Network ACLs.</span></div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 9) Tier your Security Groups :</b> Create different security groups for different tiers of your infrastructure architecture inside your VPC. If you have Web, App, DB tiers create different security group for each of them. Creating tier wise security groups will increase the infrastructure security inside Amazon VPC. EC2 instances in each tier can talk only on application specified ports and not at all ports. If you create Amazon VPC security groups for each and every tier/service separately it will be easier to open a port to a particular service. Don't use same security group for multiple tiers of instances, this is a bad practice. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Example: Open ports for security group instead of IP ranges : For example : People have tendency to open for port 8080 to 10.10.0.0/24 (web layer) range. Instead of that, open port 8080 to web-security-group. This will make sure only web security group instances will be able to contact on port 8080. If someone launches NAT instance with NAT-Security-Group in 10.10.0.0/24, he won't be able to contact on port 8080 as it allows access from only web security group. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 10 )</b> <b>Standardize your Security Group Naming conventions : </b>Following a security group naming conventions inside Amazon VPC will improve operations/management for large scale deployments inside VPC. It also avoids manual errors, leaks and saves cost and time overall.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;">For example: Simple ones like Prod_DMZ_Web_SG or Dev_MGMT_Utility_SG (or) complex coded ones for large scale deployments like</span></div>
<div>
<span style="font-family: Verdana, sans-serif;">USVA5LXWEBP001- US East Virginia AZ 5 Linux Web Server Production 001</span></div>
<div>
<span style="font-family: Verdana, sans-serif;">This helps in better management of security groups. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 11) ELB on Amazon VPC: </b>When using Amazon ELB for Web Applications, put all other EC2 instances( Tiers like App,cache,DB,BG etc) in private subnets as much possible. Unless there is a specific requirement where instances need outside world access and EIP attached, put all instances in private subnet only. Only ELBs should be provisioned in Public Subnet as secure practice in Amazon VPC environment.</span></div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 12) Control your outgoing traffic in Amazon VPC:</b> If you are looking for better security, for the traffic going to internet gateway use Software's like Squid or Sophos to restrict the ports,URL,Domains etc so that all traffic go through the proxy tier controlled and it also gets logged. Using these proxy/security systems we can also restrict the unwanted ports, by doing so, if there is any security compromise to the application running inside Amazon VPC they can be detected by auditing the restricted connections captured from the logs. This helps in corrective security measure. </span></div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 13) Plan your NAT Instance Type:</b> Whenever your Application EC2 instances residing inside private subnet of Amazon VPC are making Web Service/HTTP/S3/SQS calls they go through NAT instance. If you have designed Auto scaling for your application tier and there are chances ten's of app EC2 instances are going to make lots of web calls concurrently, NAT instance will become a performance bottleneck at this juncture. Size your NAT instance capacity depending upon application needs for avoiding performance bottlenecks. Using the NAT instances provides us with advantages of saving cost of Elastic IP and provides extra security by not exposing the instances to outside world for accessing the internet. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 14) Spread your NAT instance with Multiple Subnets: </b>What if you have hundreds of EC2 instances inside your Amazon VPC and they are making lots of heavy web service/HTTP calls concurrently. A single NAT instance with even largest EC2 size cannot handle that bandwidth sometimes and may become performance bottleneck. In Such scenarios, span your EC2 across multiple subnets and create NAT's for each subnet. This way you can spread your out going bandwidth and improve the performance in your VPC based deployments.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice 15) Use EIP when needed:</b> At times you may need to keep a part of your application services to be kept in Public subnet for external communication. It is recommended practice to associate them with Amazon Elastic IP and white list these IP address in the target services used by them </span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 16) NAT instance practices :</b> If needed, enable Multi factor authentication on NAT instance. SSH and RDP ports are open only on sources and destination IP's, not global network (0.0.0.0/0). SSH / RDP ports are opened only on static exit IP's not dynamic exit IP's. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Multi factor authentication ref link : Linux: <a href="http://www.howtogeek.com/121650/how-to-secure-ssh-with-google-authenticators-two-factor-authentication/">http://www.howtogeek.com/121650/how-to-secure-ssh-with-google-authenticators-two-factor-authentication/</a> </span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Windows : <a href="http://www.rohos.com/2013/02/google-authenticator-windows-login/">http://www.rohos.com/2013/02/google-authenticator-windows-login/</a></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice</b><b> 17) Plan your Tunnel between On-Premise DC to Amazon VPC: </b></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Select the right mechanism to connect your on premises DC to Amazon VPC. This will help you to connect the EC2 instance via private IP's in a secure manner. </span></div>
<div>
<ul>
<li><span style="font-family: Verdana, sans-serif;">Option 1: Secure IPSec tunnel to connect a corporate network with Amazon VPC (</span><a href="http://aws.amazon.com/articles/8800869755706543" style="font-family: Verdana, sans-serif;">http://aws.amazon.com/articles/8800869755706543</a><span style="font-family: Verdana, sans-serif;">) </span></li>
<li><span style="font-family: Verdana, sans-serif;">Option 2 : Secure communication between sites using the AWS VPN CloudHub (</span><a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPN_CloudHub.html" style="font-family: Verdana, sans-serif;">http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPN_CloudHub.html</a><span style="font-family: Verdana, sans-serif;">)</span></li>
<li><span style="font-family: Verdana, sans-serif;">Option 3: Use Direct connect between Amazon VPC and on premise when you have lots of data to be transferred with reduced latency (or) you have spread your mission critical workloads across cloud and on premise. Example: Oracle RAC in your DC and Web/App tier in your Amazon VPC. </span><span style="font-family: Verdana, sans-serif;">Contact us if you need help on setting up direct connect between Amazon VPC and DC.</span></li>
</ul>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice 18) Always span your Amazon VPC across multiple subnets in Multiple Availability zones inside a Region</b>. This helps is architecting high availability inside your Amazon VPC properly. Example: Classification of the VPC subnet : WEB Tier Subnet : 10.0.10.0/24 in Az1 and 10.0.11.0/24 in Az2, Application Tier Subnet : 10.0.12.0/24 and 10.0.13.0/24, DB Tier Subnet : 10.0.14.0/24 and 10.0.15.0/24, Cache Tier Subnet : 10.0.16.0/24 and 10.0.17.0/24 etc</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice 19) </b>Good security practice is that to have only public subnet with route table which carries route to internet gateway. Apply this wherever applicable.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice 20) Keep your Data closer :</b> For small scale deployments in AWS where cost is critical than high availability, It is better to keep the Web/App in same availability zone as of ElastiCache , RDS etc inside your Amazon VPC. Design your subnets accordingly to suit this. This is not a recommended architecture for applications demanding High Availability.</span></div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice 21) Allow and Deny Network ACL :</b> Create Internet outbound allow and deny network ACL in your VPC. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;">First network ACL: Allow all the HTTP and HTTPS outbound traffic on public internet facing subnet. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Second network ACL: Deny all the HTTP/HTTPS traffic. Allow all the traffic to Squid proxy server or any virtual appliance. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><a href="http://techlib.barracuda.com/display/BNGv54/How+to+Deploy+the+Barracuda+NG+Firewall+in+an+Amazon+Virtual+Private+Cloud">http://techlib.barracuda.com/display/BNGv54/How+to+Deploy+the+Barracuda+NG+Firewall+in+an+Amazon+Virtual+Private+Cloud </a></span></div>
</div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice 22 ) Restricting Network ACL : </b>Block all the inbound and outbound ports. Only allow application request ports. These are stateless traffic filters that apply to all traffic inbound or outbound from a Subnet within VPC. AWS recommended Outbound rules : <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html">http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html</a></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Practice 23)</b> Create route tables only when needed and use the Associations option to map subnets to the route table in your Amazon VPC</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Practice 24) Use Amazon VPC Peering <span style="color: red;">(new)</span> : </b>Amazon Web Services has introduced VPC peering feature which is quite useful one. AWS VPC peering connection is a networking connection between two Amazon VPCs that enables you to route traffic between them using private IP addresses. Currently it can be in same AWS region, Instances in either VPC can communicate with each other as if they are within the same network. Since </span><span style="font-family: Verdana, sans-serif;">AWS uses the existing infrastructure of a VPC to create a VPC peering connection; it is neither a gateway nor a VPN connection, and does not rely on a separate piece of physical hardware (which essentially means there is no single point of failure for communication or a bandwidth bottleneck).</span><br />
<div>
<span style="font-family: Verdana, sans-serif;">We have seen it is useful in following scenarios :</span></div>
<div>
<ol>
<li><span style="font-family: Verdana, sans-serif;">Large Enterprises usually run Multiple Amazon VPC in single region and some of their applications are so interconnected that they may need to access them privately + securely inside AWS. Example Active Directory, Exchange, Common business services will be usually interconnected.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Large Enterprise have different AWS accounts for different business units/teams/departments , at times systems deployed by some business units in different AWS accounts need to be shared or need to consume a shared resource privately. Example: CRM , HRMS ,File Sharing etc can be internal and shared. In such scenarios VPC peering comes very useful.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Customer can peer their VPC with their core suppliers to have tighter integrated access of their systems.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Companies offering Infra/Application Managed Services on AWS can now safely peer into customer Amazon VPC and provide monitoring and management of AWS resources.</span></li>
</ol>
</div>
<b style="font-family: Verdana, sans-serif;">Practice 25) Use Amazon VPC: </b><span style="font-family: Verdana, sans-serif;">It is highly recommended that migrate all your new workloads inside Amazon VPC rather than Amazon Classic Cloud. I also strongly recommend to migrate your existing workloads from Amazon Classic cloud to Amazon VPC in phases or one shot which ever is feasible. In addition to the benefits of the VPC that is detailed in the start of the article, AWS has started introducing lots of features which are compatible only inside VPC and in the AWS marketplace as well there are lots of products which are compatible only with Amazon VPC. So make sure you leverage this strength of VPC. If you require any help for this migration please contact me. </span></div>
</div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>readers feel free to suggest more.. I will link relevant ones in this article</b></span></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com13tag:blogger.com,1999:blog-4836510665235566234.post-40084670223586303572014-01-28T09:29:00.001-08:002014-01-28T09:46:23.412-08:00451 Research Report: 8KMiles crosses the chasm in cloud-based identity federation<b><span style="font-family: Verdana, sans-serif;">Analyst: Wendy Nather 22 Nov, 2013</span></b><br />
<b><span style="font-family: Verdana, sans-serif;"><br /></span></b><span style="font-family: Verdana, sans-serif;"><b>Original Report URL from 451 Research website : <a href="https://451research.com/report-short?entityId=79384&tmpl=print">https://451research.com/report-short?entityId=79384</a></b></span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Full Report is published down...</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">8KMiles has been heavily invested in cloud integration. As one of Amazon Web Services' Premier Consulting Partners for 2013, it has helped customers stand up everything from Amazon's Elastic Block Store to its S3 and Relational Database services. So it made sense to continue to add </span><span style="font-family: Verdana, sans-serif;">cloud integration services in the identity and access management (IAM) space. To this end, the company acquired Sunnyvale, California-based FuGen Solutions in May to obtain its Cloud Identity Broker and Multi-Domain Identity Services Platform.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif; font-size: large;">The 451 Take</span></b><br />
<span style="font-family: Verdana, sans-serif;">A combination of design and operations support helps 8KMiles, and its subsidiary FuGen makes on-ramping of federated identity partners easier, particularly for enterprises that don't have the infrastructure or expertise to figure it out themselves. A migration opportunity can become a hosting opportunity, while a hosting opportunity could turn into the kind of identity and attribute exchange that is still needed. Other efforts are </span><span style="font-family: Verdana, sans-serif;">underway to build such an exchange, but 8KMiles and FuGen could get out ahead of it – although it might help if they settled on one company name to promote the unity they're offering.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Context</span></b><br />
<span style="font-family: Verdana, sans-serif;">Did identity federation get any easier when the execution venue moved from legacy systems to the cloud? Actually, that's a trick question, because most of it hasn't moved – it's just been stretched. Even without the dynamism and scale requirements of the cloud, an enterprise's federation efforts with its partners suffer from complexity that many organizations aren't equipped to handle.</span><br />
<span style="font-family: Verdana, sans-serif;">There are many types of federation, and only some of them are binary: that is, one organization completely trusts the other one, so that it accepts any identity offered. A common example is federation between a health insurance provider and a partner that provides pharmacy benefits: there can be a</span><br />
<span style="font-family: Verdana, sans-serif;">one-to-one acceptance because it's the same business case (benefits for an insured client) and the same level of security risk. Because it's the same business case, both sides can validate the user in the same way and no additional validation is needed. A user can be passed through single sign-on from one site to another in a fairly seamless fashion.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">However, not all federation is binary. Take the example of a state education agency: it has thousands of school district employees that need to use the agency's applications. The agency would like to have the districts set up access for those users, but it is still legally on the hook to approve every </span><span style="font-family: Verdana, sans-serif;">access. This means that the agency has to rely on some assertions by the district, but must take an additional step of its own for validation and approval before it can fully accept that user into its systems. These validation workflows often use attributes of the user's identity: whether the user is an employee of the district (which only the district is authoritative about), which roles the user is assigned (which might be determined by the agency), or whether the user is also a member of a different organization (such as working for a second district).</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Attributes may sound complicated, and the business rules behind them can be. But an attribute is really the reason why you're allowing access to that user. You're allowing access because the insurance provider says this is a registered subscriber; you're allowing it because the Department of Motor Vehicles (DMV) asserts that this is a licensed driver; you're allowing it because the user is a registered PayPal customer. And you can only rely on that attribute when it comes from the right authoritative party: only PayPal can say with certainty who its current customers are. </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The ecosystem of attributes has yet to be addressed in a coherent way. Many websites and applications will be happy to accept the credentials of a Facebook user, because they only care that someone at Facebook (presumably) validated the user account. That's all the validation they need. But that's not enough for many other organizations, especially where legal and regulatory issues are on the line. But if you could get all these authoritative parties in one place...</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">This is where 8KMiles and FuGen come in.</span><br />
<span style="font-family: Verdana, sans-serif;">Founded in 2007, 8KMiles is led by Suresh Venkatachari, its chairman and CEO, who also founded consulting firm SolutionNET. The company has 140 employees among its locations in California, Virginia, Canada and India. In May, 8KMiles acquired FuGen Solutions for $7.5m, with the target </span><span style="font-family: Verdana, sans-serif;">becoming a subsidiary.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Products and services</span></b><br />
<span style="font-family: Verdana, sans-serif;">8KMiles offers both consulting services (cloud migration, engineering and application development) and frameworks for assembling secure cloud systems. The company provides a turnkey architecture for implementing a secure private cloud, including firewall and DDoS protection services, secure remote access, system administrator access and monitoring, and disk encryption. This can be deployed either as an Amazon virtual private cloud or in an organization's own datacenter. 8KMiles similarly offers a secure enterprise collaboration implementation that combines Alfresco's content management and Amazon's RDS. An AWS Direct Connect package contains both design and management of the network, points of</span><br />
<span style="font-family: Verdana, sans-serif;">presence, and security.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">When 8KMiles bought FuGen, it obtained both a cloud identity brokerage and the target's Multi-Domain Identity Services Platform (MISP). The platform supports the partner onboarding and federation management activities, as well as what the vendor calls last-mile single-sign-on integration to a centralized hub for smaller customers that don't have legacy IAM systems to connect, or who don't have the expertise to put everything together. The platform is vendor-agnostic in that it can be used with any IAM provider's systems to connect and federate partners. The authentication protocols supported include SAML 1.1, SAML 2.0, WS-Federation, WS-Trust, OpenID and OAuth. MISP comes with rules-based validation and reporting, criteria certification, monitoring and logging, and storage of scenarios, data messages, templates and certification reports.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">One of the strengths of the broker and platform offerings is that FuGen and 8KMiles staff can duplicate the customer's complex federation requirements in their virtualized environment. The vendor can build the hub and test all of the integrations with the partners' systems in a lab setting. Once it's been assembled and shown to work properly, the company can walk the customer through implementing the working version on its own </span><span style="font-family: Verdana, sans-serif;">systems, providing instructions down to the level of the configuration file changes. In cases where the customer does not have specialized IAM expertise or a test network, FuGen can provide both.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">These services are available for community providers, SaaS application firms, identity and attribute vendors, and many others. FuGen's customers range from one of the largest financial services institutions to media providers, large IT suppliers and defense contractors (Amazon AWS customers use FuGen's federated identity features). </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The idea of creating a vendor-agnostic federation space is a good one – as the number of partners grows with which FuGen has already built integrations, the onboarding for future customers goes more quickly. For example, if FuGen has already done the hard work of figuring out connectors for a large payment provider that happens to use Oracle for an IAM system and Ping Identity for cloud-based SSO, then any other partners that want to federate with that large payment provider using the same products will have most of the work already done. The network effect comes into play here: the more partners FuGen integrates, the stronger its offerings grow as a cloud-based ID federation service.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">For the reasons described above, many enterprises end up relying on a varied set of IDs and attributes, all coming from different partners. Building a central ID and attribute exchange could speed federation projects for government, healthcare, finance and other verticals if FuGen can pre-integrate those providers. When businesses can join a virtual marketplace where they can get the attributes they need from their state DMV, PayPal and business process outsourcer, and all of the integration work is done for them, then the community has a good chance of growth. Many identity and attribute exchange projects are already underway (and FuGen is already part of some of these open initiatives) – the one advantage is that the </span><span style="font-family: Verdana, sans-serif;">company helps facilitate the plumbing, not just the framework. Also, this isn't just about the cloud: enterprises can still federate with one another using their own systems, with FuGen's services to set it up. The one hitch is that this is a potential that hasn't been fully realized. 8KMiles and FuGen would have to figure out how to charge for this service, since charging by ID or partner account might be too dynamic to support a licensing structure. (This isn't to say that a cloud provider can't charge dynamically, it's just that determining how many IDs are in use at any given time is a tricky proposition.) The vendor could charge an onboarding project fee, but services after that – such as monitoring, support, troubleshooting and integration tweaks – would need a different incremental pricing structure. If a large provider is hooked into the hub, and new partners join it, does the provider get charged more, or just the new partners? Identity and attribute management are both still developing areas of technology, and with the cloud as a delivery method, many aspects have to be reconsidered.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Competition</span></b><br />
<span style="font-family: Verdana, sans-serif;">The term 'identity broker' is unintentionally confusing, since it is most often used to describe technology that helps intermediate an enterprise's portfolio of ID stores and services, usually to provide single sign-on for that enterprise's users or its customers. This is not the same as a third-party identity exchange, such as the kind envisioned by the Identity Ecosystem Steering Group (whose website, incidentally, is powered by Ping). There is also a lot of discussion in the IAM community about who can and should act as identity providers, and the candidates include social media such as Facebook or Google, financial institutions and telcos, since all of these appear to have the largest user bases.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">However, none of these identity providers in and of themselves can supply all of the assurance and validation that different business cases require. It doesn't matter whether Verizon has verified a user for phone service if a relying party has to figure out whether the user is really the same one who walked into the emergency room last night. Some organizations have much stronger requirements for identity assurance, and will have to assemble their own validation lists from multiple ID providers.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Not only does the ID and attribute exchange need to be vendor-agnostic, it also needs to be easy to join. This is where the pre-integration and onboarding services are crucial. Customers don't have to let FuGen host the hub, but it helps with the kind of complex troubleshooting that federated IAM can sometimes require. The opportunity for FuGen is that it can be a broker for the brokers, so to speak: each enterprise in an ideal world would have just one interface to expose to the world, but those interfaces still need to be matched up with the other ones.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The term 'broker' is confusing, but if we focus on 'exchange,' we get closer to our original meaning and can consider the competition. SecureKey Technologies was recently awarded a contract by the US Postal Service to create the Federal Cloud Credential Exchange. Criterion Systems was one </span><span style="font-family: Verdana, sans-serif;">of the National Strategy for Trusted Identities in Cyberspace pilot grant recipients in 2012, and is building its ID DataWeb Attribute Exchange Network, with an ecosystem of technology partners and relying parties such as Ping, CA Technologies, Fixmo, Verizon, Experian and Wave Systems. If firms </span><span style="font-family: Verdana, sans-serif;">like these manage to build a working exchange, it could rival what 8KMiles and FuGen can do. Again, the latter are helping customers set up the integration, not just acting as a provider, so the operational features of their offering set it apart from these exchange projects. The race will be to see who can collect the largest amount of trusted resources and participants in a broadly working exchange. Vendor neutrality and </span><span style="font-family: Verdana, sans-serif;">open standards will play a role, but so will user-friendliness. If FuGen can offer both the onramp services and the day-to-day operation in a way that preserves trust, it could have the magic formula.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">SWOT Analysis</span></b><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Strengths</span></b><br />
<span style="font-family: Verdana, sans-serif;">As a cloud broker, 8KMiles expanded its repertoire with the acquisition of FuGen. Identity management is certainly a key part of cloud migration and operation, and FuGen's virtualized lab environment helps it work out all of the bugs in a complex identity federation system without impacting the customer.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Weakness</span></b><br />
<span style="font-family: Verdana, sans-serif;">FuGen may be known in the IAM industry, particularly due to its participation in public initiatives, but customers may find the name too confusing alongside 8KMiles (neither name really says what the company does). It also has a lot of potential in supporting an identity and attribute exchange, but that potential needs to be realized.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Opportunities</span></b><br />
<span style="font-family: Verdana, sans-serif;">Nobody has really figured out federation yet. Even though some straightforward, homogeneous business use cases are working fine, the more complicated ecosystems are still in the committee/framework/pilot stages. If 8KMiles/FuGen can onramp enough critical-mass partners, it could become a de facto hub before these committees can turn around.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Threats</span></b><br />
<span style="font-family: Verdana, sans-serif;">Vendors such as SecureKey and Criterion are building exchanges too, although they're in the early stages.</span><br />
<span style="font-family: Verdana, sans-serif;">8KMiles/FuGen will also be confused with many other cloud IAM technology vendors due to the misuse of the term broker. </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Analyst(s): Wendy Nather , </span></b><b><span style="font-family: Verdana, sans-serif;">451 Research</span></b>Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-36288930376800531232014-01-23T08:53:00.001-08:002014-01-23T08:53:19.931-08:00AWS Cost Saving Tip 18: Using AWS Tagging with Graffiti Monkey to Saves costs<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: Verdana, sans-serif;">Tagging AWS assets and resources is a simple and an important feature that helps IT teams streamline their process of managing the environments. We can categorize AWS resources in variety of ways using the AWS Tagging feature. For example: </span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">Environment based Tags: Environment based tags can be used by customers who have Larger AWS footprints. These tags will help you to distinguish between production, development, and staging infrastructure deployed on AWS</span></li>
<li><span style="font-family: Verdana, sans-serif;">Application based Tags: Used to describe the set of disparate resources (or clusters) that work together to deliver a product or service to a customer. Can be applied on Small to Big AWS Production environments to identify the resources used</span></li>
<li><span style="font-family: Verdana, sans-serif;">Role based Tags– Used to describe the function of a particular node (web server, database server, load balancer, etc.). </span><span style="font-family: Verdana, sans-serif;">Owner – Used to identify the individual who is responsible for the instance. Both these cases are usable for small AWS foot prints.</span></li>
</ul>
<br />
<span style="font-family: Verdana, sans-serif;"><b>How it helps ?</b></span><br />
<span style="font-family: Verdana, sans-serif;">Tags can be very useful in understanding our AWS Billing. When you run large foot prints on AWS, constantly we need to monitor and manage our resource usage. Any IT operations team which manages AWS resources, constantly need to work on the following questions to optimize and manage their operations efficiently. Which AWS resources are used ? Which resources are lying idle (volumes, snapshots) in detached state ? Which Tiers or environments are consuming more billing than others? Which owner is exceeding the AWS cost allocated ? In short by starting to tag the AWS resources we will know how those assets are translating to monthly cost and take optimization decisions. Tagging is an simple but an essential feature that helps us to reduce and keep costs in control by improving our management efficiency in AWS.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>AWS Tags in Detail:</b></span><br />
<span style="font-family: Verdana, sans-serif;">Each tag consists of a key and a value, both of which the user can define. Currently AWS does not provide facility to TAG all AWS resources. Some of them can be Tagged only using API actions or the Command Line. The following Screen Shot shows all AWS resources and the restrictions that apply to them. Resources with tagging restrictions of None can be tagged with API actions, the CLI, and the console.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjimFdsdfmF0ExJVWF7PiC7nIZ4fj_tTeqsgrY-vCWG7IWtiYyyZnp3n4h2mYhWB3rSY-A1ChvNInBNxYlcQtRyYvPrYxANM35M2yDuqPajTV-YyhBuYs0GZfTWYOsMur8LAxMq-X-Fmu3K/s1600/AWS+Resources+Tagging.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjimFdsdfmF0ExJVWF7PiC7nIZ4fj_tTeqsgrY-vCWG7IWtiYyyZnp3n4h2mYhWB3rSY-A1ChvNInBNxYlcQtRyYvPrYxANM35M2yDuqPajTV-YyhBuYs0GZfTWYOsMur8LAxMq-X-Fmu3K/s1600/AWS+Resources+Tagging.jpg" height="640" width="498" /></a></div>
<span style="font-family: Verdana, sans-serif;">The following diagram illustrates how tagging works. In this example, i am assigning two tags </span><span style="font-family: Verdana, sans-serif;">to each of my EC2 instances, one called Owner and another called Stack. Each of the tags also has an associated value.</span><br />
<span style="font-family: Verdana, sans-serif;">Owner = 8KMiles , </span><span style="font-family: Verdana, sans-serif;">Stack = Development, Stack = Production as illustrated in the below diagram</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiffQBh6AcIOxa0ZVO7OBmVAfnzb1bP7PJkBRzk9Iy8BMLEYWOYIFBJImko0RabKfl5xR3yEx8O0hlUqVnu8e0Bq0wfCi6WHkF5dOcbbLi7gkjT0UYSvAsaxCgNBseFxOTeXIyt8CCD0nn_/s1600/2-AWS+EC2+Tagging+Example.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiffQBh6AcIOxa0ZVO7OBmVAfnzb1bP7PJkBRzk9Iy8BMLEYWOYIFBJImko0RabKfl5xR3yEx8O0hlUqVnu8e0Bq0wfCi6WHkF5dOcbbLi7gkjT0UYSvAsaxCgNBseFxOTeXIyt8CCD0nn_/s1600/2-AWS+EC2+Tagging+Example.jpg" height="190" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5JlDSaYfuUfKjbIxmJqzTQ-mYBSbASIsMmOiu6DvXqgWe6sx8OD0Dz3GiT5774XGHOX4kCJuWgNQGTLL6nPehDPLhxalZ6H4scVJbsdd5A9sSUCl1l2rd1bklZw5RcWc7l0uC6yJc1bso/s1600/3-AWS+EC2+Tagging+Example.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5JlDSaYfuUfKjbIxmJqzTQ-mYBSbASIsMmOiu6DvXqgWe6sx8OD0Dz3GiT5774XGHOX4kCJuWgNQGTLL6nPehDPLhxalZ6H4scVJbsdd5A9sSUCl1l2rd1bklZw5RcWc7l0uC6yJc1bso/s1600/3-AWS+EC2+Tagging+Example.jpg" height="181" width="320" /></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">You can assign tags using the AWS Management Console, the Amazon EC2 command line interface (CLI), and the Amazon EC2 API.You can define tags from the EC2 console by selecting the relevant instance and selecting the “Tags” option. Alternatively, you can select the instance, click “Actions” and select “Add/Edit Tags.” Below Screenshot shows how tags can be assigned using AWS Management Console</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWFSagqUSE7tB6S0OjJSKfQucNnublA_ZdKUqe2TJJnRb4SGHImFMM255cS3zZam-hmjD8HuYGSlIqbRY6InuuLByYGNsvRlzsSLHVlANAda9HBhfu_64dZDMhAUtVuzkmBJqrwDh41QJf/s1600/4-How+to+assign+Tags+to+AWS+resources+using+console.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWFSagqUSE7tB6S0OjJSKfQucNnublA_ZdKUqe2TJJnRb4SGHImFMM255cS3zZam-hmjD8HuYGSlIqbRY6InuuLByYGNsvRlzsSLHVlANAda9HBhfu_64dZDMhAUtVuzkmBJqrwDh41QJf/s1600/4-How+to+assign+Tags+to+AWS+resources+using+console.jpg" height="326" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">We have assigned Name, Owner , Environment and Contact for the 8KMiles Production instance using the AWS Tagging feature. The above is not the usual EC2 Managed Service naming standard , but just an simple illustration of the tag use case.</span><br />
<span style="font-family: Verdana, sans-serif;"><b>AWS Tagging Constraints and the Cost Leakage:</b></span><br />
<span style="font-family: Verdana, sans-serif;">Currently AWS Tagging has some limitations as well. For example each tag associated with a single AWS resource, applies only to that particular resource and it is not propagated to the dependent resources attached to it. Example: Imagine if you have an EC2 instance, most of the times this EC2 instance is attached with dependent resources like EBS volumes, Snapshots etc. Usually there will be 1-2 EBS Volumes on normal scenarios and up to 8 EBS Volumes or more on RAID Scenarios. Number of snapshots are more in number and usually becomes unmanageable. </span><span style="font-family: Verdana, sans-serif;">Imagine how this problem will be propagated if you are running a Large AWS foot print with thousands of EC2 instances. </span><br />
<span style="font-family: Verdana, sans-serif;">You have to find each EBS volumes associated with a particular EC2 and Tag it separately.Managing this is easier said than done and usually slightest mis- management of these resources will lead to cost leakage. What if i can use a tool to automate tagging for dependent AWS resources. It will surely reduce the manual labor associated and help us remove all the dependent resources when we are removing the EC2. This avoids leakages and reduces cost of Cloud Operations in very dynamic AWS environments.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b><span style="font-family: Verdana, sans-serif;">Welcome to </span><span style="font-family: Verdana, sans-serif;">Graffiti Monkey. </span></b><br />
<span style="font-family: Verdana, sans-serif;">The Graffiti Monkey goes around tagging things automatically</span><span style="font-family: Verdana, sans-serif;">. By looking at the tags of an EC2 instance, it copies those tags to the EBS Volumes that are attached to it, and then copies those tags to the EBS Snapshots as well. This small automation reduces hours of Manual labor on large AWS footprints, helps us efficiently manage the cloud infrastructure and reduce the cost leakages.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 1: Installation of Graffiti Monkey </b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjboYBt3GikdxAuYQvO5Iuci4UP-ZOn0odtQOwwMXFXOJc769HKrQQuapxhHJi4upw95ZQsYmqAvJCTOfbmDgHspgwc868XFLa6iHtpc8a92l7xGjIM1Asf-5j6C2SWOwy_YL52Tm4QRrop/s1600/5-Installing+Graffiti+Monkey.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjboYBt3GikdxAuYQvO5Iuci4UP-ZOn0odtQOwwMXFXOJc769HKrQQuapxhHJi4upw95ZQsYmqAvJCTOfbmDgHspgwc868XFLa6iHtpc8a92l7xGjIM1Asf-5j6C2SWOwy_YL52Tm4QRrop/s1600/5-Installing+Graffiti+Monkey.jpg" height="166" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Easiest way to install Graffiti Monkey using the usual PyPI channels.Please refer the below screenshot</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjONlXyvjFwS5EnjvaCwXUv_M5GeAKrX_zZWJqU2g7RHu_kWsQYR-4pMVXfTsbvl-JKnm7FBJBnc-6Q1TvFBMoUT6XnJNxrGzQf_0AQirVhWPM6oeGcVDkkq1IQ6weMN0mEV9gvHjciB8GW/s1600/6-Installing+Graffiti+Monkey.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjONlXyvjFwS5EnjvaCwXUv_M5GeAKrX_zZWJqU2g7RHu_kWsQYR-4pMVXfTsbvl-JKnm7FBJBnc-6Q1TvFBMoUT6XnJNxrGzQf_0AQirVhWPM6oeGcVDkkq1IQ6weMN0mEV9gvHjciB8GW/s1600/6-Installing+Graffiti+Monkey.jpg" height="136" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;">link for installing pip.http://www.pip-installer.org/en/latest/installing.html</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Step 2: Using Graffiti Monkey</b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8IMvlUjz0PWQMyxASIdGLXBDfWuER9wvXZP73z3LMY3Gvfc3v8lxfBonQZ1L77uA_oJY8MHq374rFBpf8ZoV_gy2QxzAVcBgRBlzVYKPusdwQmoK0JsVbDmU-RJbgJFjohmmo9yQfFAhL/s1600/8-Applying+Graffiti+Monkey.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8IMvlUjz0PWQMyxASIdGLXBDfWuER9wvXZP73z3LMY3Gvfc3v8lxfBonQZ1L77uA_oJY8MHq374rFBpf8ZoV_gy2QxzAVcBgRBlzVYKPusdwQmoK0JsVbDmU-RJbgJFjohmmo9yQfFAhL/s1600/8-Applying+Graffiti+Monkey.jpg" height="114" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">It will identify the tags associated with EC2 instance and applies them to dependent resources like EBS volumes and snapshots saving hours of manual labor</span>.<br />
<br />
<span style="font-family: Verdana, sans-serif;">This Post is co authored with Nitheesh 8Kmiles</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<div class="MsoNormal" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><b><span style="font-size: large;">Other Cost Saving Tips</span></b></span><br /><span style="font-size: x-small;"><span style="font-family: Verdana, sans-serif;"></span></span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-SQS-Long-polling-Batch-requests.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests</a></span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Choosing-right-search-tier-Amazon-cloudsearch-Apache-Solr.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 2: How right search technology choice saves cost in AWS ?</a></span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-Cloudfront-Price-class.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs</a></span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-ElastiCache-cluster-nodes.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster</a></span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?</a></span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-AutoScaling-termination-policy-ClosestToNextInstanceHour.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings</a></span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Object-expiration.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 7: Use Amazon S3 Object Expiration</a></span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Reduced-Redundancy-storage-rrs-vs-std.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage</a> </span><span style="font-family: Verdana, sans-serif;"> </span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Delete-Amazon-EBS-snapshots-regularly-retention-period.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place</a></span><span style="font-family: Verdana, sans-serif;"> </span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-make-right-EBS-Volumes-Choice-PIOPS-vs-Standard.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs</a> </span><br /><a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-elasticity-Amazon-Elastic-MapReduce-EMR.html" style="color: #4d469c; font-family: Verdana, sans-serif;">Cost Saving Tip 11: <span style="color: #0000ee;">How elastic thinking saves cost in Amazon EMR Clusters ?</span> </a><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-Adding-Spot-EC2-with-Amazon-Elastic-MapReduce-EMR.html" style="color: #4d469c;">Cost Saving Tip 12: <span style="color: #0000ee;">Add Spot Instances with Amazon EMR</span></a> </span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-use-Amazon-Glacier-for-archives-and-save-costs.html" style="color: #4d469c;">Cost Saving Tip 13: <span style="color: #0000ee;">Use Amazon Glacier for archive data and save costs</span></a> </span><br /><span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-deletion-in-Amazon-glacier.html" style="color: #4d469c;">Cost Saving Tip 14: <span style="color: #0000ee;">Plan your deletion in Amazon Glacier and avoid cost leakage</span></a> </span><br /><div>
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html" style="color: #4d469c;">Cost Saving Tip 15: <span style="color: #0000ee;">Plan your Amazon Glacier Retrieval strategy and avoid cost leakage</span></a> </span></div>
</div>
<div class="MsoNormal" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; line-height: 18.200000762939453px;">
<a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html" style="color: #4d469c; font-family: Verdana, sans-serif; line-height: 18px;">Cost Saving Tip 16: </a><span style="color: #0000ee; font-family: Verdana, sans-serif;"><span style="line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/10/Amazon-Web-Services-Cost-Saving-Tips-periodically-remove-EBS-AMI-S3-EIP-ELB-and-save-costs.html" style="color: #4d469c;">Periodically remove your unwanted AWS resources and save costs</a></span></span><span style="font-family: Verdana, sans-serif;"><span style="line-height: 18px;"> </span></span><b style="font-family: Verdana, sans-serif; line-height: 18px;"><span style="color: red;">(new)</span></b></div>
<div class="MsoNormal" style="background-color: white; color: #444444; line-height: 18.200000762939453px;">
<h3 class="post-title entry-title" itemprop="name" style="line-height: normal; margin: 0px; position: relative;">
<span style="font-family: Verdana, sans-serif; font-size: small;"><a href="http://harish11g.blogspot.in/2013/10/Amazon-Web-Services-Cost-Saving-Tips-Add-Remove-Amazon-RDS-Read-replica-dynamically-and-save-costs.html" style="font-weight: normal;">AWS Cost Saving Tip 17: Add Amazon RDS Read Replica's on need basis and save costs</a> (new)</span></h3>
</div>
<span style="font-family: Verdana, sans-serif;"><br /></span>Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-53121309090249351812014-01-20T07:06:00.002-08:002014-08-29T10:40:06.406-07:00Architecting Highly Available ElastiCache Redis replication cluster in AWS VPC<span style="font-family: Verdana, sans-serif;">In this post lets explore how to architect and create a Highly Available + Scalable Redis Cache Cluster for your web application in AWS VPC. Following is the architecture in which the ElastiCache Redis Cluster is assembled:</span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">Redis Cache Cluster inside Amazon VPC for better control and security</span></li>
<li><span style="font-family: Verdana, sans-serif;">Master Redis Node 1 will be created in AZ-1 of </span><span style="font-family: Verdana, sans-serif;">US-West</span></li>
<li><span style="font-family: Verdana, sans-serif;">Redis Read Replica Node 2 will be created in AZ-2 of </span><span style="font-family: Verdana, sans-serif;">US-West</span></li>
<li><span style="font-family: Verdana, sans-serif;">Redis Read Replica Node 3 will be created in AZ-3 of </span><span style="font-family: Verdana, sans-serif;">US-West</span></li>
</ul>
<br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk_WuFjbG7d8qSbH_oyXs5UEwcYQH9XqPTfbxc03lIWAl5VKMc9PL4LxAlkgWOdo9rxT4peWapUpT-l4JxfiJIpkEKR5rPd7YisNUqge20bmZkwPE_rEEW3wI_uC3M-oAAY3Z_xmBW0bsR/s1600/ElastiCache+Redis+replication+cluster+architecture+in+AWS.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk_WuFjbG7d8qSbH_oyXs5UEwcYQH9XqPTfbxc03lIWAl5VKMc9PL4LxAlkgWOdo9rxT4peWapUpT-l4JxfiJIpkEKR5rPd7YisNUqge20bmZkwPE_rEEW3wI_uC3M-oAAY3Z_xmBW0bsR/s1600/ElastiCache+Redis+replication+cluster+architecture+in+AWS.jpg" height="488" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;">You can position all the 3 Redis Nodes in different Availability zones for Achieving High Availability (or) you can position Master + RR 1 in AZ1 and RR 2 in AZ2. This reduces the Inter - AZ latency and might give better performance for heavily used clusters.</span><br />
<b style="font-family: Verdana, sans-serif;">Step 1: Creating Cache Subnet groups:</b><br />
<span style="font-family: Verdana, sans-serif;">To create Cache Subnet group navigate to the dashboard of ElastiCache, select Cache Subnet groups and then click "Create Cache Subnet group". Add the Subnet Id and the Availability Zone you need to use for the ElastiCache cluster.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfAMh3JU5X_ZfVoxY_xQ6koDw5ulzB6LkqEkdsWNqTXxhcEYKc_8tAraLFa_ZakRUEd8OX24d53w7IBBXOacigkXGF4ky-i0i2-kvvvK7mEG5_-I4cnRph0dkRiFzAqxr4yg-n4-QY1hGd/s1600/11-+Amazon+ElastiCache+Redis+Replication+cluster+Cache+Subnet+Group.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfAMh3JU5X_ZfVoxY_xQ6koDw5ulzB6LkqEkdsWNqTXxhcEYKc_8tAraLFa_ZakRUEd8OX24d53w7IBBXOacigkXGF4ky-i0i2-kvvvK7mEG5_-I4cnRph0dkRiFzAqxr4yg-n4-QY1hGd/s1600/11-+Amazon+ElastiCache+Redis+Replication+cluster+Cache+Subnet+Group.jpg" height="278" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;">We have created Amazon VPC spreading across 3 availability zones. In this post we are going to place the Redis Master and 2 Redis Replica Slaves in these 3 availability zones. Since Redis will be most of the times accessed by your application tier it is better if you place them on </span><span style="font-family: Verdana, sans-serif;">Private Subnet</span><span style="font-family: Verdana, sans-serif;"> of your VPC.</span><br />
<b style="font-family: Verdana, sans-serif;">Step 2: Creating Redis Cache Cluster: </b><br />
<span style="font-family: Verdana, sans-serif;">To create Cache Cluster navigate to the dashboard of ElastiCache, select Launch Cache Cluster and provide the necessary details. We are launching it inside Amazon VPC, so we have to select the Cache Subnet group .</span><br />
<span style="font-family: Verdana, sans-serif;">Note: It is mandatory to create Cache Subnet group before Launch if you need ElastiCache Redis cluster in Amazon VPC.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicTDUcIJRZ6poBgd1Ti1Qc1hHbqfjotst6BuIIpe5mvqzXnN1YNEBk4y751MBwmxP3O_iTkbVz6XRZnwSO2R9YTE4UUZVNrXmQDhZGrU4tzGgngCH3wZgfs8CCcz0eJzaH75lXGLyMEBCR/s1600/1-+Amazon+ElastiCache+Redis+Cache+cluster+creation.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicTDUcIJRZ6poBgd1Ti1Qc1hHbqfjotst6BuIIpe5mvqzXnN1YNEBk4y751MBwmxP3O_iTkbVz6XRZnwSO2R9YTE4UUZVNrXmQDhZGrU4tzGgngCH3wZgfs8CCcz0eJzaH75lXGLyMEBCR/s1600/1-+Amazon+ElastiCache+Redis+Cache+cluster+creation.jpg" height="544" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;">For test purposes i have used m1.small EC2 instance for the Redis. Since this is a fresh Redis installation, i have not mentioned S3 bucket from where the persistent Redis Snapshot will be used as input. </span><span style="font-family: Verdana, sans-serif;">On successful creation of the Cache Cluster you can see the details in the dashboard.</span><br />
<b style="font-family: Verdana, sans-serif;">Step 3: Replication Group Creation:</b><br />
<span style="font-family: Verdana, sans-serif;">To create Replication group select the option of Replication Groups from dashboard and then select the “Create Replication Group”</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiz7SykITb_P3weA7R6xAjj6ffurGfzheZ1Kx1IrJuZBu4NHkEAAcR660ZKDt8zC_V8Ev9xuXZc_dlZW71-GL6fcud25cAkytRhLN9P9AwZpgo21-a6v_279BxTEHTWMDXnIyBCclOH_lZ1/s1600/2-+Amazon+ElastiCache+Redis+Replication+group+creation.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiz7SykITb_P3weA7R6xAjj6ffurGfzheZ1Kx1IrJuZBu4NHkEAAcR660ZKDt8zC_V8Ev9xuXZc_dlZW71-GL6fcud25cAkytRhLN9P9AwZpgo21-a6v_279BxTEHTWMDXnIyBCclOH_lZ1/s1600/2-+Amazon+ElastiCache+Redis+Replication+group+creation.jpg" height="118" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Select the master Redis node "</span><span style="font-family: Verdana, sans-serif;">redisinsidevpc" created previously as the </span><span style="font-family: Verdana, sans-serif;">primary cluster id of the Cache cluster. Give the Replication group id and description as illustrated below.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGCeJSMq_t1nIpZX6a_NHoCFy0uYrZs3ZOeNmLA7zPz9a3EWESq9o3NzBGfa5cNYy0CHZ0iNjRf_APcGymhLr_j6vtzTGmaL-Hh7h6NvTMjCstitR3lzd_JVlD1NUIS_FiDIe7eH_hhwWC/s1600/3-+Amazon+ElastiCache+Redis+Replication+group+creation.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGCeJSMq_t1nIpZX6a_NHoCFy0uYrZs3ZOeNmLA7zPz9a3EWESq9o3NzBGfa5cNYy0CHZ0iNjRf_APcGymhLr_j6vtzTGmaL-Hh7h6NvTMjCstitR3lzd_JVlD1NUIS_FiDIe7eH_hhwWC/s1600/3-+Amazon+ElastiCache+Redis+Replication+group+creation.jpg" height="254" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;">Note: Replication Group should be created only after the Primary Cache Cluster node is UP and running, else you will get the error as shown below.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuz1Yd2FZaYgcZNXAs5NOsH3QN51EPoHwl3xII_YWMtxHFQip9HvllEcW2xvtqsdFdD9_xsxatndhNRUEOn_6CNx3JSmhqVI846ZnbxJH8SLAq8Nz7XK4IS1fuxJWIWCcg6t1cwAbN-l0b/s1600/4-+Amazon+ElastiCache+Redis+Replication+cluster.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuz1Yd2FZaYgcZNXAs5NOsH3QN51EPoHwl3xII_YWMtxHFQip9HvllEcW2xvtqsdFdD9_xsxatndhNRUEOn_6CNx3JSmhqVI846ZnbxJH8SLAq8Nz7XK4IS1fuxJWIWCcg6t1cwAbN-l0b/s1600/4-+Amazon+ElastiCache+Redis+Replication+cluster.jpg" height="244" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;">On the successful creation of the Replication group you can see the following details. You can observe from below screenshot that there is only one primary node in US-WEST-2A and zero Redis Read Replica's are attached to it.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn7VLKg86vDzprXIPMABxZskfVyeN6ZZCe2bYgmdzB2zuS1ym84P7BVHN_CistL9az5JvlD3EmC1rtaeeSZu1NY4oOfyQdF_YjQgmu3uaJ_PWjzjCzPpksQI0DbTVsMDRC66NN7QUxS0kY/s1600/5-+Amazon+ElastiCache+Redis+Replication+group+creation.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn7VLKg86vDzprXIPMABxZskfVyeN6ZZCe2bYgmdzB2zuS1ym84P7BVHN_CistL9az5JvlD3EmC1rtaeeSZu1NY4oOfyQdF_YjQgmu3uaJ_PWjzjCzPpksQI0DbTVsMDRC66NN7QUxS0kY/s1600/5-+Amazon+ElastiCache+Redis+Replication+group+creation.jpg" height="276" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<b style="font-family: Verdana, sans-serif;">Step 4: Adding Read Replica Nodes:</b><br />
<span style="font-family: Verdana, sans-serif;">When you select the Replication group, you can see the option to add Redis Read Replica. We are adding 2 Redis Read Replica named Redis-RR1 (in US-West-2B) and Redis-RR2 (in US-WEST-2C). Both the Read replica's are pointed to the master node "</span><span style="font-family: Verdana, sans-serif;">redisinsidevpc". </span><span style="font-family: Verdana, sans-serif;">Currently we can add up to 5 Read replica Nodes for a Redis Master Node. This is more than enough to handle Thousands of messages per second. If you combine it with Redis Pipeline handling 100K messages per second from a node is like cake walk.</span><br />
<span style="font-family: Verdana, sans-serif;">Adding Read Replica 1 in Us-West -2B</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgi4DZ_DL8Ru2xSwLpbcSx2IfO5KU2EzU2jpaCP_fLONVjoeW5bGbXoOtu3ejLRnjhRZaDW2EvWVbWuS_I5mRfOFo60bnnBwwLGUddUSdYV0jPOcZCsRC9cz0ljfrQ7qUQHgU1k7aPONhO/s1600/6-+Amazon+ElastiCache+Redis+Adding+read+replica+Redis+nodes.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgi4DZ_DL8Ru2xSwLpbcSx2IfO5KU2EzU2jpaCP_fLONVjoeW5bGbXoOtu3ejLRnjhRZaDW2EvWVbWuS_I5mRfOFo60bnnBwwLGUddUSdYV0jPOcZCsRC9cz0ljfrQ7qUQHgU1k7aPONhO/s1600/6-+Amazon+ElastiCache+Redis+Adding+read+replica+Redis+nodes.jpg" height="242" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;">Adding Read Replica 2 in US-West-2c</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-5G5XfOEG_HK3KtwTIyAir-X23Bwzg6J2PhW9y8LSKJ6Q3-AUYEqQXZPaOh3HzAI1sHXcDL7jw4wngO3Y67ih4B0gPUuLm3VxyzlTtjH2vP9Mp0_UpFBJ_jO0e4Z-qIqOtpvrBJ5m2DTf/s1600/7-+Amazon+ElastiCache+Redis+Replication+cluster+adding+read+replica+nodes.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-5G5XfOEG_HK3KtwTIyAir-X23Bwzg6J2PhW9y8LSKJ6Q3-AUYEqQXZPaOh3HzAI1sHXcDL7jw4wngO3Y67ih4B0gPUuLm3VxyzlTtjH2vP9Mp0_UpFBJ_jO0e4Z-qIqOtpvrBJ5m2DTf/s1600/7-+Amazon+ElastiCache+Redis+Replication+cluster+adding+read+replica+nodes.jpg" height="250" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">On successful creation you can see the following details of Replication group in the dashboard. Now you can see there are 3 Redis nodes listed with Number of read Replica's as 2. Placing the Read Replica's and master node in multiple AZ will increase the high availability and protects you from node and AZ level failure. On our sample tests inter AZ Replication deployments had <2 second replication lag for massive writes on master and <1 second replication lag between master slave inside same AZ deployments. We pumped @100K messages per second for few minutes on m1.large Redis instance cluster. </span><br />
<span style="font-family: Verdana, sans-serif;">In event, if you need additional read scalability i recommend to use more read Replica slaves added to the master. </span><br />
<span style="font-family: Verdana, sans-serif;">In your application tier you need to use the primary Endpoint </span><b style="font-family: Verdana, sans-serif;">"redis-replication.qcdze2.0001.usw2.cache.amazon.aws.com:6379"</b><span style="font-family: Verdana, sans-serif;"> shown below to connect to Redis. </span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3x6zxIYbutXBlJN-idllAiEA0AVRb2plkrI9kYCxcSwlk-5FpylEES3vmnPwdu0S-HBPqsTzWfmy5wdLt6LyS7b7iEldvNAirdG4b8J3DAPfVUHbVITjt69LftvSIcMTaXhzcl_dAnq4H/s1600/8-+Amazon+ElastiCache+Redis+Replication+cluster.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3x6zxIYbutXBlJN-idllAiEA0AVRb2plkrI9kYCxcSwlk-5FpylEES3vmnPwdu0S-HBPqsTzWfmy5wdLt6LyS7b7iEldvNAirdG4b8J3DAPfVUHbVITjt69LftvSIcMTaXhzcl_dAnq4H/s1600/8-+Amazon+ElastiCache+Redis+Replication+cluster.jpg" height="264" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;">If you need to delete/reboot/Modify you can make it through the options available here.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Verdana, sans-serif; margin-left: 1em; margin-right: 1em;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieLavYLAuXIpfHSw_vQwHIWCqOqPmxt8M02U4z3iSONJZGwWTW5uAABkA7rUgv6m3RnzJW7OqKqn1-aVllcGm__YJyRo-lHZq1ZXbXauf7oAO8qq1rwu5UBPOQ8CO8Lhuzui24sAReJRdV/s1600/10-+Amazon+ElastiCache+Redis+Replication+cluster.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieLavYLAuXIpfHSw_vQwHIWCqOqPmxt8M02U4z3iSONJZGwWTW5uAABkA7rUgv6m3RnzJW7OqKqn1-aVllcGm__YJyRo-lHZq1ZXbXauf7oAO8qq1rwu5UBPOQ8CO8Lhuzui24sAReJRdV/s1600/10-+Amazon+ElastiCache+Redis+Replication+cluster.jpg" height="188" width="640" /></a></span></div>
<div>
<b style="font-family: Verdana, sans-serif;">Step 5: Promoting the Read replica:</b></div>
<span style="font-family: Verdana, sans-serif;">You can also promote any node as the Primary cluster using the Promote/Demote option. There will be only one Primary Node.</span><br />
<span style="font-family: Verdana, sans-serif;">Note: </span><span style="font-family: Verdana, sans-serif;">This step is not part of the cluster creation process.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjh3RdRnATVPhN4ZxqlHwAaHOruhjvyrKx-pa44p2vL-XO_l-1tGEIQQ8gGQ6zZm1-rzHwZpnNdDPQspSRGVXsBHEPIuUfcBFb7QX3UgfBcF1OaY5Z-42kX2YlBMc2bfp4M3y7Sk7x89gRV/s1600/9-+Amazon+ElastiCache+Redis+Replication+cluster.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjh3RdRnATVPhN4ZxqlHwAaHOruhjvyrKx-pa44p2vL-XO_l-1tGEIQQ8gGQ6zZm1-rzHwZpnNdDPQspSRGVXsBHEPIuUfcBFb7QX3UgfBcF1OaY5Z-42kX2YlBMc2bfp4M3y7Sk7x89gRV/s1600/9-+Amazon+ElastiCache+Redis+Replication+cluster.jpg" height="145" width="400" /></span></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">This promotion has to be carried out with caution and proper understanding for maintaining data consistency. </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Post was co authored with Senthil 8KMiles</b></span><br />
<span style="font-family: Verdana, sans-serif;"><b><br /></b></span>
<span style="font-family: Verdana, sans-serif;"><b>Other related posts:</b></span><br />
<h3 class="post-title entry-title" itemprop="name" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; margin: 0px; position: relative;">
<span style="font-size: small;"><a href="http://harish11g.blogspot.in/2014/08/art-of-architecting-highly-scalable-available-elasticache-redis-tier.html">Billion Messages - Art of Architecting scalable ElastiCache Redis tier</a></span></h3>
<div>
<b style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;"><u><a href="http://harish11g.blogspot.in/2012/11/amazon-elasticache-memcached-ec2.html" style="color: #4d469c; text-decoration: none;">Caching architectures using Memcached & Amazon ElastiCache</a></u></b><br style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;" /><b style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;"><u><a href="http://harish11g.blogspot.in/2013/01/amazon-elasticache-memcached-internals_8.html" style="color: #4d469c; text-decoration: none;">Deep dive into Amazon Elasticache</a></u></b><br style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;" /><b style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;"><u><a href="http://harish11g.blogspot.in/2012/12/amazon-elasticache-distributed-caching.html" style="color: #4d469c; text-decoration: none;">Distributed Cache On Steroids: Amazon ElastiCache</a></u></b><br style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;" /><b style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;"><u><a href="http://harish11g.blogspot.in/2012/11/configuring-amazon-elasticache-launch.html" style="color: #4d469c; text-decoration: none;">Launching Amazon ElastiCache in 3 Easy Steps</a></u></b><br style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;" /><b style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18.200000762939453px;"><u><a href="http://harish11g.blogspot.in/2013/08/using-Amazon-ElastiCache-to-improve-performance-tips.html" style="color: #4d469c; text-decoration: none;">AWS Performance Tip 3: Use Amazon ElastiCache to improve application performance</a></u></b></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com4tag:blogger.com,1999:blog-4836510665235566234.post-19726055185526424852014-01-18T22:16:00.006-08:002014-01-20T09:47:02.895-08:00Load Testing tool comparison - JMeter on it's own vs JMeter & BlazeMeter together<span style="font-family: Verdana, sans-serif;">Load testing is an important aspect of web applications life cycle on Amazon Cloud. Some of our customers ask us to generate 50000+ RPS to load test the scalability of their application deployed on Amazon cloud. Whenever we used to help such customers and migrate their applications on Amazon cloud for achieving scalability, load testing phase itself becomes a pain. Setting up the Load Testing infrastructure, writing automation around it, Managing, Maintaining and monitoring the load test infrastructure is an headache. Our Load testers and Infrastructure teams were spending considerable time and efforts on the above , instead of focusing only on load testing. We usually work with variety of tools from Grinder, JMeter, HP Load Runner to custom engineered load testing tools during the load testing phase. Some time back , our team started playing around with a SAAS based load testing tool called BlazeMeter. In this article i am going to share our experience in the form comparison between BlazeMeter and JMeter and why BlazeMeter has a bright future.</span><br />
<span style="font-family: Verdana, sans-serif;">Blazemeter is a Saas based high scalable load testing tool that handle up to 300,000+ concurrent users. Their load test infrastructure is spread across major AWS regions. Since most of us have been using JMeter for years , the 100 % compatibility it provides to existing JMeter scripts is a good feature. Blazemeter also provides a Chrome Extension which can record browser actions & convert it to .jmx file.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif; font-size: large;"><b><u>10 Things i like about BlazeMeter</u></b></span><br />
<br />
<span style="font-family: Verdana, sans-serif;"><b>Point 1)</b> Load Test becomes effective only when the load comes from different IP Addresses similar to real world scenario and not from a single source IP. When multiple virtual user load is generated from the same IP, the router as well as the server tries to cache information and optimize the throughput many times. Hence by using multiple IP addresses for the host, the EC2 server will get an illusion of receiving requests from multiple source IP's. Also it is better that load is generated from multiple IP's for Amazon ELB to evenly distribute load. <a href="http://harish11g.blogspot.in/2012/07/aws-elastic-load-balancing-elb-amazon.html">Refer URL</a>. BlazeMeter has capability to generate load from IP's which is very important on load testing the cloud applications.</span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 2)</b> <b>Customizing the Network Emulation: </b>Usually online applications will be accessed from multiple devices like PC's, Laptop and mobiles. These devices have multiple network types such as 3G, broad band etc.Also at times times our online application will be accessed from locations which has poor network bandwidth , Both these parameters play an important role in capacity planning and load testing. We can chose the Bandwidth and network type emulation while doing the load test using BlazeMeter. Example we can configure the network type such as Unlimited Internet, 3G, Cable, Wifi etc and. Bandwidth download limit per device can also be set.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 3)</b> <b>Controlling the Throughput: </b>Target throughput is a parameter of Apache JMeter that can be used to achieve a required throughput value of the application. A server’s performance need not always satisfy the target throughput value mentioned in JMeter. It could provide more throughput or lesser.The target throughput parameter can be controlled in run time in Blazemeter. Live server monitoring can help us identify if our servers are performing well for say 5000 Hits/sec & change the throughput value in run-time to a higher or lower value based on the server’s performance.</span></div>
<div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 4) Controlling the Agents: </b>Apache JMeter works based on Master-Agent based architecture where the Master controls multiple agents generating the load. The number of agents parameter has to be usually decided before the starting of the test while using JMeter based load testing on Amazon Cloud. Option to dynamically change the throughput value is a very good feature to have while load testing a cloud application requiring thousands of Requests per second. BlazeMeter enables us to add or remove agent instances when a test is running. Any instance can be marked as Master or Slave(Agent) while the test is running.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 5)</b> <b>Controlling No. of Simulated Users on Slaves (Agents)</b> : A load test strategy is mainly determined by following parameters like number of concurrent users, ramp up time, no. of test engines and test iterations and the test duration. Apache JMeter allows us to manually configure these values before the test is started. New EC2 instances have to provisioned for the Agents, the IP addresses (Usually Elastic IP) of the slaves/agents has to be manually added to the master. The entire setup has to be maintained, managed and monitored during the test cycles. This is ok for an load testing environment with few load test agents and low RPS, imagine an environment where you have generate thousands of RPS and having 50+ agents running. This process of managing the EC2 load test infrastructure will become tedious process overall for the load testing teams. In </span><span style="font-family: Verdana, sans-serif;">BlazeMeter</span><span style="font-family: Verdana, sans-serif;">, once the number of concurrent users is given, the number of test engines, number of threads and engine capacity is chosen automatically. This can be made semi-automatic, where the number of engines & number of threads as well can be selected by user and only engine capacity is chosen by </span><span style="font-family: Verdana, sans-serif;">BlazeMeter</span><span style="font-family: Verdana, sans-serif;">. Since it is a managed Load Test infrastructure, the Load Testers can concentrate the testing and not managing 100's of EC2 load agents.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 6)</b> <b>Integrated Monitoring: </b></span></div>
<div>
<ul>
<li><span style="font-family: Verdana, sans-serif;">BlazeMeter offers live monitoring of essential parameters of test servers when the test is running which enables us to decide on the number & instance type for the test. In the conventional Apache JMeter load test setup in Amazon EC2 we have to observe the Key parameters using AWS Cloudwatch.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Blaze Meter provides AWS Cloud watch integration.An account with IAM access has to be created and</span></li>
<li><span style="font-family: Verdana, sans-serif;">AWS Access Key & Secret Key values have to be configured so that the metrics are available in the Blazemeter’s dashboard. This features helps us to understand how the assets in the cloud are reacting to our load tests and help us accordingly tune the infrastructure.</span></li>
<li><span style="font-family: Verdana, sans-serif;">While performing load testing, it is important not only to monitor your Web Servers & Databases but also the agents from where the load is generated . The New Relic plugin gives us the front end KPIs and back end KPIs. </span></li>
<ul>
<li><span style="font-family: Verdana, sans-serif;">BlazeMeters’s frontend KPIs provide insight on how many users are actually trying to access your website, mobile site or mobile apps.</span></li>
<li><span style="font-family: Verdana, sans-serif;">BlazeMeters’s backend KPIs show how many users are getting through to your applications.</span></li>
</ul>
</ul>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 7)</b> Blazemeter allows us to have a different csv file per load test engine. Though this possible in Apache JMeter, it had to be done manually by copying the files onto the JMeter Agent EC2 instances and have the same filename since the agents refer to the Master’s properties. Blazemeter allows us to parameterize the values of even filenames and have different csv files in each engine without giving us to the trouble of copying files into specific EC2 instances & holds the files in a common repository so that it can be referred from there to each agent.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 8)</b> <b>Run the load test using older version of JMeter scripts:</b> Old scripts can be reusable with this feature of </span><span style="font-family: Verdana, sans-serif;">BlazeMeter</span><span style="font-family: Verdana, sans-serif;"> which lets us run the test using any version of Apache JMeter right from version 2.3.2 to 2.10. Some complex scripts prepared some months/years ago can be still be made usable and need not be redone. Saves efforts and costs.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 9)</b> <b>Schedule the Test & Stay Relaxed:</b> BlazeMeter as well as JMeter lets you schedule your test duration & test time so that we can run longevity test at any time of the day. Even weekly scheduling is possible in </span><span style="font-family: Verdana, sans-serif;">BlazeMeter</span><span style="font-family: Verdana, sans-serif;"> it is an added advantage, though it is not widely used.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b>Point 10)</b> <b>Interesting Plug-ins provided by Blazemeter :</b></span></div>
<div>
<ul>
<li><span style="font-family: Verdana, sans-serif;">Integration with Google Analytics: At the time of scripting, it is enough if we select the Google Analytics Option & provide account details of Google Analytics. BlazeMeter obtains the last 12 months of data and creates a test with 5 most visited pages and sets up the number of concurrent users based on that record.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Integration with WordPress: BlazeMeter provides integration with WordPress where WordPress users can test their App by using the BlazeMeter plug-in without any scripting.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Integration with Drupal & Jenkins: Plugins are available to load test Drupal & Jenkins servers as well. </span></li>
</ul>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;">Post Co Authored with Harine 8KMiles.</span></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com4tag:blogger.com,1999:blog-4836510665235566234.post-85030614000697226002014-01-13T01:07:00.001-08:002014-01-13T01:36:33.024-08:00AWS CloudTrail and Splunk for better Cloud IT Operations<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">In any Cloud IT operations, gaining visibility into user activities is important for achieving compliance and meeting the regulatory standards of the enterprise. For years in AWS, we used to solve this with a combination of tools for log collection, alerts and privileged Identity access management. At times, it becomes cumbersome to install and manage these tools and overall operation cost effectively on Amazon Cloud. Like any other company pioneering managed services in AWS infrastructure, Our Cloud operations was in touch with Amazon Cloud product team for years giving input in this area. As an outcome of this, we see a new product line called as AWS Cloud Trail launched last year.</span><br />
<br />
<div class="Textbody" style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 0px;">
<span style="font-family: Verdana, sans-serif;">AWS Cloud Trail provides a record of AWS API calls made on your AWS account. AWS CloudTrail can be used for <span class="StrongEmphasis">Resource Life Cycle Tracking, Operational Troubleshooting, Compliance Aid, and Security Analytics on AWS cloud infrastructure by integrating with the analytics tools like Splunk, Loggly, Sumo logic and etc.</span><o:p></o:p></span></div>
</div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">In this article we are going to explore topics like :</span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">How AWS CloudTrail can be integrated with Splunk App
for AWS ?</span></li>
<li><span style="font-family: Verdana, sans-serif;">How can we use it to operate our managed services efficiently ( with sample use cases answering following questions)</span></li>
<ul>
<li><span style="font-family: Verdana, sans-serif;">How do i get to know when security groups or Network details are changed ?</span></li>
<li><span style="font-family: Verdana, sans-serif;">How do i know when EC2 instances are launched, terminated etc ?</span></li>
<li><span style="font-family: Verdana, sans-serif;">How do i know what activities are carried on the Amazon Elastic IP's ?</span></li>
<li><span style="font-family: Verdana, sans-serif;">What activities and changes happened in the IAM policies over the period ?</span></li>
<li><span style="font-family: Verdana, sans-serif;">What are the Rare errors related to AWS account activities that can be tracked ?</span></li>
</ul>
</ul>
</div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif; font-size: large;"><span style="line-height: 18px;"><b>How AWS CloudTrail + Splunk Integration works ?</b></span></span><br />
<span style="font-family: Verdana, sans-serif;"><span style="line-height: 18px;"><b><br /></b></span></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpd7DBPym1CVds_Nhc9eLY58JZ4pnx4Hk941UI60zihoIS9nbmQnId3QV7f2LlVvUBPGEQZ89tzUJo5cAG6py4xca3PnIidYWunQlYtzW6h_J7mJds5GdcQ9TCC17sSpJWMy2kASkDTaEb/s1600/1-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpd7DBPym1CVds_Nhc9eLY58JZ4pnx4Hk941UI60zihoIS9nbmQnId3QV7f2LlVvUBPGEQZ89tzUJo5cAG6py4xca3PnIidYWunQlYtzW6h_J7mJds5GdcQ9TCC17sSpJWMy2kASkDTaEb/s640/1-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="506" /></a></div>
<br /></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Currently CloudTrail logs API calls on the following AWS services and
in future you can expect more services to be launched. The current list is as follows :</span></div>
<div class="Textbody" style="margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 35.35pt; margin-right: 0in; margin-top: 0in; mso-list: l0 level1 lfo1; tab-stops: 35.35pt; text-indent: -14.15pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt;">
</span><!--[endif]--><a href="http://aws.amazon.com/ec2/" target="_self"><span class="InternetLink">Elastic Compute Cloud</span></a> (EC2) <o:p></o:p></span></div>
<div class="Textbody" style="margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 35.35pt; margin-right: 0in; margin-top: 0in; mso-list: l0 level1 lfo1; tab-stops: 35.35pt; text-indent: -14.15pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt;">
</span><!--[endif]--><a href="http://aws.amazon.com/ebs/" target="_self"><span class="InternetLink">Elastic Block Store</span></a> (EBS) <o:p></o:p></span></div>
<div class="Textbody" style="margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 35.35pt; margin-right: 0in; margin-top: 0in; mso-list: l0 level1 lfo1; tab-stops: 35.35pt; text-indent: -14.15pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt;">
</span><!--[endif]--><a href="http://aws.amazon.com/vpc/" target="_self"><span class="InternetLink">Virtual Private Cloud</span></a> (VPC) <o:p></o:p></span></div>
<div class="Textbody" style="margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 35.35pt; margin-right: 0in; margin-top: 0in; mso-list: l0 level1 lfo1; tab-stops: 35.35pt; text-indent: -14.15pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt;">
</span><!--[endif]--><a href="http://aws.amazon.com/rds/" target="_self"><span class="InternetLink">Relational Database Service</span></a> (RDS) <o:p></o:p></span></div>
<div class="Textbody" style="margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 35.35pt; margin-right: 0in; margin-top: 0in; mso-list: l0 level1 lfo1; tab-stops: 35.35pt; text-indent: -14.15pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt;">
</span><!--[endif]--><a href="http://aws.amazon.com/iam/" target="_self"><span class="InternetLink">Identity and Access Management</span></a> (IAM) <o:p></o:p></span></div>
<div class="Textbody" style="margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 35.35pt; margin-right: 0in; margin-top: 0in; mso-list: l0 level1 lfo1; tab-stops: 35.35pt; text-indent: -14.15pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt;">
</span><!--[endif]--><a href="http://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html" target="_self"><span class="InternetLink">Security Token Service</span></a> (STS) <o:p></o:p></span></div>
<div class="Textbody" style="margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 35.35pt; margin-right: 0in; margin-top: 0in; mso-list: l0 level1 lfo1; tab-stops: 35.35pt; text-indent: -14.15pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;">·<span style="font-size: 7pt;">
</span><!--[endif]--><a href="http://aws.amazon.com/redshift/" target="_self"><span class="InternetLink">Redshift</span></a> <o:p></o:p></span></div>
<div class="Textbody" style="margin-left: 35.35pt; mso-list: l0 level1 lfo1; tab-stops: 35.35pt; text-indent: -14.15pt;">
<!--[if !supportLists]--><span style="font-family: Verdana, sans-serif;"><span class="StrongEmphasis">·<span style="font-size: 7pt;">
</span></span><!--[endif]--><a href="https://aws.amazon.com/cloudtrail/" target="_self"><span class="InternetLink"><span style="mso-bidi-font-family: Calibri; text-decoration: none; text-underline: none;">CloudTrail</span></span></a><span class="StrongEmphasis"> </span><span class="StrongEmphasis"><o:p></o:p></span></span></div>
<div class="Textbody" style="margin-left: 35.35pt;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">AWS CloudTrail captures AWS API calls made by or on behalf of an AWS
account. Once the API calls are logged on files, it delivers these log files to
an Amazon S3 bucket that you specify. Each call detail is logged in JSON
format. Usually the JSON file is zipped and delivered to the bucket. SNS
notification can be enabled to operations team when the log file is delivered.</span></div>
<div class="Textbody">
<span style="font-family: Verdana, sans-serif;">Sample format of the log file nomenclature is shown below:<o:p></o:p></span></div>
<div class="PreformattedText">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; line-height: 115%; mso-bidi-font-family: Calibri;">https://s3-us-west-2.amazonaws.com/<bucket_name>/<prefix_name>/AWSLogs/<AccountID>/CloudTrail/<o:p></o:p></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; line-height: 115%; mso-bidi-font-family: Calibri;">us-west-2/2013/10/25/<AccountID>_CloudTrail_us-west-2_2014-11-25T12:25Z_bPlKAoJtDqSWWlTT.json.gz<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><b><span style="font-size: large;">How to activate AWS CloudTrail ?</span></b><b><o:p></o:p></b></span></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">By default CloudTrail is not enabled in your AWS account. To activate
follow the below steps<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">1. Open the CloudTrail console in the AWS Management console<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">2. Click Get Started<o:p></o:p></span></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">3. On the Turn on CloudTrail page, you will be presented with the
option to create a new Amazon S3 bucket or using an existing Amazon S3 bucket. In
this article, I have selected the option to create new S3 bucket and provided
the Bucket name as cloudtraillogs-splunk as illustrated below.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: red; font-family: Verdana, sans-serif;"><b><br /></b></span></div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzyLkqeHyVXljC5cfmtka8rKbw2SYJE4VqgBQClISDy4MNyKAuiit4bdrO36f6MsjPvIpVOV9roPEBtefPGtfZ4qLMd-OrUKaTPM63yv5HlbHWI8vzJ00bX4g4_5iGRlZLFcVIiKEUHwoM/s1600/2-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="202" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzyLkqeHyVXljC5cfmtka8rKbw2SYJE4VqgBQClISDy4MNyKAuiit4bdrO36f6MsjPvIpVOV9roPEBtefPGtfZ4qLMd-OrUKaTPM63yv5HlbHWI8vzJ00bX4g4_5iGRlZLFcVIiKEUHwoM/s640/2-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;">4. If
you need to configure SNS notification and other settings, Click the option
“Advanced”. Once clicked, below options will be displayed and these options are
optional and not mandatory. I have enabled the option of SNS notification for
every log delivery to our operations team.</span></div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjE8QGB3CuwPE2clau9qCuejc6GQiZt_NhMMKMg4swg3cLtve6yoxXgc-kNP4Iy9sM3a2q-p5dUyuxjZ050HcRUcwTJS4JzSuWoro08ntGFebovgAAmrEF2SUSd1Vmt3sMCBDZHPhWMgxfi/s1600/3-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="228" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjE8QGB3CuwPE2clau9qCuejc6GQiZt_NhMMKMg4swg3cLtve6yoxXgc-kNP4Iy9sM3a2q-p5dUyuxjZ050HcRUcwTJS4JzSuWoro08ntGFebovgAAmrEF2SUSd1Vmt3sMCBDZHPhWMgxfi/s640/3-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<b style="font-family: Verdana, sans-serif;">Log
file prefix:</b><span style="font-family: Verdana, sans-serif;"> Enter a prefix for the Amazon S3 bucket. It is usually
not needed since AWS automatically prefixes path automatically. You can see
that path when you select the “View log file location”. But if you want Pattern
for identification you can go ahead with the prefix.</span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><b>Select
Yes or No to “Include global services”?</b> IAM and AWS STS are the global services so if
you want to log activities of those services as well, please select “Yes”. It
is recommend to Select them because you can review the actions of the IAM.<o:p></o:p></span></div>
<div class="Textbody">
<span style="font-family: Verdana, sans-serif;"><b>SNS notification
for every log files delivery? </b> If you
want delivery, please enter the SNS topic name. The entered topic will be
created in the SNS service and you must subscribe to it. After Saving, Click
Subscribe. Once it is done, the service is activated and the log files are generated.<o:p></o:p></span></div>
<div class="MsoNormal">
</div>
<div class="Textbody">
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">5. You can also edit the CloudTrail configuration and Turn off CloudTrail
using the console. The console is illustrated in the below screen Shot.</span></div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbOko29A15-OswnnC4EyAJSE1TZYQbx_7-kMXmM3a714_6MKydJ8JqzRh_IfcMzP3Hb5O5_9fcQryYXNrawsNa5yyvNQaUMxU7wj7XEMKP1Lg68FUglOcsCfpkB3FHxhQuVpY6SYYlrhxT/s1600/4-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="202" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbOko29A15-OswnnC4EyAJSE1TZYQbx_7-kMXmM3a714_6MKydJ8JqzRh_IfcMzP3Hb5O5_9fcQryYXNrawsNa5yyvNQaUMxU7wj7XEMKP1Lg68FUglOcsCfpkB3FHxhQuVpY6SYYlrhxT/s640/4-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<b style="font-family: Verdana, sans-serif;"><br /></b>
<b style="font-family: Verdana, sans-serif;"><span style="font-size: large;">What is Splunk App for AWS ?</span></b></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">Splunk App for AWS is designed to take the data from the AWS
CloudTrail and provide the results using built-in dashboards. It is provided as
a plug-in. Operations / Managed Services team can view this app and monitor and review some activities efficiently.<o:p></o:p></span></div>
<div class="MsoNormal">
<b><span style="font-family: Verdana, sans-serif;"><br /></span></b>
<b><span style="font-family: Verdana, sans-serif; font-size: large;">What is
interesting with Splunk App for AWS?</span></b><br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">It gives important insights into security-related activity such as unauthorized access attempts, simultaneous logins from geographically disparate locations and frequent changes to access control privileges. You can ensure security and compliance with continuous monitoring and a full audit trail of user activity using the Splunk App.</span></li>
<li><span style="font-family: Verdana, sans-serif;">It provides option to configure email alerts based on the schedule</span></li>
<li><span style="font-family: Verdana, sans-serif;">It provides easy option to monitor events, generate reports and export the reports and Search for events.</span></li>
</ul>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif; font-size: large;"><b>Steps to Integrate Splunk with AWS Cloud Trail :</b></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><b><br /></b></span></div>
</div>
<div class="Textbody">
<span style="font-family: Verdana, sans-serif;"><b>Stage 1: Installing Splunk on AWS:</b><b><o:p></o:p></b></span></div>
<div class="Textbody">
<span style="font-family: Verdana, sans-serif;">In order to use the Splunk App for AWS for this article, we are using the
Enterprise Splunk AMI for illustrating the use cases.<o:p></o:p></span></div>
<div class="Textbody">
<span style="font-family: Verdana, sans-serif;">1. Launch an EC2 Linux instance and based on the choice of Operating
system download the Enterprise Splunk from the link
http://www.splunk.com/download?r=header<o:p></o:p></span></div>
<div class="MsoNormal">
</div>
<div class="Textbody">
<span style="font-family: Verdana, sans-serif;">2. Install the Splunk Enterprise<o:p></o:p></span></div>
<div class="Textbody">
<span style="color: red; font-family: Verdana, sans-serif;"><br /></span></div>
<div class="Textbody">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigHD6ZsNB8GqRW7tcsjCY9NnQfROm4QByd8MFlxlZwZVuoqeg0IZDVodcW6WIVjZ6RrVPNEkJeMSy9kpWG5LFq__x8fAQ6ARDNTqz3O__nGFfBgGFsk962vs7dT0JIpdaFmYYbe55n0ELJ/s1600/5-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="90" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigHD6ZsNB8GqRW7tcsjCY9NnQfROm4QByd8MFlxlZwZVuoqeg0IZDVodcW6WIVjZ6RrVPNEkJeMSy9kpWG5LFq__x8fAQ6ARDNTqz3O__nGFfBgGFsk962vs7dT0JIpdaFmYYbe55n0ELJ/s640/5-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<br /></div>
<div class="Textbody">
<span style="color: red; font-family: Verdana, sans-serif;"><br /></span></div>
<div class="Textbody">
<span style="font-family: Verdana, sans-serif;">3. Export the following environmental values and then start the Splunk
service.<o:p></o:p></span></div>
<div class="PreformattedText">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; line-height: 115%; mso-bidi-font-family: Calibri;"># export
SPLUNK_HOME=/opt/splunk<o:p></o:p></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; line-height: 115%; mso-bidi-font-family: Calibri;"># export PATH=$SPLUNK_HOME/bin:$PATH$ <o:p></o:p></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; line-height: 115%; mso-bidi-font-family: Calibri;">$ splunk start<o:p></o:p></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; line-height: 115%; mso-bidi-font-family: Calibri;">Commands to Restart and Check Status are below.<o:p></o:p></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; mso-bidi-font-family: Calibri;">$ splunk restart<o:p></o:p></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; mso-bidi-font-family: Calibri;">$ splunk status<o:p></o:p></span></div>
<div class="Textbody">
</div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; line-height: 115%; mso-bidi-font-family: Calibri;">4. On successful start the Spunk will be listening on the port 8000.
Access the Splunk using the link http://<EC2 Public DNS>:8000. The below screen
will be presented<o:p></o:p></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNOdIaQ_DFn1s3gAxxW0UkK25JB3jn6Y96pBXJ-aCkdYrTfHiBnAYhvKpKNc4djbuY6XtEoMAwLGDzXW4r9SucFi01j1xAKK-s3rOmGtKiag_ngOfo2KE-i06h7xZJmVEkytRo7QjNq7hI/s1600/6-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="312" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNOdIaQ_DFn1s3gAxxW0UkK25JB3jn6Y96pBXJ-aCkdYrTfHiBnAYhvKpKNc4djbuY6XtEoMAwLGDzXW4r9SucFi01j1xAKK-s3rOmGtKiag_ngOfo2KE-i06h7xZJmVEkytRo7QjNq7hI/s640/6-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<br /></div>
<div class="Textbody">
<b><span style="font-family: Verdana, sans-serif;">Stage 2: Installing and Configuring Splunk App for AWS:<o:p></o:p></span></b></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12.0pt; mso-bidi-font-family: Calibri;">Splunk App for AWS fetches the data from the Amazon S3 bucket and provides
the report in the Splunk GUI. Following are the steps to integrate Splunk app plugin with AWS Cloud Trail<o:p></o:p></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif;"><span style="font-size: 12.0pt; mso-bidi-font-family: Calibri;">1. Download the Splunk App for AWS from the link </span><span style="color: windowtext; font-size: 12.0pt; mso-bidi-font-family: Calibri; text-decoration: none; text-underline: none;"><a href="http://apps.splunk.com/app/1274/">http://apps.splunk.com/app/1274/</a></span></span><br />
<span style="font-family: Verdana, sans-serif; font-size: 12pt;">2. Log into Splunk Enterprise and select the Manage Apps from the App
menu</span><br />
<span style="font-family: Verdana, sans-serif; font-size: 12pt;">3. Click Install app from the file button.</span><br />
<span style="font-family: Verdana, sans-serif; font-size: 12pt;">4. On the Upload a file screen and Browse to locate the downloaded
[app].tar.gz file, click Open and Upload</span><br />
<span style="font-family: Verdana, sans-serif; font-size: 12pt;">5. Click Restart Splunk, confirm to restart.</span><br />
<span style="font-family: Verdana, sans-serif; font-size: 12pt;">6. After Splunk restarted, login again, you can find the below screen.</span><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUdV34wXwdHLit7edRyCSEPJgEKI6fOftxqxVoJm81C8WNwQiBxJDzyvVOqdpnYHi2EtGHn20p5sF7K-yZV7uF6ez_6lgMRswBlZ4nAUbDE7HZPYSXr3f-6hHe2EOKLSScDg9w_yhIF1Ez/s1600/7-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" height="304" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUdV34wXwdHLit7edRyCSEPJgEKI6fOftxqxVoJm81C8WNwQiBxJDzyvVOqdpnYHi2EtGHn20p5sF7K-yZV7uF6ez_6lgMRswBlZ4nAUbDE7HZPYSXr3f-6hHe2EOKLSScDg9w_yhIF1Ez/s640/7-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12pt;">7.From Settings in the top right corner of the page Select Data input,
In that you can see an option to add AWS CloudTrail Log settings.</span><br />
<span style="font-family: Verdana, sans-serif;">8. Click Add new and you will be asked for the following mandatory details.</span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<ul>
<li><span style="font-family: Verdana, sans-serif;">AWS Access key</span></li>
<li><span style="font-family: Verdana, sans-serif;">AWS Secret key</span></li>
<li><span style="font-family: Verdana, sans-serif;">AWS SQS name</span></li>
<li><span style="font-family: Verdana, sans-serif;">AWS SQS region</span></li>
</ul>
<span style="font-family: Verdana, sans-serif;"></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixTbwNSAdHURWTebRvV10zkd-30bfsSP6cnhAV-rYxIw7d6fO36gwII33ga01hgokOoLcVA1nIHUIma80gx8PcurFYRfIIw5yV_KOHsiz-Fxna8oxb2Vqp0FsyvgUAZ0Zu7KoM9xnzfVLD/s1600/8-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixTbwNSAdHURWTebRvV10zkd-30bfsSP6cnhAV-rYxIw7d6fO36gwII33ga01hgokOoLcVA1nIHUIma80gx8PcurFYRfIIw5yV_KOHsiz-Fxna8oxb2Vqp0FsyvgUAZ0Zu7KoM9xnzfVLD/s640/8-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<b><span style="font-family: Verdana, sans-serif;">SQS Creation for topics</span></b><br />
<span style="font-family: Verdana, sans-serif; font-size: 12pt;">In order to use Splunk App, CloudTrail logging should be configured to
send SNS topic and that topic should point to the Amazon SQS queue. You can see
cloudtrail-spunkalert SNS topic is subscribed to the SQS queue Cloud Trail’s
endpoint.</span><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9b5fIvD2L991snZWQkKgUydjI6a6IjWL1SO_1ViJsvrHzHCv0CHrUeCPzpC-v1U4U7NOAOnk5TJ1ySPUbX4uveNtkAq9Wp5hNSGj50oBqhkIxdrDvAy1sDIYw3z2zra0cz5F9aQXdzk70/s1600/9-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9b5fIvD2L991snZWQkKgUydjI6a6IjWL1SO_1ViJsvrHzHCv0CHrUeCPzpC-v1U4U7NOAOnk5TJ1ySPUbX4uveNtkAq9Wp5hNSGj50oBqhkIxdrDvAy1sDIYw3z2zra0cz5F9aQXdzk70/s640/9-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif; font-size: 12pt; line-height: 115%;">SQS queue should be created manually with the proper permissions to
process messages. We have created a queue named CloudTrail in SQS. Below screenshot details the CloudTrail SQS queue:</span><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggaBgUIPDfWXCXe3MCdAqpxEKLC4zsExmfaWP3ezv1z2hzrmyF9_cYd4vdYGTG3otDbpyjKKGZcLJdv5SqyS5xKMouuSbENwZV3RJWE4BnNkixwgN7bLvrF4DASKb3y81i8L9RPgTY_Tk7/s1600/10-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" height="94" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggaBgUIPDfWXCXe3MCdAqpxEKLC4zsExmfaWP3ezv1z2hzrmyF9_cYd4vdYGTG3otDbpyjKKGZcLJdv5SqyS5xKMouuSbENwZV3RJWE4BnNkixwgN7bLvrF4DASKb3y81i8L9RPgTY_Tk7/s640/10-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<span style="font-family: Verdana, sans-serif;">The above step completes the configuration.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif; font-size: large;"><b>What we can monitor and review using CloudTrail + Splunk ?</b></span><br />
<span style="font-family: Verdana, sans-serif;">The Splunk App for AWS has built in dashboard with lot of options to review Network Actions, IAM Actions, EC2 options etc. Now lets go through some of the common review activities the IT operations/ Managed Services team can perform using the AWS CloudTrail and Splunk Integration:</span></div>
<div class="MsoNormal">
<b style="font-family: Verdana, sans-serif; line-height: 18px;">Case 1: Reviewing AWS Security Group related activities using AWS CloudTrail and Splunk:</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3v1guTUkRkcd1YDkzOTMbleuksi2yGG3Oj3ow6bUwMSyCFM87lp6Iw9e8gkc0FSJtU8vLkaSJ7ZfcKW5q2JQivDDjbwpfXdaT7h31czMwleidSjpHJKlefTMv7iiZi5XS1BF1qRLRBYm-/s1600/11-+AWS+CloudTrail+Splunk+Network+security+group+actions.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="140" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3v1guTUkRkcd1YDkzOTMbleuksi2yGG3Oj3ow6bUwMSyCFM87lp6Iw9e8gkc0FSJtU8vLkaSJ7ZfcKW5q2JQivDDjbwpfXdaT7h31czMwleidSjpHJKlefTMv7iiZi5XS1BF1qRLRBYm-/s640/11-+AWS+CloudTrail+Splunk+Network+security+group+actions.jpg" width="640" /></a></div>
<div>
<span style="font-family: Verdana, sans-serif; line-height: 18px;">From the above Screen shots , we can infer that a new Security group was created and authorized. The Audit details pertaining to Security group + additional details like IAM User, AWS Account, Source IP address and Time stamp of this activity is captured by AWS CloudTrail and presented in Splunk. The Operations/ Managed Services team can constantly monitor for such changes in the infrastructure and take actions accordingly.</span></div>
<div>
<br /></div>
<span style="font-family: Verdana, sans-serif; line-height: 18px;"><b>Case 2: Reviewing Amazon EC2 related activities using AWS CloudTrail and Splunk:</b></span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9voohgMXcnmh3idDWq5dVNfpuOCm-MTOGujIG8Y8Lzy35otPk91fJNz3uzdEoJElnNNF632u3tDxoK-86lwE1PMjiGDhud0ejaKnMCxFE89EANtqVkuScaEsdAGjTg6gg48rPCku8cxy9/s1600/12-+AWS+CloudTrail+Splunk+EC2+related+actions.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="76" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9voohgMXcnmh3idDWq5dVNfpuOCm-MTOGujIG8Y8Lzy35otPk91fJNz3uzdEoJElnNNF632u3tDxoK-86lwE1PMjiGDhud0ejaKnMCxFE89EANtqVkuScaEsdAGjTg6gg48rPCku8cxy9/s640/12-+AWS+CloudTrail+Splunk+EC2+related+actions.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0qZln2F2fj2VO5tljZM_1eVC5CpsxcQCfZAjoP1UYkqFNKb0Eoz7syV2rlquvLlRs4ntyefvb_roytmeEG2VlpTo207x3O5TVI1ixzONEAfB5EUGf-buhXup4L6PGJ1HI8-WcvaERv0vG/s1600/13-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0qZln2F2fj2VO5tljZM_1eVC5CpsxcQCfZAjoP1UYkqFNKb0Eoz7syV2rlquvLlRs4ntyefvb_roytmeEG2VlpTo207x3O5TVI1ixzONEAfB5EUGf-buhXup4L6PGJ1HI8-WcvaERv0vG/s640/13-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif; line-height: 18px;">From the above Screen shots , we can infer that </span><span style="font-family: Verdana, sans-serif; line-height: 18px;">Start, Stop and Terminate EC2 Instance activities are carried out in the AWS Account in the specified period. The EC2 Instance actions are noted wi</span><span style="font-family: Verdana, sans-serif; line-height: 18px;">th IAM User name and time stamp for monitoring purpose.</span><br />
<span style="font-family: Verdana, sans-serif;"><b>Case 3: Rare errors pertaining the AWS account can be reviewed </b></span><b style="font-family: Verdana, sans-serif; line-height: 18px;">using AWS CloudTrail and Splunk:</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPZxX6sM12zkZJXucbitmu8TQ9uDPCUOBbS9qTeIwd0seuMx5jvrFTSDeN4J1ZpQspOJtjTymDIGxMXkgZiZqDfHgVeAoAJ55IeGXFmoD6gTJqe2shGPioPzqB1q_76Xy06AlXpMEsH1bv/s1600/14-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="160" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPZxX6sM12zkZJXucbitmu8TQ9uDPCUOBbS9qTeIwd0seuMx5jvrFTSDeN4J1ZpQspOJtjTymDIGxMXkgZiZqDfHgVeAoAJ55IeGXFmoD6gTJqe2shGPioPzqB1q_76Xy06AlXpMEsH1bv/s640/14-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif; line-height: 18px;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Case 4: Reviewing AWS IAM related activities </b></span><b style="font-family: Verdana, sans-serif; line-height: 18px;">using AWS CloudTrail and Splunk:</b><br />
<span style="font-family: Verdana, sans-serif; line-height: 18px;">AWS IAM Activities like CreateUser, DeleteUser, DeleteLoginProfile etc are tracked by AWS CloudTrail as logs. These activities are presented to the IT Ops/MS team on the IAM actions module of Splunk. The diagrams representing the same is listed below:</span></div>
<div class="PreformattedText" style="margin-bottom: 14.15pt;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg9Wi4_vGRJnMXj-Qi4bnEB-_muTj32w530ublYOG45by1kmEp4BWdz52fl3NGAGlkkZG0JPWBpwGnMnZZn1e8iyDSS0DvywQJV1XKapO3KOEZ9Eq2XgPxUeUOK1oR-goyMUTmX5YCEeDT/s1600/15-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="64" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg9Wi4_vGRJnMXj-Qi4bnEB-_muTj32w530ublYOG45by1kmEp4BWdz52fl3NGAGlkkZG0JPWBpwGnMnZZn1e8iyDSS0DvywQJV1XKapO3KOEZ9Eq2XgPxUeUOK1oR-goyMUTmX5YCEeDT/s640/15-+AWS+CloudTrail+Splunk+for+Managed+Services.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfUZ0lubc8YYsQFH2dwRu-hFxBJK3jqvZZjQFIPWqkjy6ZLWi8n_nhAOgRb6eseA8d38yA2MHufIynkTLAmCqwBe5hy1eiClvQnD4j1mEPpW9D8xr1vuK-854o3AHQATCb2QThBh6GzwXQ/s1600/16-+AWS+CloudTrail+Splunk+Amazon+IAM+Actions.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="188" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfUZ0lubc8YYsQFH2dwRu-hFxBJK3jqvZZjQFIPWqkjy6ZLWi8n_nhAOgRb6eseA8d38yA2MHufIynkTLAmCqwBe5hy1eiClvQnD4j1mEPpW9D8xr1vuK-854o3AHQATCb2QThBh6GzwXQ/s640/16-+AWS+CloudTrail+Splunk+Amazon+IAM+Actions.jpg" width="640" /></a></div>
<br />
<span style="font-family: Verdana, sans-serif;"><b>Case 5: Reviewing Amazon ElasticIP related activities </b></span><b style="font-family: Verdana, sans-serif; line-height: 18px;">using AWS CloudTrail and Splunk:</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh80G2We_-3qqizdT7g3XRYwop7eqdqZDrzZJT3S1aoiqrmwYDZ2lqixGMK0CZfI6FTIJt0L_jaG_cn2v-byYZCbj8zZiE74SrrntfYPfPm6tuIQ7mHJ07-LhDJsa6ml3qfX68TqKynLcFm/s1600/17-+AWS+CloudTrail+Splunk+Amazon+EIP+related+actions.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="54" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh80G2We_-3qqizdT7g3XRYwop7eqdqZDrzZJT3S1aoiqrmwYDZ2lqixGMK0CZfI6FTIJt0L_jaG_cn2v-byYZCbj8zZiE74SrrntfYPfPm6tuIQ7mHJ07-LhDJsa6ml3qfX68TqKynLcFm/s640/17-+AWS+CloudTrail+Splunk+Amazon+EIP+related+actions.jpg" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;">From the above screen shot, We can infer that One EIP Allocate Address activity and One EIP Release Address 107.20.189.161 is carried out during the specified period. The Action, Time stamp , IAM user Name and AWS Account details are captured and can be used for audit purpose.</span><br />
<div>
<b style="font-family: Verdana, sans-serif;"><br /></b></div>
<div>
<b style="font-family: Verdana, sans-serif;">Case 6: View all the events occurred on your AWS Account as Graph</b></div>
<div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijKIjOmsmroUUR4RCB6u3cZI-_2KpBtJQWwOfaj-F3IFs8sRI0FYv0IXvqbZGL9pE3kSKr37PGBJomt9egWDA_IgaV5unVcYh9JYL3g5Qy5RfqGexwCcRmwcgQrzk-G5Gf69aT70IS3hyS/s1600/18-+AWS+CloudTrail+Splunk+events+type+in+graph.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="314" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijKIjOmsmroUUR4RCB6u3cZI-_2KpBtJQWwOfaj-F3IFs8sRI0FYv0IXvqbZGL9pE3kSKr37PGBJomt9egWDA_IgaV5unVcYh9JYL3g5Qy5RfqGexwCcRmwcgQrzk-G5Gf69aT70IS3hyS/s640/18-+AWS+CloudTrail+Splunk+events+type+in+graph.jpg" width="640" /></a></div>
<br /></div>
</div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><b>Case 7: Query your AWS CloudTrail logs for viewing activities and patterns<o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAuxF0KdQcfv3Z4iX5nNYdZUuB1kfW0dDfcG7huTl-aFeqi6M1lTtNk51ZmfLNTQUUIOCsqFREfE37XxZy6VGyxrR-EMYPr_WDhD_XbI8ZuvhyphenhyphenYZZL32w5Bteg_I24q-pWqI2aOylCuM5G/s1600/19-+AWS+CloudTrail+Querying+using+splunk.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="226" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAuxF0KdQcfv3Z4iX5nNYdZUuB1kfW0dDfcG7huTl-aFeqi6M1lTtNk51ZmfLNTQUUIOCsqFREfE37XxZy6VGyxrR-EMYPr_WDhD_XbI8ZuvhyphenhyphenYZZL32w5Bteg_I24q-pWqI2aOylCuM5G/s640/19-+AWS+CloudTrail+Querying+using+splunk.jpg" width="640" /></a></div>
<span style="font-family: Verdana, sans-serif;">Splunk app provides the capability to query the AWS CloudTrail logs for viewing specific or overall actions carried out during specified period. This is a powerful feature used by operations/managed services team to view the events and take actions accordingly. </span><br />
<br />
<span style="font-family: Verdana, sans-serif;"><b>Please talk to us to understand more about our Managed Services offerings and how we can help your cloud infrastructure. <a href="http://in.linkedin.com/in/harishganesan/">Contact</a></b></span><br />
<br />
<b style="font-family: Verdana, sans-serif;">This article was co authored with <a href="http://in.linkedin.com/in/sent2020">Senthil</a>.</b></div>
</div>
</div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com2tag:blogger.com,1999:blog-4836510665235566234.post-10532950351878456462013-10-22T10:11:00.001-07:002013-10-22T10:13:07.971-07:00AWS Cost Saving Tip 17: Add Amazon RDS Read Replica's on need basis and save costs<span style="font-family: Verdana, sans-serif;">Many online applications are usually Read intensive in nature. Though the usage of Cache, NoSQL, CDN are becoming part of the core architecture in many such applications, still most of them are heavily dependent upon Traditional RDBMS (like MySQL, Oracle) for serving their requests.</span><br />
<span style="font-family: Verdana, sans-serif;">Architects usually design their DB tier with Master + Read Slaves model to cater the read intensive needs of the application. Writes and Immediate Write + Reads are directed to Master and all other Read queries are balanced between Read Slaves. Translating this model to AWS infrastructure means, using RDS MySQL Master + Multiple Read Replica Slaves.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; line-height: 18px;">To know more about Load balancing Amazon RDS Read Replica's </span><u style="background-color: white; color: #444444; font-family: Verdana, sans-serif; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/08/Load-balancing-Amazon-RDS-MySQL-read-replica-slaves-using-HAProxy.html">Click here</a></u><span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtjq50IVF2Y0rIEZzZYSiwMHp3r5SSQ0utbZ12-14_u78EvirsRebAGXNbZr5WnLA1CjNmDMWTMJP6t9gLk8plaBg8c4QxW_3Goyg7rqy2yb20kqIb5PjtlLHmT0Scxpg3kRsCkwvfYFrB/s1600/Add+Amazon+RDS+MySQL+Read+Replica+elastically+and+save+costs.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Verdana, sans-serif;"><img border="0" height="164" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtjq50IVF2Y0rIEZzZYSiwMHp3r5SSQ0utbZ12-14_u78EvirsRebAGXNbZr5WnLA1CjNmDMWTMJP6t9gLk8plaBg8c4QxW_3Goyg7rqy2yb20kqIb5PjtlLHmT0Scxpg3kRsCkwvfYFrB/s640/Add+Amazon+RDS+MySQL+Read+Replica+elastically+and+save+costs.jpg" width="640" /></span></a></div>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Scenario: Weekly load volatility impacts: </b></span><br />
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;">Online Ticketing and Travel companies are usually prone to weekly fluctuations in their load pattern. Imagine during peak load they need five M2.4XLarge RDS MySQL Read Replica's to serve their read intensive site.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">As an Infra Architect you have two options in the Elastic AWS world: </span></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><u>Option 1:</u> Run your DB tier with peak capacity all days of the week. ( May be suitable for Traditional infra environments, but a bad idea in AWS infra)</span></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><u>Option 2:</u> Elastically increase the number of RDS Read Replica instances during the peak period and decrease the number of instances once the peak period is over. </span><span style="font-family: Verdana, sans-serif;">(AWS is built for leveraging such cases)</span></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">Number of EC2 Instances<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">Number of days in a week<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">EC2 Instance Type<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">Weekly($)<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">Monthly($)<o:p></o:p></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">5<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">7<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">M2.4XLarge<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">1860<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">7442<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">1860<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">7442<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">2<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">4<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">M2.4XLarge<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">425<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">1701<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">5<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">3<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">M2.4XLarge<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">797<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">3189<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">1222<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">4890<span style="font-size: small;"><o:p></o:p></span></span></span></b></div>
</td>
</tr>
</tbody></table>
<span style="font-family: Verdana, sans-serif;"><br /></span><span style="font-family: Verdana, sans-serif;">Cost for Option 1: Running all 5 of them for a Week costs : <b>$1860</b></span><br />
<span style="font-family: Verdana, sans-serif;">Cost for Option 2: Running with 2 RR normally for 4 days a week and elastically increasing to Five M2.4Xlarge during the peak days(i.e. three days a week) will cost :<b> $1222</b></span></div>
<div class="MsoNormal" style="background-color: white; color: #444444; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<b><span style="font-family: Verdana, sans-serif;"><span style="color: #444444;">Savings: From the above analysis you can observe that elastically increasing/Decreasing Amazon RDS RR based on your load you can achieve </span>~30<span style="color: red;"> </span><span style="color: #444444;">% cost savings in a week for the mentioned use case.</span></span></b><br />
<b><span style="font-family: Verdana, sans-serif;"><span style="color: #444444;"><br /></span></span></b></div>
<div class="MsoNormal" style="background-color: white; color: #444444; line-height: 18px;">
<div>
<span style="font-family: Verdana, sans-serif;">To know more about "How Amazon Auto Scaling can save costs on your Web/App Tier ? </span><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html" style="font-family: Verdana, sans-serif;">Click here</a><span style="font-family: Verdana, sans-serif;"> "</span><br />
<b><span style="font-family: Verdana, sans-serif;"><br /></span></b></div>
</div>
<span style="font-family: Verdana, sans-serif;"><b>Scenario: Impact during Campaigns </b></span><br />
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;">Imagine your online business demands big campaigns once every quarter. The campaign usually runs for 2 weeks and traffic increases 4-5X during these times. During normal period you run 1 RDS Read replica and during campaigns you need 5 RDS Read replicas. </span></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">If the campaign is planned once a year during holiday season, companies lease hardware, but if the campaign is every quarter or more frequent it becomes irrelevant to lease-build-maintain peak capacity.</span><br />
<br /></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<div class="MsoNormal" style="background-position: initial initial; background-repeat: initial initial; line-height: 13.5pt; margin-bottom: 0.0001pt;">
<br /></div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">Number of EC2 Instances<span style="font-size: small;"><o:p></o:p></span></span></b></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">Number of Months<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">EC2 Instance Type<o:p></o:p></span></b></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="font-family: Verdana, sans-serif;">Monthly($)<o:p></o:p></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">5<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">10<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">M2.4XLarge<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: Verdana, sans-serif;">All times</span></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">98877<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: red;"><span style="font-family: Verdana, sans-serif;">98877<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">1<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">10<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">M2.4XLarge<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: Verdana, sans-serif;">Other times</span></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">16479<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">5<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">2*<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">M2.4XLarge<o:p></o:p></span></span></b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: Verdana, sans-serif;">During campaigns</span></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">16479<o:p></o:p></span></span></b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.75pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 95.8pt;" valign="top" width="128"><div class="MsoNormal" style="line-height: 13.5pt; margin-bottom: .0001pt; margin-bottom: 0in;">
<b><span style="color: #77933c;"><span style="font-family: Verdana, sans-serif;">32959<o:p></o:p></span></span></b></div>
</td>
</tr>
</tbody></table>
<div class="MsoNormal" style="background-position: initial initial; background-repeat: initial initial; line-height: 13.5pt; margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;">Instance Hours per day:
744<o:p></o:p></span></div>
<div class="MsoNormal" style="background-position: initial initial; background-repeat: initial initial; line-height: 13.5pt; margin-bottom: 0.0001pt;">
<span style="font-family: Verdana, sans-serif;">*Campaign weeks are
aggregated to months<o:p></o:p></span></div>
<span style="line-height: 115%;"><span style="font-family: Verdana, sans-serif;">US-Oregon region - $2.215 (M2.4Xlarge RDS RR pricing as
on posting)</span></span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;">Cost for Option 1: Running all 5 of them for a year costs : 98877 USD</span></div>
<div class="MsoNormal" style="background-color: white; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;">Cost for Option 2: Running with 1 RR on regular weeks and elastically increasing to Five M2.4Xlarge during the peak weeks will cost : 32959 USD a year</span></div>
<div class="MsoNormal" style="background-color: white; color: #444444; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div class="MsoNormal" style="background-color: white; color: #444444; line-height: 18px;">
<b><span style="font-family: Verdana, sans-serif;">Savings: From the above analysis you can observe that elastically increasing/Decreasing Amazon RDS RR you can achieve ~55 % cost savings in a year.</span></b></div>
<div class="MsoNormal" style="background-color: white; color: #444444; line-height: 18px;">
<b><span style="font-family: Verdana, sans-serif;"><br /></span></b></div>
<div class="MsoNormal" style="background-color: white; color: #444444; line-height: 18px;">
</div>
<div class="MsoNormal" style="background-color: white; color: #444444; line-height: 18px;">
<div class="MsoNormal">
<span style="font-family: Verdana, sans-serif;">For in depth analysis on Amazon RDS Second tier Read Replica's </span><u style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Exploring-Amazon-RDS-MySQL-Second-Tier-Read-Replica.html">Click here</a></u><span style="font-family: Verdana, sans-serif;"> </span></div>
</div>
<div class="MsoNormal" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Other Cost Saving Tips</b></span><br />
<span style="font-family: Verdana, sans-serif;"></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-SQS-Long-polling-Batch-requests.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Choosing-right-search-tier-Amazon-cloudsearch-Apache-Solr.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 2: How right search technology choice saves cost in AWS ?</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-Cloudfront-Price-class.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-ElastiCache-cluster-nodes.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-AutoScaling-termination-policy-ClosestToNextInstanceHour.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Object-expiration.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 7: Use Amazon S3 Object Expiration</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Reduced-Redundancy-storage-rrs-vs-std.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage</a> </span><span style="font-family: Verdana, sans-serif;"> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Delete-Amazon-EBS-snapshots-regularly-retention-period.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place</a></span><span style="font-family: Verdana, sans-serif;"> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-make-right-EBS-Volumes-Choice-PIOPS-vs-Standard.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs</a> </span><br />
<a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-elasticity-Amazon-Elastic-MapReduce-EMR.html" style="color: #4d469c; font-family: Verdana, sans-serif; text-decoration: none;">Cost Saving Tip 11: <span style="color: #0000ee;"><u>How elastic thinking saves cost in Amazon EMR Clusters ?</u></span> </a><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-Adding-Spot-EC2-with-Amazon-Elastic-MapReduce-EMR.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 12: <span style="color: #0000ee;"><u>Add Spot Instances with Amazon EMR</u></span></a> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-use-Amazon-Glacier-for-archives-and-save-costs.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 13: <span style="color: #0000ee;"><u>Use Amazon Glacier for archive data and save costs</u></span></a> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-deletion-in-Amazon-glacier.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 14: <span style="color: #0000ee;"><u>Plan your deletion in Amazon Glacier and avoid cost leakage</u></span></a> </span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 15: <span style="color: #0000ee;"><u>Plan your Amazon Glacier Retrieval strategy and avoid cost leakage</u></span></a> </span></div>
</div>
<div class="MsoNormal" style="background-color: white;">
<a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html" style="color: #4d469c; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">Cost Saving Tip 16: </a><span style="color: #0000ee; font-family: Verdana, sans-serif;"><span style="font-size: x-small; line-height: 18px;"><u><a href="http://harish11g.blogspot.in/2013/10/Amazon-Web-Services-Cost-Saving-Tips-periodically-remove-EBS-AMI-S3-EIP-ELB-and-save-costs.html">Periodically remove your unwanted AWS resources and save costs</a></u></span></span><span style="color: #444444; font-family: Verdana, sans-serif; font-size: x-small;"><span style="line-height: 18px;"> </span></span><b style="color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><span style="color: red;">(new)</span></b></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-9546064145847161332013-10-06T08:50:00.001-07:002013-10-06T08:50:22.537-07:00Integrate Jenkins and GitHub to trigger build process on code commit<div class="title fS lT" style="background-color: white; color: #333333; font-size: 33px; line-height: 1em; margin: 0px 0px 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;">In this post we can explore how to integrate GitHub repository with jenkins CI server. </span></div>
<div class="body hasMarkup" style="background-color: white; color: #333333; line-height: 18px; margin: 0px; padding: 0px;">
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">This blogpost talks about installing GitHub plugin on jenkins server, configuring GitHub repository on jenkins and creating service hooks on GitHub which will trigger build process as soon as someone’s commit new code to GitHub repository.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
</div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<strong><span style="font-family: Verdana, sans-serif;">INSTALL and CONFIGURE GITHUB PLUGIN ON JENKINS</span></strong></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">To install GitHub plugin, navigate to your Jenkins Dashboard, click on “Manage Jenkins” and select “Manage Plugins”.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/10013c673fa4c521d688858c687d587a/tumblr_inline_mtydpkP7821qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
</div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">On the plugins page, select “Available” tab and search for “GitHub plugin” under “External Site/Tool Integrations”.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/40b288b61b4fbef63f45b60a293d3cbc/tumblr_inline_mtydw5QMzt1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Install the plugin. This will restart Jenkins server.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/880681492ea771387b4c3f6ad51c5bbc/tumblr_inline_mtydx8iIHL1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><strong><br /></strong>Once done, navigate back to dashboard for “Manage Jenkins” and select “Configure System”. Under that, you might observe that Git is not installed.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/ad98795c95c63f3f177e868d099dbd73/tumblr_inline_mtye1bAuF01qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">If so, login into your EC2 instance and issue command :</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"># yum install git</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/8ff57d94a4cf7903ce37e7223c1ebd92/tumblr_inline_mtye2dbpql1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Once done, please go further down the page and provide your username and email details for GitHub account.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/213e8d08a51df8f319c51b0b82fe73f3/tumblr_inline_mtye4iFsIH1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">That’s it !! We have installed and configured our GitHub plugin on jenkins.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<strong><span style="font-family: Verdana, sans-serif;">CREATE JENKINS JOB AND CONFIGURE GITHUB REPOSITORY</span></strong></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">After installing GitHub plugin, it is time to create Jenkins job and integrate GitHub repository to it.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">On Jenkins dashboard, click on “New Job” and provide name to “Build a free style software project” job.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/be9eec8e2fbd85b4f4b60a931f561dec/tumblr_inline_mtyeddqJhC1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Provide the path of GitHub repository under “GitHub Project” and provide “repository URL” for “GIT” under “Source Code Management”.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/d4ba0de2a946096da2f7c06fec13a9b7/tumblr_inline_mtyegblEHT1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Lower down the page, don’t forget to enable “Build when a change is pushed to GitHub”.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/6f0f934ef5fb0142d178d150e80a9577/tumblr_inline_mtyehxmtc81qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Now, we have configured Jenkins job in such a way that whenever a change is committed to GitHub repository - this will trigger build process on Jenkins.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<strong><span style="font-family: Verdana, sans-serif;">CONFIGURE GITHUB SERVICE HOOK</span></strong></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">To trigger the build process on Jenkins, we need to configure service hook on GitHub. In order to achieve this, navigate to GitHub repository settings and configure Jenkins Hook URL for GitHub plugin.</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">The URL format : http://<jenkins-username>:<jenkins-password>@<Elastic-IP-Address>:8080/github-webhook/</span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/2cf6c482dcc102d94a2d56f6663f58fb/tumblr_inline_mtyeuqjmj31qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="font-size: 13px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">That’s it !! GitHub is now integrated with Jenkins. If any developer commits any code to GitHub repository, this will trigger the build process on Jenkins.</span></div>
<div style="margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">This article was originally posted @<a href="http://sanketdangi.com/post/62740311628/integrate-jenkins-github-trigger-build-process">http://sanketdangi.com/post/62740311628/integrate-jenkins-github-trigger-build-process</a></span></div>
</div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-84713628387977854082013-10-06T08:43:00.000-07:002013-10-06T08:51:23.096-07:00Installing Jenkins on Amazon EC2<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Jenkins is one of the most popular open source continuous integration tool. It is written in Java. If you want to read about jenkins before installing it, please do read <a href="https://wiki.jenkins-ci.org/display/JENKINS/Home" style="color: #6ca516; outline: none;" target="_blank" title="Jenkins Wiki">this</a> link.</span></div>
<div style="background-color: white; margin-bottom: 20px; padding: 0px;">
<span style="color: #333333; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;">Following post illustrates a simple </span><span style="background-color: transparent; line-height: 18px;"><span style="color: #333333; font-family: Verdana, sans-serif; font-size: x-small;">Installation of Jenkins on Amazon Linux EC2 Instance.</span></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<strong style="font-family: Verdana, sans-serif;">Environment Details :</strong></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Operating System : Amazon Linux 2013.03.01</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">AMI-ID : ami-3ffed17a</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Security Group Ports : 8080</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
</div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<strong><span style="font-family: Verdana, sans-serif;">Installation Steps :</span></strong></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">1. Please launch an Amazon Linux instance using Amazon Linux AMI.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">2. Login to your Amazon Linux instance.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">3. Become root using “sudo su -” command.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">4. Update your repositories</span></div>
<pre style="background-color: white !important; border-color: rgb(187, 187, 187) rgb(221, 221, 221) rgb(221, 221, 221) rgb(187, 187, 187); border-style: solid; border-width: 1px; font-size: 13px; line-height: 1.5; margin: 2em; overflow: auto; padding: 1em;"><span style="font-family: Verdana, sans-serif;"># yum update</span></pre>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/b90f26347b8295d0cb2089cdb0e71ecc/tumblr_inline_mtxvvsyvYq1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
</div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">5. Get Jenkins repository using below command</span></div>
<pre style="background-color: white !important; border-color: rgb(187, 187, 187) rgb(221, 221, 221) rgb(221, 221, 221) rgb(187, 187, 187); border-style: solid; border-width: 1px; font-size: 13px; line-height: 1.5; margin: 2em; overflow: auto; padding: 1em;"><span style="font-family: Verdana, sans-serif;">wget -O /etc/yum.repos.d/jenkins.repo <a class="external-link" href="http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo" rel="nofollow" style="color: #6ca516; outline: none; text-decoration: none;" target="_blank"></a><a href="http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo" style="color: #6ca516; outline: none; text-decoration: none;" target="_blank">http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo</a></span></pre>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/264c20beebf921dbe4fc992ad278eaad/tumblr_inline_mtxvz1CG0b1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">6. Get Jenkins repository key</span></div>
<pre style="background-color: white !important; border-color: rgb(187, 187, 187) rgb(221, 221, 221) rgb(221, 221, 221) rgb(187, 187, 187); border-style: solid; border-width: 1px; font-size: 13px; line-height: 1.5; margin: 2em; overflow: auto; padding: 1em;"><span style="font-family: Verdana, sans-serif;">rpm --import <a class="external-link" href="http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key" rel="nofollow" style="color: #6ca516; outline: none; text-decoration: none;" target="_blank"></a><a href="http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key" style="color: #6ca516; outline: none; text-decoration: none;" target="_blank">http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key</a></span></pre>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/1aedbc7eb3df481780802078581e99c1/tumblr_inline_mtxw1vsgux1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">7. Install jenkins package</span></div>
<pre style="background-color: white !important; border-color: rgb(187, 187, 187) rgb(221, 221, 221) rgb(221, 221, 221) rgb(187, 187, 187); border-style: solid; border-width: 1px; font-size: 13px; line-height: 1.5; margin: 2em; overflow: auto; padding: 1em;"><span style="font-family: Verdana, sans-serif;"># yum install jenkins</span></pre>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/7fbf4b26cf28b957908cd41076e7fe45/tumblr_inline_mtxw2q6I9F1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">8. Start jenkins and make sure it starts automatically at system startup</span></div>
<pre style="background-color: white !important; border-color: rgb(187, 187, 187) rgb(221, 221, 221) rgb(221, 221, 221) rgb(187, 187, 187); border-style: solid; border-width: 1px; font-size: 13px; line-height: 1.5; margin: 2em; overflow: auto; padding: 1em;"><span style="font-family: Verdana, sans-serif;"># service jenkins start</span></pre>
<pre style="background-color: white !important; border-color: rgb(187, 187, 187) rgb(221, 221, 221) rgb(221, 221, 221) rgb(187, 187, 187); border-style: solid; border-width: 1px; font-size: 13px; line-height: 1.5; margin: 2em; overflow: auto; padding: 1em;"><span style="font-family: Verdana, sans-serif;"># chkconfig jenkins on</span></pre>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/fea9f8b4261281862e14037064bee251/tumblr_inline_mtxw55W6tV1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">9. Open your browser and navigate to http://<Elastic-IP>:8080. You will see jenkins dashboard.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/06526475fd656b57e2a47714ec98573a/tumblr_inline_mtxw6sXqFQ1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">That’s it. You have your jenkins setup up and running. Now, you can create jobs to build the code.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<strong><span style="font-family: Verdana, sans-serif;">Securing Jenkins Setup :</span></strong></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">As seen, your above installed jenkins setup is directly accessible through internet and there are no security measures to prevent direct access. In order to secure your jenkins setup, you need to enable some sort-of security authorization on it.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">To enable security for jenkins setup, please select “Manage Jenkins” on jenkins dashboard and click on “Configure Global Security”.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/eb33f23fdd8dbe51b2aba8ada1ec930c/tumblr_inline_mtxwzo571e1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">After this, please select “Enable Security”.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/42685bf702204900609c6b605832ce5f/tumblr_inline_mtxx2856h41qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Jenkins provide different level of security methods like integrating it with your LDAP setup or using Unix user/group database. In our case, we will create “Jenkins’ Own User Database”. So, please select “Jenkins’s Own User Database” under security realm. Don’t forget to select “Allow users to sign up”. Also, select “Matrix-based security” under Authorization. </span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/309dcaf754634717778120cd43103819/tumblr_inline_mtxx5yQjx71qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Now, please add an user (for eg., admin) to jenkins user database by adding username under matrix-based security section.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/b0832059a60b2a21ee9e2be42b0f0c95/tumblr_inline_mtxxb36O3Q1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Once done, please provide necessary privileges to this user. In our case, we make this user as an administrator, so he will have full privileges.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/9e8906d7f5c3349bf76e6da1e5868eda/tumblr_inline_mtxxdfC8Vr1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
</div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Once done, please click on save. Now, you will be asked to provide username and password for the user to login. But we don’t have any. So, please click on “Create an account”.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/68f9426841c8163dc45fe1c464fdc714/tumblr_inline_mtxxgmplXK1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Now, please provide your username ( “admin” in our case) and provide necessary details like password, name and email address. </span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/e898aefb0d4d35bc6796e3306db51975/tumblr_inline_mtxxj4esOn1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Once done, you will be logged into jenkins and your whole setup is secured.</span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><img alt="image" src="http://media.tumblr.com/9e15bc50350cd24cf8e9e7c2c3233ebd/tumblr_inline_mtxxkrHN0e1qiid96.png" style="border: 0px; vertical-align: bottom;" /></span></div>
<div style="background-color: white; color: #333333; font-size: 13px; line-height: 18px; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;">Now, you can create jenkins jobs without any security issues.</span></div>
<div style="background-color: white; margin-bottom: 20px; padding: 0px;">
<span style="font-family: Verdana, sans-serif;"><span style="color: #333333;"><span style="line-height: 18px;">This article was originally posted @ </span></span><a href="http://sanketdangi.com/post/62715793234/install-configure-jenkins-on-amazon-linux" style="background-color: transparent;">http://sanketdangi.com/post/62715793234/install-configure-jenkins-on-amazon-linux</a></span></div>
<div style="background-color: white; margin-bottom: 20px; padding: 0px;">
<br /></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-15367263911573444922013-10-06T06:55:00.000-07:002013-10-13T10:42:23.919-07:00AWS Cost Saving Tip 16: Periodically remove your unwanted AWS resources and save costs<span style="font-family: Verdana, sans-serif;">Thanks "@HighScalability.com" for referencing this post<b> <u><a href="http://highscalability.com/blog/2013/10/11/stuff-the-internet-says-on-scalability-for-october-11th-2013.html">"</a></u><a href="http://highscalability.com/blog/2013/10/11/stuff-the-internet-says-on-scalability-for-october-11th-2013.html"><span style="color: #333333;"><span style="background-color: white; letter-spacing: 0px; line-height: 1em; margin-bottom: 15px; margin-top: 15px; text-transform: capitalize;">Stuff The Internet Says On Scalability For October 11th, 2013</span></span><span style="background-color: white; color: #333333; line-height: 1em; text-transform: capitalize;">"</span></a></b></span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><b><br /></b></span></div>
<span style="font-family: Verdana, sans-serif;">Following are some of the AWS assets that needs to be periodically reviewed for under utilization and should be removed as part of your IT process to avoid cost leakage in AWS. </span><br />
<br />
<span style="font-family: Verdana, sans-serif;"><b>Remove unwanted AMI's:</b> Periodically audit for unwanted AMI's in your dev/test/production environment and remove them. Tagging the AMI's with proper identifiers can help you during the cleansing process.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>EBS snapshots</b> are incremental in nature and only the changes are moved to them. The unwanted snapshots should be periodically identified (Tagging snapshots will help you) and deleted as part of your IT Infrastructure operations. EBS snapshots are priced $0.095 per GB-month of data stored and you can save few hundred $$$ depending upon the volume of unwanted snapshots stored.To know more about <u><a href="http://harish11g.blogspot.in/2013/04/Understanding-Amazon-Elastic-block-store-intro.html">EBS Snapshots</a></u></span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>EBS Volumes:</b> In development and testing environments of large enterprises we can often find lots of EBS volumes provisioned and lying unattached or unused. EBS Volumes are priced at $0.10 per GB-month of provisioned storage. Usually 10-30 GB are the most used volume ranges for the dev/test. Imagine a enterprise having few hundred EC2's for their Dev/Test infrastructure and 10-25% of their storage are unused, Periodically identifying , tagging, consolidating and cleansing them will save few hundred to thousands of $$$ for the enterprises.</span><span style="font-family: Verdana, sans-serif;">To know more about<a href="http://harish11g.blogspot.in/2013/04/Understanding-Amazon-Elastic-block-store-intro.html"> </a></span><u style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Understanding-Amazon-Elastic-block-store-intro.html">EBS Volumes</a></u><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>S3 Buckets and Objects:</b> Unused objects in S3 buckets have to be frequently deleted. Objects with a limited life period can be configured to expire dynamically using S3 object expiration policies. Objects that are rarely used can be archived to Amazon Glacier from S3. This exercise is an important candidate for Media and online companies using AWS. To know more about other S3 Cost saving tips <u><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Reduced-Redundancy-storage-rrs-vs-std.html">1</a></u> | <u><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Object-expiration.html">2</a></u>.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Elastic IP</b> is a scarce resource in AWS cloud, in event the EIP's attached to your account are not associated with any running EC2 instance, you will be charged $0.005 per hour on a prorata basis.</span><br />
<span style="font-family: Verdana, sans-serif;">EIP's can usually be leaked in: </span><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">Dev/Test environments, where the developer disassociates the EIP and keeps it unused.</span></li>
<li><span style="font-family: Verdana, sans-serif;">In production, where sometimes EIP are assigned to Auto Scaled EC2 instances using scripts and they are not properly released back after EC2 instances are terminated by AWS.</span></li>
</ul>
<br />
<span style="font-family: Verdana, sans-serif;">Though if the EIP is kept unused can lead to around only ~$3.5 leakage per month, if no proper tracking procedures are in place many such EIP's can be residing in your account unused and lead to more such leakages.</span><br />
<br />
<span style="font-family: Verdana, sans-serif;"><b>ELB:</b> Unused and Unwanted Elastic Load Balancers should be removed periodically to avoid cost leakage. A single ELB can cost around ~216 USD if kept unused a year. Usually ELB leakage is prone to happen in Dev/Test environments compared to production. To know more about <u><a href="http://harish11g.blogspot.in/2013/08/exploring-load-balancing-in-Amazon-web-services-aws-cloud-in-detail.html">Amazon ELB</a></u></span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Monitoring :</b> Detailed monitoring option of CloudWatch is generally not required for Development EC2 instances. On the other hand, they are required on Load Testing and Production environments for monitoring the health. Custom Metrics are also good candidates for Production and Load Test environments. Constant review of the infrastructure in AWS has to be done to identify whether such resources(over provisioning) are associated with Dev/Test env and related cost leakages can be avoided.Detailed monitoring costs $3.50 per instance per month, provided at 1-minute frequency and Custom Metrics costs $0.50 per metric per month.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Glacier archives : </b><span style="background-color: white; line-height: 18px;">Amazon Glacier is designed for use cases where data is retained for months, years, or decades. But there are times where you may find some data life has expired and they can be removed. </span><span style="background-color: white; line-height: 18px;">Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. Please ensure it is part of you IT process. To know more about other Cost savings in Glacier <u><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-use-Amazon-Glacier-for-archives-and-save-costs.html">1</a></u> | <u><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-deletion-in-Amazon-glacier.html">2</a></u> | <u><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html">3</a></u></span></span><br />
<br />
<b><span style="font-family: Verdana, sans-serif;">Other small stuff :</span></b><br />
<br />
<ul>
<li><span style="font-family: Verdana, sans-serif;">AWS charges $1.00 per month for inactive data pipelines. They should be periodically reviewed for leakages. </span><span style="font-family: Verdana, sans-serif;"> </span></li>
</ul>
<br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div>
<b><span style="font-family: Verdana, sans-serif;">Note:</span></b></div>
<div>
<ul>
<li><span style="font-family: Verdana, sans-serif;">There are times where an AWS asset can be created in a non frequently used AWS Region mistakenly by a developer and it can be lying idle for months. Have an automated or manual IT process to periodically cleanse the above leakages points in <b><u>"ALL" AWS Regions</u></b> and not just your most frequently used regions.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Many times on large AWS deployments/Dev/Test environments, it becomes difficult to track who created and who is currently using the AWS resources. AMI's, EBS snapshots, Volumes and S3 buckets should be tagged for proper identification. Identifiable names should be kept for other assets (wherever applicable) , so that it helps your IT team during the cleansing process. </span></li>
<li><span style="font-family: Verdana, sans-serif;">Employ automation using scripts/programs or use governance tools ( which i will be discussing later) to monitor under utilization of resources and there by remove them.</span></li>
</ul>
</div>
<div>
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><b>Other Cost Saving Tips</b></span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"></span><br style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18px;" />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-SQS-Long-polling-Batch-requests.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests</a></span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Choosing-right-search-tier-Amazon-cloudsearch-Apache-Solr.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 2: How right search technology choice saves cost in AWS ?</a></span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-Cloudfront-Price-class.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs</a></span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-ElastiCache-cluster-nodes.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster</a></span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?</a></span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-AutoScaling-termination-policy-ClosestToNextInstanceHour.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings</a></span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Object-expiration.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 7: Use Amazon S3 Object Expiration</a></span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Reduced-Redundancy-storage-rrs-vs-std.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage</a> </span><span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"> </span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Delete-Amazon-EBS-snapshots-regularly-retention-period.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place</a></span><span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"> </span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-make-right-EBS-Volumes-Choice-PIOPS-vs-Standard.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs</a> </span><br />
<a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-elasticity-Amazon-Elastic-MapReduce-EMR.html" style="background-color: white; color: #4d469c; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">Cost Saving Tip 11: <span style="color: #0000ee;"><u>How elastic thinking saves cost in Amazon EMR Clusters ?</u></span> </a><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-Adding-Spot-EC2-with-Amazon-Elastic-MapReduce-EMR.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 12: <span style="color: #0000ee;"><u>Add Spot Instances with Amazon EMR</u></span></a> </span><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-use-Amazon-Glacier-for-archives-and-save-costs.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 13: <span style="color: #0000ee;"><u>Use Amazon Glacier for archive data and save costs</u></span></a> </span><b style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><span style="color: red;">(new)</span></b><br />
<span style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-deletion-in-Amazon-glacier.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 14: <span style="color: #0000ee;"><u>Plan your deletion in Amazon Glacier and avoid cost leakage</u></span></a> </span><b style="background-color: white; color: #444444; font-family: Verdana, sans-serif; font-size: 13px; line-height: 18px;"><span style="color: red;">(new)</span></b><br />
<div style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18px;">
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html" style="color: #4d469c; text-decoration: none;">Cost Saving Tip 15: <span style="color: #0000ee;"><u>Plan your Amazon Glacier Retrieval strategy and avoid cost leakage</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b></div>
</div>
<div>
<div class="MsoNormal">
<br /></div>
</div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-74356350235503863622013-09-21T22:29:00.000-07:002013-10-13T10:57:30.223-07:00AWS Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage<b style="font-family: Verdana, sans-serif;"><br /></b>
<b style="font-family: Verdana, sans-serif;">Tip 1: Use Multi-part upload and range retrieval:</b><br />
<span style="font-family: Verdana, sans-serif;">You store data in Amazon Glacier as an archive. An archive is any object, such as a photo, video, or document that you store and usually </span><span style="font-family: Verdana, sans-serif;">represents a single file or several files</span><span style="font-family: Verdana, sans-serif;">. It is a base unit of storage in Amazon Glacier. You can upload an archive in a single request. For large archives, Amazon Glacier provides a multipart upload API that enables you to upload an archive in parts and enables range retrieval in future.</span><br />
<span style="font-family: Verdana, sans-serif;">In reality many times, you may need to retrieve a small selection of those files from a archive, in which case you could retrieve only the ranges of the archive that contained the required files.You may also choose to perform range retrievals in order to reduce or eliminate your retrieval fees. If you exceed your free retrieval allowance, you pay a retrieval fee that is based on your peak retrieval rate. Spreading out a retrieval of an archive in smaller parts could therefore allow you reduce your retrieval fees, by reducing your peak retrieval rate.</span><br />
<br />
<span style="font-family: Verdana, sans-serif;"><b>Tip 2: Spread your retrievals and avoid cost leakage:</b></span><br />
<span style="font-family: Verdana, sans-serif;">Let’s assume you are storing 75 TB of data in Amazon Glacier and you would like to retrieve 140 GB. The amount you pay is determined by how fast you retrieve the data. </span><br />
<span style="font-family: Verdana, sans-serif;"><b>Option 1:</b> Storing 75 TB and retrieving 140 GB in 4 hours costs</span><span style="font-family: Verdana, sans-serif;"> $21.60. </span><br />
<span style="font-family: Verdana, sans-serif;"><b>Option 2:</b> </span><span style="font-family: Verdana, sans-serif;">Storing 75 TB and retrieving 140 GB in 8 hours costs </span><span style="font-family: Verdana, sans-serif;">$10.80 (half the cost of option 1)</span><br />
<span style="font-family: Verdana, sans-serif;"><b>Option 3:</b> </span><span style="font-family: Verdana, sans-serif;">Storing 75 TB and retrieving 140 GB evenly over 28 hours costs NIL because </span><span style="font-family: Verdana, sans-serif;">you would no longer exceed your daily free retrieval allowance and would therefore not be charged a Retrieval Fee. </span><span style="font-family: Verdana, sans-serif;">Imagine how much cost is leaked if option 1 is followed frequently instead option 3 where the billing is lower or free most times. This can be avoided by better understanding of Amazon Glacier, planning your retrieval in advance with proper spread.</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">In Amazon Glacier, the suggested practice to lower costs is to plan your retrieval strategy in advance by </span><span style="font-family: Verdana, sans-serif;">spreading out the retrievals over longer periods of time. This will reduce or eliminate the retrieval fees and lowers your billing.</span><br />
<span style="font-family: Verdana, sans-serif;">Use the aggressive retrieval mechanisms only on use cases where timing is important, Example : You have an o<span style="background-color: white; line-height: 17.984375px;">ld archived news footage in Amazon Glacier, and suddenly based on the current event this old new footage needs to be retrieved, related and delivered. In this case you can use peak retrieval rate and other times you can spread your retrieval strategy.</span></span><br />
<br />
<span style="font-family: Verdana, sans-serif;"><span style="line-height: 17.99715805053711px;">To explore </span>Log Analysis and Archive with Amazon S3 and Glacier. <u><a href="http://harish11g.blogspot.in/2013/03/log-analysis-archival-using-amazons3-and-glacier-full-summary.html">Refer this detailed article series.</a></u></span><br />
<div>
<br /></div>
<span style="font-family: Verdana, sans-serif;"><b>Other Tips</b></span><br />
<span style="font-family: Verdana, sans-serif;"></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-SQS-Long-polling-Batch-requests.html">Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Choosing-right-search-tier-Amazon-cloudsearch-Apache-Solr.html">Cost Saving Tip 2: How right search technology choice saves cost in AWS ?</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-Cloudfront-Price-class.html">Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-ElastiCache-cluster-nodes.html">Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html">Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-AutoScaling-termination-policy-ClosestToNextInstanceHour.html">Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Object-expiration.html">Cost Saving Tip 7: Use Amazon S3 Object Expiration</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Reduced-Redundancy-storage-rrs-vs-std.html">Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage</a> </span><span style="font-family: Verdana, sans-serif;"> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Delete-Amazon-EBS-snapshots-regularly-retention-period.html">Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place</a></span><span style="font-family: Verdana, sans-serif;"> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-make-right-EBS-Volumes-Choice-PIOPS-vs-Standard.html">Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs</a> </span><br />
<a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-elasticity-Amazon-Elastic-MapReduce-EMR.html" style="font-family: Verdana, sans-serif;">Cost Saving Tip 11: <span style="color: #0000ee;"><u>How elastic thinking saves cost in Amazon EMR Clusters ?</u></span> </a><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-Adding-Spot-EC2-with-Amazon-Elastic-MapReduce-EMR.html">Cost Saving Tip 12: <span style="color: #0000ee;"><u>Add Spot Instances with Amazon EMR</u></span></a> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-use-Amazon-Glacier-for-archives-and-save-costs.html">Cost Saving Tip 13: <span style="color: #0000ee;"><u>Use Amazon Glacier for archive data and save costs</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-deletion-in-Amazon-glacier.html">Cost Saving Tip 14: <span style="color: #0000ee;"><u>Plan your deletion in Amazon Glacier and avoid cost leakage</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b><br />
<div>
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html">Cost Saving Tip 15: <span style="color: #0000ee;"><u>Plan your Amazon Glacier Retrieval strategy and avoid cost leakage</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b></div>
<div>
<b style="font-family: Verdana, sans-serif;"><span style="color: red;"><br /></span></b></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-17765288142091762612013-09-21T10:17:00.001-07:002013-10-13T10:56:53.251-07:00AWS Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage<span style="font-family: Verdana, sans-serif;">Amazon Glacier is designed for use cases where data is retained for months, years, or decades. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Example : In the US East Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months. </span><span style="font-family: Verdana, sans-serif;">So if you deleted 1 gigabyte of data 1 month after uploading it, you would be charged a $0.02 early deletion fee. If, instead you deleted 1 gigabyte after 2 months, you would be charged a $0.01 early deletion fee.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><b>Avoid Cost leakage:</b></span><br />
<span style="font-family: Verdana, sans-serif;">Imagine your IT process does not have this important point considered and assume they delete ~10 TB within 3 months every quarter from Amazon Glacier Archives, they are unnecessarily leaking ~2000 USD cost to AWS every quarter. Instead p</span><span style="font-family: Verdana, sans-serif;">lan your deletion strategy well in advance build your IT process around this Cost constraint and delete items only after 3 months and avoid this unnecessary leakage.</span><br />
<span style="font-family: Verdana, sans-serif;"><i>(Note: Storing rarely accessed archives in Amazon Glacier and deleting early is still cheap compared to storing 10 TB in Amazon S3 and paying ~8K USD bill per month.)</i></span><br />
<span style="font-family: Verdana, sans-serif;"><i><br /></i></span>
<span style="font-family: Verdana, sans-serif;"><span style="line-height: 17.99715805053711px;">To explore </span>Log Analysis and Archive with Amazon S3 and Glacier. <u><a href="http://harish11g.blogspot.in/2013/03/log-analysis-archival-using-amazons3-and-glacier-full-summary.html">Refer this detailed article series.</a></u></span><br />
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<b style="font-family: Verdana, sans-serif;">Other Tips</b><br />
<span style="font-family: Verdana, sans-serif;"></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-SQS-Long-polling-Batch-requests.html">Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Choosing-right-search-tier-Amazon-cloudsearch-Apache-Solr.html">Cost Saving Tip 2: How right search technology choice saves cost in AWS ?</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-Cloudfront-Price-class.html">Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-ElastiCache-cluster-nodes.html">Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html">Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-AutoScaling-termination-policy-ClosestToNextInstanceHour.html">Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Object-expiration.html">Cost Saving Tip 7: Use Amazon S3 Object Expiration</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Reduced-Redundancy-storage-rrs-vs-std.html">Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage</a> </span><span style="font-family: Verdana, sans-serif;"> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Delete-Amazon-EBS-snapshots-regularly-retention-period.html">Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place</a></span><span style="font-family: Verdana, sans-serif;"> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-make-right-EBS-Volumes-Choice-PIOPS-vs-Standard.html">Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs</a> </span><br />
<a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-elasticity-Amazon-Elastic-MapReduce-EMR.html" style="font-family: Verdana, sans-serif;">Cost Saving Tip 11: <span style="color: #0000ee;"><u>How elastic thinking saves cost in Amazon EMR Clusters ?</u></span> </a><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-Adding-Spot-EC2-with-Amazon-Elastic-MapReduce-EMR.html">Cost Saving Tip 12: <span style="color: #0000ee;"><u>Add Spot Instances with Amazon EMR</u></span></a> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-use-Amazon-Glacier-for-archives-and-save-costs.html">Cost Saving Tip 13: <span style="color: #0000ee;"><u>Use Amazon Glacier for archive data and save costs</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-deletion-in-Amazon-glacier.html">Cost Saving Tip 14: <span style="color: #0000ee;"><u>Plan your deletion in Amazon Glacier and avoid cost leakage</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b><br />
<div>
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html">Cost Saving Tip 15: <span style="color: #0000ee;"><u>Plan your Amazon Glacier Retrieval strategy and avoid cost leakage</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b></div>
<div>
<br /></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0tag:blogger.com,1999:blog-4836510665235566234.post-68650376581015093112013-09-21T07:33:00.001-07:002013-10-13T10:56:17.344-07:00AWS Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div style="background-color: white; margin-bottom: 1em; padding: 0px;">
<div style="line-height: 17.99715805053711px;">
<span style="font-family: Verdana, sans-serif;">Amazon S3 is a storage service that is widely used for storing frequently accessible data like documents, images, videos, log files etc. It is designed for use cases demanding low latency and frequent access. Once the</span><span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;"> assets grow over a period of time, S3 becomes a costly storage option to the enterprises for rarely accessed items. Such rarely accessed items can be moved to a durable, stable archive systems for saving costs. Some of the use cases for archives are :</span></div>
<div style="line-height: 17.99715805053711px;">
</div>
<ul>
<li><span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">Media assets like news footage, movies, HD content can grow to tens or hundreds of petabytes over years. Old Archived footage sometimes can become valuable based on current global events and access is needed only during that time.</span></li>
<li><span style="font-family: Verdana, sans-serif;">Enterprises need to archive data like email, legal records, financial documents etc for complying to their their regulatory and business needs. They are needed only during audits. </span></li>
<li><span style="font-family: Verdana, sans-serif;">Organizations like libraries, </span><span style="font-family: Verdana, sans-serif;">historical societies, non-profits, governments are increasing their efforts to preserve valuable but aging digital content and artifacts that are no longer readily available. These archive efforts can grow to petabytes over time.</span></li>
</ul>
<span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">Since Archives do not require frequent access or low latency,f</span><span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">or savings costs in Amazon Web Services Infrastructure, you can access real time data from S3 and move rarely retrieved data to Cold storage data archival solutions like Amazon Glacier. </span><br />
<span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;"><b><br /></b></span>
<span style="font-family: Verdana, sans-serif;"><span style="line-height: 17.99715805053711px;">To explore </span>Log Analysis and Archive with Amazon S3 and Glacier. <u><a href="http://harish11g.blogspot.in/2013/03/log-analysis-archival-using-amazons3-and-glacier-full-summary.html">Refer this detailed article series.</a></u></span><br />
<span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;"><b><br /></b></span>
<span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;"><b>How does it work ?</b></span><br />
<span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">You can automatically move the objects from S3 to Amazon Glacier using S3 object life cycle rules. </span><span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">Objects which needs to archived to Glacier are configured with object prefixes, relative or absolute time specifier and time periods. Amazon S3 will go through the life cycle policies every day and moves these objects which are qualified to Amazon Glacier. Since the objects are moved from S3 to Glacier, it will be deleted in S3, but the </span><span style="background-color: transparent; font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">index entry will remain as-is</span><span style="font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">.</span></div>
<div style="background-color: white; line-height: 17.99715805053711px; margin-bottom: 1em; padding: 0px;">
<span style="background-color: transparent; font-family: Verdana, sans-serif; line-height: 17.99715805053711px;"><b>What tools can be used ?</b></span><br />
<span style="background-color: transparent; font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">You can additionally use the following tools/methodologies to take advantage of this functionality:</span><br />
<ul>
<li><span style="background-color: transparent; font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">Latest release of CloudBerry Explorer freeware comes with the feature: </span><span style="background-color: transparent; font-family: Verdana, sans-serif; line-height: 17.99715805053711px;"><a href="http://www.cloudberrylab.com/blog/how-to-archive-data-from-amazon-s3-to-glacier-with-cloudberry-explorer/" style="background-color: transparent; line-height: 17.99715805053711px;">http://www.cloudberrylab.com/blog/how-to-archive-data-from-amazon-s3-to-glacier-with-cloudberry-explorer/</a></span></li>
<li><span style="background-color: transparent; font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">Use AWS SDK (Java/.Net based) to access the Amazon Glacier.</span></li>
<li><span style="background-color: transparent; font-family: Verdana, sans-serif; line-height: 17.99715805053711px;">Use Fast Glacier is free for personal use while a commercial licence is also available. </span><a href="http://fastglacier.com/" style="background-color: transparent; line-height: 17.99715805053711px;"><span style="font-family: Verdana, sans-serif;">http://fastglacier.com/</span></a></li>
</ul>
</div>
<span style="font-family: Verdana, sans-serif;"><b>Cost Savings ?</b></span><br />
<span style="font-family: Verdana, sans-serif;"><b><br /></b></span>
<span style="font-family: Verdana, sans-serif;"><span style="line-height: 17.984375px;"><content under progress></span></span><br />
<span style="font-family: Verdana, sans-serif;"><span style="line-height: 17.984375px;"><br /></span></span>
<span style="font-family: Verdana, sans-serif;"><b>Other Tips</b></span><br />
<span style="font-family: Verdana, sans-serif;"></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-SQS-Long-polling-Batch-requests.html">Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Choosing-right-search-tier-Amazon-cloudsearch-Apache-Solr.html">Cost Saving Tip 2: How right search technology choice saves cost in AWS ?</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-Cloudfront-Price-class.html">Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-ElastiCache-cluster-nodes.html">Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html">Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-AutoScaling-termination-policy-ClosestToNextInstanceHour.html">Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Object-expiration.html">Cost Saving Tip 7: Use Amazon S3 Object Expiration</a></span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Amazon-S3-Reduced-Redundancy-storage-rrs-vs-std.html">Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage</a> </span><span style="font-family: Verdana, sans-serif;"> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-Delete-Amazon-EBS-snapshots-regularly-retention-period.html">Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place</a></span><span style="font-family: Verdana, sans-serif;"> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/05/Amazon-Web-Services-AWS-Cost-Saving-Tips-make-right-EBS-Volumes-Choice-PIOPS-vs-Standard.html">Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs</a> </span><br />
<a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-elasticity-Amazon-Elastic-MapReduce-EMR.html" style="font-family: Verdana, sans-serif;">Cost Saving Tip 11: <span style="color: #0000ee;"><u>How elastic thinking saves cost in Amazon EMR Clusters ?</u></span> </a><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/06/Amazon-Web-Services-AWS-Cost-Saving-Tips-Adding-Spot-EC2-with-Amazon-Elastic-MapReduce-EMR.html">Cost Saving Tip 12: <span style="color: #0000ee;"><u>Add Spot Instances with Amazon EMR</u></span></a> </span><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-use-Amazon-Glacier-for-archives-and-save-costs.html">Cost Saving Tip 13: <span style="color: #0000ee;"><u>Use Amazon Glacier for archive data and save costs</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b><br />
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-deletion-in-Amazon-glacier.html">Cost Saving Tip 14: <span style="color: #0000ee;"><u>Plan your deletion in Amazon Glacier and avoid cost leakage</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b><br />
<div>
<span style="font-family: Verdana, sans-serif;"><a href="http://harish11g.blogspot.in/2013/09/Amazon-Web-Services-AWS-Cost-Saving-Tips-plan-data-retrieval-strategy-in-amazon-glacier-save-costs.html">Cost Saving Tip 15: <span style="color: #0000ee;"><u>Plan your Amazon Glacier Retrieval strategy and avoid cost leakage</u></span></a> </span><b style="font-family: Verdana, sans-serif;"><span style="color: red;">(new)</span></b></div>
<div>
<b style="font-family: Verdana, sans-serif;"><span style="color: red;"><br /></span></b></div>
Harish Ganesanhttp://www.blogger.com/profile/02069892404142116481noreply@blogger.com0