Apache Solr to Amazon CloudSearch migration tool

In this post, we are introducing a new tool called S2C – Apache Solr to Amazon CloudSearch Migration Tool. S2C is a Linux console based utility that helps developers / engineers to migrate search index from Apache Solr to Amazon CloudSearch.

Very often customers initially build search for their website or application on top of Solr, but later run into challenges like elastic scaling and managing the Solr servers. This is a typical scenario we have observed in our years of search implementation experience. For such use cases, Amazon CloudSearch is a good choice. Amazon CloudSearch is a fully-managed service in the cloud that makes it easy to set up, manage, and scale a search solution for your website. To know more, please read the Amazon CloudSearch documentation

We are seeing growing trend every year, organizations of various sizes are migrating their workloads to Amazon CloudSearch and leveraging the benefits of fully managed service. For example, Measured Search, an analytics and e-Commerce platform vendor, found it easier to migrate to Amazon CloudSearch rather than scale Solr themselves (see article for details).

Since Amazon CloudSearch is built on top of Solr, it exposes all the key features of Solr while providing the benefits of a fully managed service in the cloud such as auto-scaling, self-healing clusters, high availability, data durability, security and monitoring.

In this post, we provide step-by-step instructions on how to use the Apache Solr to Amazon CloudSearch Migration (S2C) tool to migrate from Apache Solr to Amazon CloudSearch.

Before we get into detail, you can download the S2C tool in the below link.
Download Link: https://s3-us-west-2.amazonaws.com/s2c-tool/s2c-cli.zip

Pre-Requisites

Before starting the migration, the following pre-requisites have to be met. The pre-requisites include installations and configuration on the migration server. The migration server could be the same Solr server or independent server that sits between your Solr server and Amazon CloudSearch instance.  

Note: We recommend running the migration from the Solr server instead of independent server as it can save time and bandwidth. It is much better if the Solr server is hosted on EC2 as the latency between EC2 and CloudSearch is relatively less.

The following installations and configuration should be done on the migration server (i.e. your Solr server or any new independent server that connects between your Solr machine and Amazon CloudSearch).    1.      The application is developed using Java. Download and Install Java 8 .Validate the JDK path and ensure the environment variables like JAVA_HOME, classpath, path is set correctly. 2.      We assume you already have setup Amazon Web services IAM account. Please ensure the IAM user has right permissions to access AWS services like CloudSearch. Note: If you do not have an AWS IAM account with above mentioned permissions, you cannot proceed further. 3.      The IAM user should have AWS Access key and Secret key. In the application hosting server, set up the Amazon environment variables for access key and secret key. It is important that the application runs using the AWS environment variables. To setup AWS environment variables, please read the below link. http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html Alternatively, you can set the following AWS environment variables by running the commands below from Linux console. export AWS_ACCESS_KEY=Access Key   export AWS_SECRET_KEY=Secret Key 4.      Note: This step is applicable only if the application is hosted on Amazon EC2. If you do not have an AWS Access key and Secret key, you can opt for IAM role attached to an EC2 instance. A new IAM role can be created and attached to EC2 during the instance launch. The IAM role should have access to AWS resources like S3, DynamoDB and CloudSearch. For more information, read the below link http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html 5.      Download the migration utility ‘S2C’, unzip the tool and copy it in your working directory.  Download Link : https://s3-us-west-2.amazonaws.com/s2c-tool/s2c-cli.zip

S2C Utility File

The downloaded ‘S2C’ migration utility should have the following sub directories and files.

Folder / Files	Description
bin	Binaries of the migration tool
lib	Libraries required for migration
application.conf	Configuration file that allows end users to input parameters	Require end-user’s input.
logback.xml	Log file configuration	Optional. Does not require end-user  / developer input
s2c	script file that executes the migration process

Configure only application.conf and logback.xml.  Do not modify any other file.                                                                                                                                         

application.conf

The application.conf file has the configuration related to the new Amazon CloudSearch domain that will be created. The parameters configured in the in the application.conf file are explained in the table below.

s2c {   api {     SchemaParser = "s2c.impl.solr.DefaultSchemaParser"     SchemaConverter = "s2c.impl.cs.DefaultSchemaConverter"     DataFetcher = "s2c.impl.solr.DefaultDataFetcher"     DataPusher = "s2c.impl.cs.DefaultDataPusher"  }	List of API that is executed step by step during the migration. Do not change this.
solr {     dir = "files"     server-url = "http://localhost:8983/solr/collection1"     fetch-limit = 100   }	dir – The base directory path of Solr. Ensure the directory is present and also its validity. Eg:/opt/solr/example/solr/collection1/conf
	server-url – Server host, port and collection path The endpoint which will be used to fetch the data. If the utility is run from a different server, ensure the IP address and port has firewall access.
	fetch-limit – number of solr documents that can be fetched for each batch call This configuration number should be carefully set by the developer. The fetch limit depends on the following factors: 1.  Record size of a Solr record(1KB or 2KB) 2.  Latency between migration server and Amazon CloudSearch 3.  Current Request Load on the Solr Server E.g.: If the total Solr documents is 100000 and fetch limit is 100, then it would take 100000 / 10 = 10000 batch calls to complete the fetch. If size of each Solr record is 2KB, then 100000 * 2KB = 200MB data is migrated.
cs {     domain = "collection1"     region = "us-east-1"     instance-type = " search.m3.xlarge"     partition-count = 1     replication-count = 1   }	domain - CloudSearch domain name Ensure that the domain name does not already exist.
	Region – AWS region for the new CloudSearch domain
	Instance type – Desired instance type for CloudSearch nodes Choose the instance type based on the volume of data and the expected query volume.
	Partition count – Number of partitions required for CloudSearch
	replication-count - Replication count for CloudSearch
wd = "/tmp"	Temporary file path to store intermediate data files and migration log files

Running the migration

Before launching the S2C migration tool, verify the following:

1.      Solr directory path – Make sure that the Solr directory path is valid and available. The tool cannot read the configuration if the path or directory is invalid.

2.      Solr configuration contents - Validate that the Solr configuration contents are correctly set inside the directory. Example: solrconfig.xml, schema.xml, stopwords.txt, etc.

3.      Make sure that the working directory is present in the file system and has write permissions for the current user. It can be an existing directory or a new directory. The working directory stores the fetched data from Solr and migration logs.

4.      Validate the disk size before starting the migration. If the available free disk space is lesser than the size of the Solr index, the fetch operations will fail.

For example, if the Solr index size is 7 GB, make sure that the disk has at least 10 GB to 20 GB of free space.

Note: The tool reads the data from Solr and stores in a temporary directory (Please see configuration wd = /tmp in the above table).

5.      Verify that the AWS environment variables are set correctly. The AWS environment variables are mentioned in the pre-requisites section above.

6.      Validate the firewall rules for IP address and ports if the migration tool is run from a different server or instance. Example: Solr default port 8983 should be opened to the EC2 instance executing this tool.

Run the following command from directory ‘{S2C filepath}’ example: /build/install/s2c-cli

./s2c Or JVM_OPTS="-Xms2048m -Xmx2048m" ./s2c (With heap size)

The above will invoke the shell ‘s2c’ script that starts the search migration process. The migration process is a series of steps that require user inputs as shown in the screen shots below.

Step 1: Parse the Solr schema

The first step of migration prompts for a confirmation to parse the Solr schema and Solr configuration file. During this step, the application generates a ‘Run Id’ folder inside the working directory.

Example: /tmp/s2c/m1416220194655

The Run Id is a unique identifier for each migration. Note down the Run Id as you will need it to resume the migration in case of any failure.

Step 2: Schema conversion from Solr to CloudSearch
The second step prompts confirmation to convert Solr schema to CloudSearch schema. Press any key to proceed further.

The second step will also list all the converted fields which are ready to be migrated from Solr to CloudSearch. If any fields are left out, this step will allow you to correct the original schema. User can abort the migration and identify the ignored fields, rectify the schema and re-run the migration again.

The below screen shot shows the fields ready for CloudSearch migration.

Step 3: Data Fetch

The third step prompts for confirmation to fetch the search index data from the Solr server. Press any key to proceed. This step will generate a temporary file which will be stored in the working directory. This temporary file will have all the fetched documents from the Solr index.

There is also option to skip the fetch process if all the Solr data is already stored in the temporary file. If this is the case, the prompt will look like the screenshot below.

Step 4: Data push to CloudSearch

The last and final step prompts for confirmation to push the search data from the temporary file store to Amazon CloudSearch. This step also creates the CloudSearch domain with the configuration specified in application.conf including desired instance type, replication count, and multi-AZ options.

If the domain is already created, the utility will prompt to use the existing domain. If you do not wish to use an existing domain, you can create a new CloudSearch domain using the same prompt.

Note: The console does not prompt for any ‘CloudSearch domain name’ but instead it uses the domain name configured in the application.conf file.

Step 5: Resume (Optional)

During the migration steps, if there is any failure during the fetch operation, it can be resumed. This is illustrated in the screen shot below. 

Step 6: Verification

Log into AWS CloudSearch management console to verify that the domain and index fields.

Amazon CloudSearch allows running test queries to validate the migration and as well the functionality of your application.

Features supported

1.      Support for other non-Linux environments is not available for now.

2.      Support for Solr Shards is not available for now. The Solr shard needs to be migrated separately.

3.      The install commands may vary for different Linux flavors. Example installing software, file editor command, permission set commands can be different for every Linux flavors. It is left to engineering team to choose the right commands during the installation and execution of this migration tool.

4.      Only fields configured as ‘stored’ in Solr schema.xml are supported. The non-stored fields are ignored during schema parsing.

5.      The document id (unique key) is required to have following attributes:

a.      Document ID should be 128 characters or less in size.

b.      Document ID can contain any letter, any number, and any of the following characters:      _ - = # ; : / ? @ &

c.       The below link will help you to understand in data  preparation before migrating to CloudSearch http://docs.aws.amazon.com/cloudsearch/latest/developerguide/preparing-data.html

6.      If the conditions are not met in a document, it will be skipped during migration. Skipped records are shown in the log file.

7.      If a field type (mapped to fields) is not stored, the stopwords mapped to that particular field type are ignored.

Example 1:

<field name="description" type="text_general" indexed="true" stored="true" />   

The above field ‘description’ will be considered for stopwords.

Example 2:

<field name="fileName" type="string" />     

The above field ‘fileName’ will not be migrated and ignored in the stopwords.

Apache Solr 4 Cook Book Review

Book Review : Apache Solr 4 Cookbook Review
Author: Rafal Kuc
Reviewed by : Harish Ganesan and Vijay Olety
Book URL : http://www.packtpub.com/apache-solr-4-cookbook/book

Comments : 

I recently read this book and i am really impressed! This book provides good understanding of Apache Solr for both developers as well as consultants.

The book starts off well with an introduction to Apache Solr, the web / app servers required, the role of Zookeeper, why clustering your data is vital ?, the various directory implementations, performance-oriented caching mechanisms, a sample crawler module which coupled with Solr gives a complete end-to-end solution, the role of Apache Tika as a extracting toolkit and the ease of customizing Solr. From then on, the book dwells into the details.

The first step is indexing. It plays a vital part in the entire search solution. Data can be in the form on .txt, .pdf or any other format. It is imperative that all such formats are easily indexable. One of the widely used tools for extracting metadata and language detection is Apache Tika. Data can also be present in a database, for which the Data Import Handler is handy. It comes in two variants – full and delta. Every detail is nicely explained with examples which can make the development time faster. DIH also helps us to modify the data while importing which I felt is a pretty neat feature! One of the nicest features included in Solr 4 is the ability to update single field in a document. I am not sure why this was included in the earlier versions but it’s a classic case of better late than never.

The next step in the pipeline is the data analysis which is achieved through the use of analyzers and tokenizers. Various use cases include elimination HTML and XML tags, copying the contents of one field to another and stemming words amongst others. The detailing that has gone into explaining every concept, the examples and the associated step-by-step explanation is really helpful.

Now that the data is indexed and the data preparation is completed, it’s time to query Apache Solr! Searches can be performed on individual words or on a phrase. You can boost or elevate certain documents over others based on your requirements. Simple concepts such as sorting and faceting of results to complex ones such as ignoring typos using n-grams and detecting duplicates are very simple to understand and perform. Faceting, in particular, is gaining momentum as it helps in implementing the auto-suggest feature and narrowing down the search criteria. A newly introduced feature called the pivot faceting was a much needed one and it vastly simplifies certain use cases related to faceting. Solr provides immense capabilities when it comes to querying and this book explains each of them in great detail taking real-world examples.

We indexed and queried the data. But as our application scales, we have to get our hands dirty and start fine-tuning the performance metrics in order to give a good user experience to our customers. This is where caches and its various flavors and granularities starts to make sense. Cache always plays a major role in any deployment and it is necessary to monitor Solr at all times to gauge its performance. This book can done a great job in clearly explaining the various types of caches, the commit operation and its impact on searchers and how to overcome these. This topic is really important for any Solr real-world deployment and this book has not let me down!

Apache Solr 4.0 introduced the most-awaited SolrCloud feature that allows us to use distributed indexing and searching. Setting up of SolrCloud cluster along with a Zookeeper ensemble to enable replication, fault-tolerance and high availability along with disaster recovery is a piece-of-a-cake now. I really appreciate the time and effort spent on documenting and explaining how to set up two collections inside a single cluster. It was a nightmare to find information on this particular topic when we implementing SolrCloud for one of our customers. But I am rest assured that others referring this book will save precious time of theirs. Adding / deleting nodes from a cluster is no longer a tedious task as the entire process is automated through the presence of Zookeeper nodes. The in-depth knowledge of the author in these topics is clearly visible and is of great help to all the readers. A touch on Zookeeper Rolling Restart, though off-topic, might enable readers to get a complete birds-eye view of the entire cluster. Certain features such as soft commit and NRT search have been explained in detail afterwards (under Real-life Situations) but I felt that at least a mention earlier on would have provided a much needed continuity in that section. For the geeky readers like me, a detailed description about load balancing across shards and replicas and their customizations, if any, would have added an extra amount of spice to this well-cooked food!

As with any other tool, Solr deployment too will run into some kind of a problem. This section details the common problems that are encountered and effective ways to overcome these. Shrinking the size of the index and allocating enough memory in advance amongst others are some of the solutions explained in detail and is clearly documented in this book.

Lastly, as every developer would have wanted it, the real-world scenarios are described and the various Solr concepts that were explained in earlier sections are put together as part of a complete end-to-end solution.

Any one trying Solr 4.0 must read this book in its entirety before recommending a Solr production architecture. As mentioned above, there are a few suggestions which if incorporated in this book would benefit readers. All in all, this book will be really helpful for the developers and consultants alike!

Cost Saving Tips : Part 2 : How right search technology choice saves cost in AWS ?

Search tier is becoming an important component of online tech stacks. Some of the widely used search software/services are Apache Solr , ElastiSearch and Amazon CloudSearch in the AWS infra. Choosing a right search technology is an essential parameter for efficient operations and cost savings in cloud. In this article we will explore the economics behind choosing Amazon CloudSearch vs Apache Solr(v3.6) on EC2 for the search tier. we will explore how you can cut leakages and save costs by making right choices for right requirements ?
I have taken 3 sample scenarios based on which the cost comparison is done and cost savings are concluded.

Scenario 1: A small application with constant load pattern

Application nature:

• Load Volatility pattern: Constant
• Utilization: Low to Medium
• Dependency on Search Layer: Low

Data requirements:

• Each document size is ~ 1 KB (For easy calculation purposes)
• 500 MB of Search Index data
• 50 K- 100 k requests per day
• Low concurrency

Batch and Index Rebuilds:

• 24 Batch uploads per day (each batch 100 documents of 1KB each).
• Explicit Index Rebuild once/twice a month
• 50 MB increase in Search index data per month

Administration Efforts:

• Initial Provisioning ( One time)
• Monitoring
• Regular Backups
• Index Rebuilds

	Amazon Cloud Search	Apache Solr v3.6
Compute	74.4	48.36
Storage (EBS)	-	12
Batch upload	0.10	0
Index Rebuild	4	0
Data IN/OUT	0	0
Admin efforts (Person Hrs/Month @ 75)	2	10
Administration cost	150	750
Total	~ 230	~811

• Amazon EC2 US-East-Region, Costs in USD ,1 Month of compute = 744 hrs, Instance type : Small
• EBS: 10 GB volume + 100 Million IO per month+ 10 GB snapshot for Apache Solr on EC2 Refer pricing @ http://aws.amazon.com/cloudsearch/
• Search Expert administration effort Hourly price averaged to minimum of 75 USD/hr 
• We can observe the infra costs are almost same, but the provisioning/admin/managing costs spikes up when it comes to Apache Solr on EC2, which can be minimized using Amazon CloudSearch. 

To know more about how Apache Solr compares with Amazon CloudSearch, Refer article:  http://harish11g.blogspot.in/2013/01/amazon-cloudsearch-vs-apache-solr_16.html

Scenario 2: Heavily utilized Search tier

Application nature:

• Load Volatility pattern: Peak and Valleys in a day
• Utilization: High
• Dependency on Search Layer: High

Data requirements:

• Each document size is ~ 1 KB (For easy calculation purposes)
• 50 GB of Search data (Index)
• 10 million requests per day, Each Response size is 10 KB, ~ 100 GB data out per day, 3 TB per month
• High concurrency

Batch and Index Rebuilds:

• 2048 Batch uploads per Month (each batch 5 MB of data).
• Explicit Index Rebuild 12 times a month
• Search Data growth: 10 GB index added every month

Administration Efforts:

• Initial Provisioning
• Partitioning and read scaling frequently
• Monitoring, Regular Backups and maintaining the HA
• Index Rebuilds

	Amazon Cloud Search	Apache Solr v3.6
Compute	~2865	~2440
Storage (EBS)	-	140
Batch upload	0.30	-
Index Rebuild	50	-
Data In/Out	-	-
Admin efforts (Person Hrs/Month @ 75)	8	24
Administration cost	600	1800
Total	~ 3515	~4380

• Amazon EC2 US-East-Region, Costs in USD ,1 Month of compute = 744 hrs, Instance type : Small
• 7 Xlarge Search Instances in Amazon Cloud Search + more instances depending upon growth
• EBS: 100 GB volume + 500 Million IO per month+ X GB snapshot for Apache Solr on EC2
• Apache Solr on EC2 Costs:  
• If we are scaling up the capacity of Solr Nodes in multiple phases from m1.xlarge to m2.4xlarge depending upon the index growth (10 GB in our case) every month, then lots of manual admin labour efforts is needed. This adds to our administration cost.  
• If we decide not to frequently scale up but start Solr initially itself with m2.4xlarge, then we have overprovisioned for first 5 months, which essentially means cost leakage again. Still monitoring, backups etc have to done on Apache Solr on EC2.The above table indicates cost calculated in this approach. Still at the end of 6th month, we need to Shard Solr on EC2 because it will exceed m2.4xlarge capacity / or Scale up again with more costly EC2 instances to keep up with the growth. Again labour efforts to be added.
• After having all this, there is no guarantee that Apache Solr on EC2 can handle the load, since the volatility pattern is spikey in nature for a day, there could be times where Solr is pounded and not performing well also. This under performance may lead to losing customers itself.
• Amazon CloudSearch eliminates all the scaling up/out/portioning complexities automatically. Labour cost is one of the important costs in large scale search tier setups and Amazon CloudSearch helps us keep it at minimum as we grow. Larger and more elastic our search setup requirements, then Amazon CloudSearch will easily beat the hell out of Apache Solr on EC2.
• Scale out during heavy load is automatic in CloudSearch and it is a manual cumbersome effort in Apache Solr on EC2. Note Scale out based on Load costs are not calculated in both Apache Solr and Amazon CloudSearch.

To know more about Apache SolrCloud deployment best practices on Amazon VPC, Refer article:  http://harish11g.blogspot.in/2013/03/Apache-Solr-cloud-on-Amazon-EC2-AWS-VPC-implementation-deployment.html

Scenario 3: Seasonal Loads 

Application nature:

• Load Volatility pattern: Seasonal load (1 week campaign every 2 months), other times minimal activity
• Utilization: High
• Dependency on Search Layer: High

Data requirements:

• Each document size is ~ 1 KB (For easy calculation purposes)
• Getting started with 5 GB of Search data
• ~750 million requests (week) or more during the campaign week
• 12 hours heavy utilization and 12 hours under utilization during campaign days.
• 10 million requests during normal days
• High concurrency during campaign week

Batch and Index Rebuilds:

• 512 Batch uploads per Month (each batch 5 MB of data).
• Explicit Index Rebuild 12 times a month
• Search Data growth: 2.5 GB added every month

Administration Efforts:

• Initial Provisioning
• Partitioning and read scaling frequently
• Monitoring, Regular Backups and maintaining the HA
• Index Rebuilds

	Amazon Cloud Search	Apache Solr v3.6
Compute	~410 + 150 (Scale out)	~357+ 70 (scale out)
Storage (EBS)	-	50
Batch upload	0.10	-
Index Rebuild	5	-
Data In/Out	-	-
Admin efforts (Person Hrs/Month @ 75)	10	40
Administration cost	750	3000
Total	~ 1320	~3477·

Amazon EC2 US-East-Region, Costs in USD ,1 Month of compute = 744 hrs, Instance type : Small

Amazon CloudSearch Costs

• 1 Xlarge Search Instances in Amazon Cloud Search during normal days
• Imagine 3 Additional Xlarge Instance are spawned during campaign period for 1 week
• Automated Scale out and Scale down. No efforts needed.
• Morning till night when heavy utilization is there additional xlarge (3 new) is launched. Night till morning where not much load is there, these additional instances will be removed accordingly.

Apache Solr on EC2 Costs:  

• 2 X m1.large EC2 instances for Solr on Normal Days
• 2 new additional m1.large instances during campaign period
• Manual effort to scale out before campaign week and scale down post campaign week.
• Additional EC2 instances are used all 24 hrs during the campaign week

From the above analysis we can infer that Amazon CloudSearch is cost effective for all labor dependent scenarios compared to Apache Solr. It is recommended to use Amazon CloudSearch as part of your stack for cost efficient operations, unless there is a strong need for Apache Solr case.

Note : I architected a solution using Apache SolrCloud (4.0) after this article. Apache SolrCloud seems to drastically reduce the complexity involved in Distributing search queries, Sharding and replication. But it does not offer automatic shard increase/decrease or replica addition during loads. It is less labor intensive than Solr 3.6, but still not comparable to Amazon CloudSearch in the infra aspects. 

To know more about How to migrate from Apache Solr to Amazon CloudSearch Refer article: http://harish11g.blogspot.in/2013/03/migration-from-Apache-Solr-to-Amazon-CloudSearch.html

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage