CloudWatch + Lambda Case 4: Control launch of Specific “C” type EC2 instances post office hours to save costs

We have a customer who has predictable load volatility between 9 am to 6 pm and uses specific large EC2 instances during office hours for analysis, they use “c4.8xlarge” for that purpose. Their IT wanted to control launch of such large instance class post office hours and during nights to control costs, currently there is no way to restrict or control this action using Amazon IAM. In short we cannot create complex IAM policy with conditions that user A belonging to group A cannot launch instance type C every day between X and Y.
Some stop gap followed is to have a job running which removes the policy from an IAM user when certain time conditions are met. So basically what we would do is, to have a job that calls an API that removes the policy which restricts an IAM user or group from launching instances. This will make the IAM policy management complex and tough to assess/govern drifts between versions.
After the introduction of the CloudWatch events our Cloud operations started controlling it with lambda functions. Whenever a Instance type is launched it will trigger a lambda function, the function will filter whether it is a specific “C” type and check for the current time, if the time falls after office hours, it will terminate the EC2 instance launched immediately.
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.



The next step would be configuring rule details with Rule definition



Finally we will review the Rules Summary



Amazon Lambda Function Code Snippet (Python)
import boto3

def lambda_handler(event, context):
    #print ("Received event: " + json.dumps(event, indent=2))
    #print ("************************************************")
 
    ec2_client = boto3.client("ec2")
 
 
    print "Event Region :", event['region']
 
    event_time = event['detail']['eventTime']
    print "Event Time :", event_time
 
    time = event_time.split('T')
    t = time[1]
    t = t.split(':')
    hour = t[0]
 
    instance_type = event['detail']['requestParameters']['instanceType']
    print "Instance Type:", instance_type
 
    instance_id = event['detail']['responseElements']['instancesSet']['items'][0]['instanceId']
    print "Instance Id:",instance_id
 
    if( instance_type.startswith( 't' ) and hour > 18 or hour < 8 ):
        print ec2_client.terminate_instances( InstanceIds = [ instance_id ] )

GitHub Gist URL:  https://github.com/cloud-automaton/automaton/blob/master/aws/events/TerminateAWSEC2.py

This post was co authored with Priya and Ramprasad of 8KMiles

CloudWatch + Lambda Case 3 -Controlling cross region EBS/RDS Snapshot copies for regulated industries

If you are part of regulated industry like Pharmaceutical/ Life sciences/BFSI running mission critical applications on AWS, at times as part of the compliance requirements you will have to restrict/control data movement to a particular geographic region in the cloud. This is becomes complex to restrict some times. Let us explore in detail:
We all know there are varieties of ways to move data from one AWS region to another, but one commonly used method is Snapshot copy across AWS regions. Usually you can restrict snapshot copy permission in IAM Policy, but what if you need the permission enabled for moving data between AWS accounts inside a region, but still want to control EBS/RDS snapshot copy action across regions. It can be only mitigated by automatically deleting the snapshot on destination AWS region in case snapshot copy activity is done.
Our Cloud operations team used to altogether remove this permission in IAM or monitor this activity using polling scripts for customers with multiple accounts who need this permission and still need control. Now after the introduction of CloudWatch Events we have configured a rule that points to an AWS Lambda which gets triggered in near real time when snapshot is copied to destination AWS region. The lambda function will initiate a deletion process immediately. Though it is reactive it is incomparably faster than manual intervention.
In this use case, Amazon CloudWatch Event will identify the EBS Snapshot copies across the regions and delete them.
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.



The next step would be configuring rule details with Rule definition


Finally we will review the Rules Summary




Amazon Lambda Function Code Snippet (Python)



GitHub Gist URL: https://github.com/cloud-automaton/automaton/blob/master/aws/events/AWSSnapShotCopy.py

This post was co authored with Muthukumar and Ramprasad of 8KMiles

CloudWatch + Lambda Case 2- Keeping watch on AWS ROOT user activity is normal or anomaly ?

As a Best Practice you should never use your AWS root account credentials to access AWS. Instead, create individual (IAM) users for anyone who needs access to your AWS account. This allows you to give each IAM user a unique set of security credentials and grant different permissions to each user. Example: Create an IAM user for yourself as well, give that user administrative privilege, and use that IAM user for all your work and never share your credentials to anyone else.
Usually Root has full access and it is not ideal to restrict the same in AWS IAM. Imagine you suddenly doubt some anomaly/suspicious activities done as Root user (using EC2 API’s etc) in your logs other than normal IAM user provisioning; this could be because Root user is compromised or forced, but ultimately it is a deviation from the best practice.
In the past we used to poll the CloudTrail logs using programs and differentiate between “root” and “Root”, and our cloud operations used to react to these anomaly behaviors. Now we can inform the cloud operations and customer stake holders near real time using CloudWatch events.
In this use case, Amazon CloudWatch Event will identify activities if any performed by an AWS ROOT user and notifications will be sent to SNS thru AWS Lambda.
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target. The lambda function will detect if the event is triggered by root user and notifies through SNS.


The next step would be configuring rule details with Rule definition



Finally we will review the Rules Summary



Amazon Lambda Function Code Snippet (Python)



GitHub Gist URL:  https://github.com/cloud-automaton/automaton/blob/master/aws/events/TrackAWSRootActivity.py

This post was co authored with Saravanan and Ramprasad of 8KMiles

CloudWatch + Lambda Case 1- Avoid malicious CloudTrail action in your AWS Account

As many of you know AWS CloudTrail provides visibility into API activity in your AWS account, Cloud Trail Logging lets you see which actions users have taken and which resources have been used, along with details such as the time and date of actions and the actions that have failed because of inadequate permissions. It enables you to answer important questions such as which user made an API call or which resources were acted upon in an API call. If a user disables CloudTrail logs accidentally or with malicious intent then audit logging events will not captured and hence you fail to have proper governance in place. The situation will get complex, If the user disables- enables back CloudTrail for a brief period of time where some important activities can go unlogged and unaudited. In short once CloudTrail logging is enabled it should not be disabled and this action needs to be defended in depth.
Our Cloud operations team had earlier written a program that periodically scans the Cloud Trail logs entries, if any log activity was missing after an X period of time it alerts the operations.  Overall reaction time on our cloud operations was >15-20 mins to mitigate this CloudTrail disable action.
Now after the introduction of CloudWatch Events we have configured a rule that points to an AWS Lambda function as target. This function gets triggered in near real time when CloudWatch is disabled and automatically enables it back without any manual interaction from Cloud operations. The advanced version of the program triggers workflow which logs entries into ticket system as well. This event model has helped us reduce the mitigation to less than a minute.
We have illustrated below the detailed steps on how to configure this event. Also we given the link for GIT with basic AWS Lambda Python code that can be used by your cloud operations.
In this use case, Amazon CloudWatch Event will identify whether an AWS account has got CloudTrail enabled or not, if not enabled, Amazon CloudWatch Events will take corrective actions by enabling the same.
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.

The next step would be configuring rule details with Rule definition

Finally we will review the Rules Summary

Amazon Lambda Function Code Snippet (Python)

import json

import boto3

print('Loading function')

""" Function to define Lambda Handler """

def lambda_handler(event, context):

    try:

        

        client = boto3.client('cloudtrail')

        if event['detail']['eventName'] == 'StopLogging':

            response = client.start_logging(Name=event['detail']['requestParameters']['name'])

        

    except Exception, e:

        sys.exit();

GitHub Gist URL:  https://github.com/cloud-automaton/automaton/blob/master/aws/events/EnableAWSCloudTrail.py

This post was co authored with Mohan and Ramprasad of 8KMiles

Pointers for running a Regulated Pharma System in the Public Cloud

Article Source: https://www.linkedin.com/pulse/considerations-running-regulated-system-public-cloud-shah-nawaz

As the Cloud becomes increasingly ubiquitous, the areas in which they can be adopted also becomes more widespread.  Regulated industries are not immune to this. Soon we will all have to figure out how to run our workloads in the Cloud while maintaining our compliance and privacy posture. If we go back in time and look at each application that had a service regulated (GxP or 21 CFR Part 11 ) workload each piece of the software, platform, server, storage, network, even the data center was individually qualified and then individually validated to the appropriate regulation.
 The Application should be validated; IT infrastructure should be qualified. (EU GMP Annex 11, 2011)
Cloud is everywhere, so are we going to individually qualify and validate each building block? Certainly that seems like job security, but wouldn’t that make the cloud less compelling, and in some case unusable?

So who should do what for a GXp hosted application?

If you have data privacy needs these should be tested as part of the validation test and formally documented and quality still needs to be addressed.

Platform Qualification documents are still needed when a regulated / validated application is hosted in a cloud environment “

You also have to realize that the current concepts of computer system validation (CSV) do not work well, e.g how does one perform an installation qualification (IQ) in the cloud when one does not know the serial number of the machine on which the software will be installed, nor in some cases its location. So we must pay attention to the purpose of the IQ , not to the implementation of the IQ  and by extension, we must consider the purpose of CSV, not just its current practice. Any task carried out in the regulated domain should have at least the attributes i.e (Repeatability, Ability to Audit & Non-repudiation) whether paper-based or computer-based in house or in the cloud.

The idea is to controlling your data, and who can access it and what they do (and did to it…) upon accessed.

The approach for compliance in the cloud needs to be different. If done correctly, compliance in the cloud can be far more efficient than any other means for providing complaint applications. Instead of worrying about qualifying each building block, the cloud vendor qualifies the platform once, to many standards and many certifications.  Most of the Tier-1 cloud vendors provide these qualifications to any customer who needs to run a validated application on the cloud vendor’s platform. At least that is the approach many of the Cloud providers are taking.  The provider qualifies the platform; the customer (or partner) validates the application.
Whether you are considering Infrastructure Services i.e (AWS, Microsoft Azure, Google compute, Digital Ocean, Platform as a Service (AWS - Machine Learning MI, AWS RDS…), software as a service, thinking of putting your applications on Amazon Web Service, Azure or simply enabling your business with CRM (Veeva) , Office365 or google analytic, most providers offer detailed documentation and certifications across a wide range of standards.
This enables their regulated customer and partners to run validated applications.

Your situation may vary, as each customers QA has a different viewpoint of necessary qualifications and documentation to support a validated application. It is certainly no longer a question if one can qualify and/or validate their application to run in a cloud. It will come down to your Quality and Compliance process, and to what extend we can amend IQ/PQ/OQ to support distinct system categories running in cloud. For example, a clinical trail portal has different levels of risk then back-office HRMS applications and is thus validated to a different level. ONE-SIZE-FITS-ALL doesn’t works. One needs to understand the unique nature and risk profile of each application and then determine what is appropriate. It may have been convenient or perhaps a standard approach for traditional data-center delivered resources, but one needs to rethink some of those measures and practices that were adopted. Now that you are considering running them in a cloud each system would need to be evaluated for its validation posture.
Various approached have been documented and talked about as it relates to running sensitive workloads in the Cloud, how life science organizations are using the Cloud across the value chain (Clinical Information System, Patient Data Archive, Patient Population Risk / Pattern Assessment, Genomic, and more…) and what levels of qualification documentation vendors must provide to customers in regulatory environments. This article considers some basic and fundamental approach to the Cloud.

Security is Everyone Business:
Although security, by far remains the biggest concern, it can also be viewed as an enabler. However, one can’t build security services with the same develop and operate mindset that we have been using for decades. As we move into the cyber security era, machine-learning more pervasive techniques needs to be considered… (There could be an entire article on this…)

"Design security in the context of the Cloud”.

Suggested Approach:

Regulated or non-regulated, it is apparent that at the end of the day one needs assurance as to who has access and what they did with that state i.e ( Data/system). We also need to have traceability and repeatability by which you can audit/report and resolve issues that are prudent to any software development lifecycle. ( Suggested Model below )

Regulatory considerations for the use of cloud computing will depend on the services you consume. Not all services are created equally to satisfy compliance requirements

Rethink Operation Capabilities:

Your operation will be different, Tower Concept will not work nor can it scale to the ever-increasing demands. Consider adopting capabilities that can provide the appropriate assurance and governance to operate your Cloud services and at the same time promote agility and time to delivery service.

Image Source: 8kmiles.com

Summary:

• Codify your Infrastructure
• Pre-Qualify your environment prior to loading GxP workloads
• Build pre-validated images for Cloud usage
• Adopt a version control & release methodology
• Automate your stack as much as you can
• Automate your testing
• Test, test and test before you start to move workloads
• To maximize effectiveness and minimize risk (and ultimately cost), security and privacy must be considered from the outset of any Cloud implementation not after implementation and deployment
• Cloud providers (Iaas and Paas) are generally not aware of a specific sectors security, privacy and regulatory needs of your sector, so design in the context of your organization
• Adopt V-Model by Design (http://en.wikipedia.org/wiki/V-Model_(software_development))

References:

http://en.wikipedia.org/wiki/Validation_%28drug_manufacture%29

http://ec.europa.eu/health/files/eudralex/vol-4/annex11_01-2011_en.pdf

http://www.ispe.org/gamp-5

http://www.prismforum.org

Apache Solr to Amazon CloudSearch migration tool

In this post, we are introducing a new tool called S2C – Apache Solr to Amazon CloudSearch Migration Tool. S2C is a Linux console based utility that helps developers / engineers to migrate search index from Apache Solr to Amazon CloudSearch.

Very often customers initially build search for their website or application on top of Solr, but later run into challenges like elastic scaling and managing the Solr servers. This is a typical scenario we have observed in our years of search implementation experience. For such use cases, Amazon CloudSearch is a good choice. Amazon CloudSearch is a fully-managed service in the cloud that makes it easy to set up, manage, and scale a search solution for your website. To know more, please read the Amazon CloudSearch documentation

We are seeing growing trend every year, organizations of various sizes are migrating their workloads to Amazon CloudSearch and leveraging the benefits of fully managed service. For example, Measured Search, an analytics and e-Commerce platform vendor, found it easier to migrate to Amazon CloudSearch rather than scale Solr themselves (see article for details).

Since Amazon CloudSearch is built on top of Solr, it exposes all the key features of Solr while providing the benefits of a fully managed service in the cloud such as auto-scaling, self-healing clusters, high availability, data durability, security and monitoring.

In this post, we provide step-by-step instructions on how to use the Apache Solr to Amazon CloudSearch Migration (S2C) tool to migrate from Apache Solr to Amazon CloudSearch.

Before we get into detail, you can download the S2C tool in the below link.
Download Link: https://s3-us-west-2.amazonaws.com/s2c-tool/s2c-cli.zip

Pre-Requisites

Before starting the migration, the following pre-requisites have to be met. The pre-requisites include installations and configuration on the migration server. The migration server could be the same Solr server or independent server that sits between your Solr server and Amazon CloudSearch instance.  

Note: We recommend running the migration from the Solr server instead of independent server as it can save time and bandwidth. It is much better if the Solr server is hosted on EC2 as the latency between EC2 and CloudSearch is relatively less.

The following installations and configuration should be done on the migration server (i.e. your Solr server or any new independent server that connects between your Solr machine and Amazon CloudSearch).    1.      The application is developed using Java. Download and Install Java 8 .Validate the JDK path and ensure the environment variables like JAVA_HOME, classpath, path is set correctly. 2.      We assume you already have setup Amazon Web services IAM account. Please ensure the IAM user has right permissions to access AWS services like CloudSearch. Note: If you do not have an AWS IAM account with above mentioned permissions, you cannot proceed further. 3.      The IAM user should have AWS Access key and Secret key. In the application hosting server, set up the Amazon environment variables for access key and secret key. It is important that the application runs using the AWS environment variables. To setup AWS environment variables, please read the below link. http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html Alternatively, you can set the following AWS environment variables by running the commands below from Linux console. export AWS_ACCESS_KEY=Access Key   export AWS_SECRET_KEY=Secret Key 4.      Note: This step is applicable only if the application is hosted on Amazon EC2. If you do not have an AWS Access key and Secret key, you can opt for IAM role attached to an EC2 instance. A new IAM role can be created and attached to EC2 during the instance launch. The IAM role should have access to AWS resources like S3, DynamoDB and CloudSearch. For more information, read the below link http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html 5.      Download the migration utility ‘S2C’, unzip the tool and copy it in your working directory.  Download Link : https://s3-us-west-2.amazonaws.com/s2c-tool/s2c-cli.zip

S2C Utility File

The downloaded ‘S2C’ migration utility should have the following sub directories and files.

Folder / Files	Description
bin	Binaries of the migration tool
lib	Libraries required for migration
application.conf	Configuration file that allows end users to input parameters	Require end-user’s input.
logback.xml	Log file configuration	Optional. Does not require end-user  / developer input
s2c	script file that executes the migration process

Configure only application.conf and logback.xml.  Do not modify any other file.                                                                                                                                         

application.conf

The application.conf file has the configuration related to the new Amazon CloudSearch domain that will be created. The parameters configured in the in the application.conf file are explained in the table below.

s2c {   api {     SchemaParser = "s2c.impl.solr.DefaultSchemaParser"     SchemaConverter = "s2c.impl.cs.DefaultSchemaConverter"     DataFetcher = "s2c.impl.solr.DefaultDataFetcher"     DataPusher = "s2c.impl.cs.DefaultDataPusher"  }	List of API that is executed step by step during the migration. Do not change this.
solr {     dir = "files"     server-url = "http://localhost:8983/solr/collection1"     fetch-limit = 100   }	dir – The base directory path of Solr. Ensure the directory is present and also its validity. Eg:/opt/solr/example/solr/collection1/conf
	server-url – Server host, port and collection path The endpoint which will be used to fetch the data. If the utility is run from a different server, ensure the IP address and port has firewall access.
	fetch-limit – number of solr documents that can be fetched for each batch call This configuration number should be carefully set by the developer. The fetch limit depends on the following factors: 1.  Record size of a Solr record(1KB or 2KB) 2.  Latency between migration server and Amazon CloudSearch 3.  Current Request Load on the Solr Server E.g.: If the total Solr documents is 100000 and fetch limit is 100, then it would take 100000 / 10 = 10000 batch calls to complete the fetch. If size of each Solr record is 2KB, then 100000 * 2KB = 200MB data is migrated.
cs {     domain = "collection1"     region = "us-east-1"     instance-type = " search.m3.xlarge"     partition-count = 1     replication-count = 1   }	domain - CloudSearch domain name Ensure that the domain name does not already exist.
	Region – AWS region for the new CloudSearch domain
	Instance type – Desired instance type for CloudSearch nodes Choose the instance type based on the volume of data and the expected query volume.
	Partition count – Number of partitions required for CloudSearch
	replication-count - Replication count for CloudSearch
wd = "/tmp"	Temporary file path to store intermediate data files and migration log files

Running the migration

Before launching the S2C migration tool, verify the following:

1.      Solr directory path – Make sure that the Solr directory path is valid and available. The tool cannot read the configuration if the path or directory is invalid.

2.      Solr configuration contents - Validate that the Solr configuration contents are correctly set inside the directory. Example: solrconfig.xml, schema.xml, stopwords.txt, etc.

3.      Make sure that the working directory is present in the file system and has write permissions for the current user. It can be an existing directory or a new directory. The working directory stores the fetched data from Solr and migration logs.

4.      Validate the disk size before starting the migration. If the available free disk space is lesser than the size of the Solr index, the fetch operations will fail.

For example, if the Solr index size is 7 GB, make sure that the disk has at least 10 GB to 20 GB of free space.

Note: The tool reads the data from Solr and stores in a temporary directory (Please see configuration wd = /tmp in the above table).

5.      Verify that the AWS environment variables are set correctly. The AWS environment variables are mentioned in the pre-requisites section above.

6.      Validate the firewall rules for IP address and ports if the migration tool is run from a different server or instance. Example: Solr default port 8983 should be opened to the EC2 instance executing this tool.

Run the following command from directory ‘{S2C filepath}’ example: /build/install/s2c-cli

./s2c Or JVM_OPTS="-Xms2048m -Xmx2048m" ./s2c (With heap size)

The above will invoke the shell ‘s2c’ script that starts the search migration process. The migration process is a series of steps that require user inputs as shown in the screen shots below.

Step 1: Parse the Solr schema

The first step of migration prompts for a confirmation to parse the Solr schema and Solr configuration file. During this step, the application generates a ‘Run Id’ folder inside the working directory.

Example: /tmp/s2c/m1416220194655

The Run Id is a unique identifier for each migration. Note down the Run Id as you will need it to resume the migration in case of any failure.

Step 2: Schema conversion from Solr to CloudSearch
The second step prompts confirmation to convert Solr schema to CloudSearch schema. Press any key to proceed further.

The second step will also list all the converted fields which are ready to be migrated from Solr to CloudSearch. If any fields are left out, this step will allow you to correct the original schema. User can abort the migration and identify the ignored fields, rectify the schema and re-run the migration again.

The below screen shot shows the fields ready for CloudSearch migration.

Step 3: Data Fetch

The third step prompts for confirmation to fetch the search index data from the Solr server. Press any key to proceed. This step will generate a temporary file which will be stored in the working directory. This temporary file will have all the fetched documents from the Solr index.

There is also option to skip the fetch process if all the Solr data is already stored in the temporary file. If this is the case, the prompt will look like the screenshot below.

Step 4: Data push to CloudSearch

The last and final step prompts for confirmation to push the search data from the temporary file store to Amazon CloudSearch. This step also creates the CloudSearch domain with the configuration specified in application.conf including desired instance type, replication count, and multi-AZ options.

If the domain is already created, the utility will prompt to use the existing domain. If you do not wish to use an existing domain, you can create a new CloudSearch domain using the same prompt.

Note: The console does not prompt for any ‘CloudSearch domain name’ but instead it uses the domain name configured in the application.conf file.

Step 5: Resume (Optional)

During the migration steps, if there is any failure during the fetch operation, it can be resumed. This is illustrated in the screen shot below. 

Step 6: Verification

Log into AWS CloudSearch management console to verify that the domain and index fields.

Amazon CloudSearch allows running test queries to validate the migration and as well the functionality of your application.

Features supported

1.      Support for other non-Linux environments is not available for now.

2.      Support for Solr Shards is not available for now. The Solr shard needs to be migrated separately.

3.      The install commands may vary for different Linux flavors. Example installing software, file editor command, permission set commands can be different for every Linux flavors. It is left to engineering team to choose the right commands during the installation and execution of this migration tool.

4.      Only fields configured as ‘stored’ in Solr schema.xml are supported. The non-stored fields are ignored during schema parsing.

5.      The document id (unique key) is required to have following attributes:

a.      Document ID should be 128 characters or less in size.

b.      Document ID can contain any letter, any number, and any of the following characters:      _ - = # ; : / ? @ &

c.       The below link will help you to understand in data  preparation before migrating to CloudSearch http://docs.aws.amazon.com/cloudsearch/latest/developerguide/preparing-data.html

6.      If the conditions are not met in a document, it will be skipped during migration. Skipped records are shown in the log file.

7.      If a field type (mapped to fields) is not stored, the stopwords mapped to that particular field type are ignored.

Example 1:

<field name="description" type="text_general" indexed="true" stored="true" />   

The above field ‘description’ will be considered for stopwords.

Example 2:

<field name="fileName" type="string" />     

The above field ‘fileName’ will not be migrated and ignored in the stopwords.