CloudWatch + Lambda Case 4: Control launch of Specific “C” type EC2 instances post office hours to save costs

We have a customer who has predictable load volatility between 9 am to 6 pm and uses specific large EC2 instances during office hours for analysis, they use “c4.8xlarge” for that purpose. Their IT wanted to control launch of such large instance class post office hours and during nights to control costs, currently there is no way to restrict or control this action using Amazon IAM. In short we cannot create complex IAM policy with conditions that user A belonging to group A cannot launch instance type C every day between X and Y.
Some stop gap followed is to have a job running which removes the policy from an IAM user when certain time conditions are met. So basically what we would do is, to have a job that calls an API that removes the policy which restricts an IAM user or group from launching instances. This will make the IAM policy management complex and tough to assess/govern drifts between versions.
After the introduction of the CloudWatch events our Cloud operations started controlling it with lambda functions. Whenever a Instance type is launched it will trigger a lambda function, the function will filter whether it is a specific “C” type and check for the current time, if the time falls after office hours, it will terminate the EC2 instance launched immediately.
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.



The next step would be configuring rule details with Rule definition



Finally we will review the Rules Summary



Amazon Lambda Function Code Snippet (Python)
import boto3

def lambda_handler(event, context):
    #print ("Received event: " + json.dumps(event, indent=2))
    #print ("************************************************")
 
    ec2_client = boto3.client("ec2")
 
 
    print "Event Region :", event['region']
 
    event_time = event['detail']['eventTime']
    print "Event Time :", event_time
 
    time = event_time.split('T')
    t = time[1]
    t = t.split(':')
    hour = t[0]
 
    instance_type = event['detail']['requestParameters']['instanceType']
    print "Instance Type:", instance_type
 
    instance_id = event['detail']['responseElements']['instancesSet']['items'][0]['instanceId']
    print "Instance Id:",instance_id
 
    if( instance_type.startswith( 't' ) and hour > 18 or hour < 8 ):
        print ec2_client.terminate_instances( InstanceIds = [ instance_id ] )

GitHub Gist URL:  https://github.com/cloud-automaton/automaton/blob/master/aws/events/TerminateAWSEC2.py

This post was co authored with Priya and Ramprasad of 8KMiles

CloudWatch + Lambda Case 3 -Controlling cross region EBS/RDS Snapshot copies for regulated industries

If you are part of regulated industry like Pharmaceutical/ Life sciences/BFSI running mission critical applications on AWS, at times as part of the compliance requirements you will have to restrict/control data movement to a particular geographic region in the cloud. This is becomes complex to restrict some times. Let us explore in detail:
We all know there are varieties of ways to move data from one AWS region to another, but one commonly used method is Snapshot copy across AWS regions. Usually you can restrict snapshot copy permission in IAM Policy, but what if you need the permission enabled for moving data between AWS accounts inside a region, but still want to control EBS/RDS snapshot copy action across regions. It can be only mitigated by automatically deleting the snapshot on destination AWS region in case snapshot copy activity is done.
Our Cloud operations team used to altogether remove this permission in IAM or monitor this activity using polling scripts for customers with multiple accounts who need this permission and still need control. Now after the introduction of CloudWatch Events we have configured a rule that points to an AWS Lambda which gets triggered in near real time when snapshot is copied to destination AWS region. The lambda function will initiate a deletion process immediately. Though it is reactive it is incomparably faster than manual intervention.
In this use case, Amazon CloudWatch Event will identify the EBS Snapshot copies across the regions and delete them.
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.



The next step would be configuring rule details with Rule definition


Finally we will review the Rules Summary




Amazon Lambda Function Code Snippet (Python)



GitHub Gist URL: https://github.com/cloud-automaton/automaton/blob/master/aws/events/AWSSnapShotCopy.py

This post was co authored with Muthukumar and Ramprasad of 8KMiles

CloudWatch + Lambda Case 2- Keeping watch on AWS ROOT user activity is normal or anomaly ?

As a Best Practice you should never use your AWS root account credentials to access AWS. Instead, create individual (IAM) users for anyone who needs access to your AWS account. This allows you to give each IAM user a unique set of security credentials and grant different permissions to each user. Example: Create an IAM user for yourself as well, give that user administrative privilege, and use that IAM user for all your work and never share your credentials to anyone else.
Usually Root has full access and it is not ideal to restrict the same in AWS IAM. Imagine you suddenly doubt some anomaly/suspicious activities done as Root user (using EC2 API’s etc) in your logs other than normal IAM user provisioning; this could be because Root user is compromised or forced, but ultimately it is a deviation from the best practice.
In the past we used to poll the CloudTrail logs using programs and differentiate between “root” and “Root”, and our cloud operations used to react to these anomaly behaviors. Now we can inform the cloud operations and customer stake holders near real time using CloudWatch events.
In this use case, Amazon CloudWatch Event will identify activities if any performed by an AWS ROOT user and notifications will be sent to SNS thru AWS Lambda.
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target. The lambda function will detect if the event is triggered by root user and notifies through SNS.


The next step would be configuring rule details with Rule definition



Finally we will review the Rules Summary



Amazon Lambda Function Code Snippet (Python)



GitHub Gist URL:  https://github.com/cloud-automaton/automaton/blob/master/aws/events/TrackAWSRootActivity.py

This post was co authored with Saravanan and Ramprasad of 8KMiles

CloudWatch + Lambda Case 1- Avoid malicious CloudTrail action in your AWS Account

As many of you know AWS CloudTrail provides visibility into API activity in your AWS account, Cloud Trail Logging lets you see which actions users have taken and which resources have been used, along with details such as the time and date of actions and the actions that have failed because of inadequate permissions. It enables you to answer important questions such as which user made an API call or which resources were acted upon in an API call. If a user disables CloudTrail logs accidentally or with malicious intent then audit logging events will not captured and hence you fail to have proper governance in place. The situation will get complex, If the user disables- enables back CloudTrail for a brief period of time where some important activities can go unlogged and unaudited. In short once CloudTrail logging is enabled it should not be disabled and this action needs to be defended in depth.
Our Cloud operations team had earlier written a program that periodically scans the Cloud Trail logs entries, if any log activity was missing after an X period of time it alerts the operations.  Overall reaction time on our cloud operations was >15-20 mins to mitigate this CloudTrail disable action.
Now after the introduction of CloudWatch Events we have configured a rule that points to an AWS Lambda function as target. This function gets triggered in near real time when CloudWatch is disabled and automatically enables it back without any manual interaction from Cloud operations. The advanced version of the program triggers workflow which logs entries into ticket system as well. This event model has helped us reduce the mitigation to less than a minute.
We have illustrated below the detailed steps on how to configure this event. Also we given the link for GIT with basic AWS Lambda Python code that can be used by your cloud operations.
In this use case, Amazon CloudWatch Event will identify whether an AWS account has got CloudTrail enabled or not, if not enabled, Amazon CloudWatch Events will take corrective actions by enabling the same.
As a first step, we will be creating a rule in Amazon CloudWatch Events dashboard. We have chosen AWS API Call as an Event to be processed by an AWSCloudTrail Lambda function as a target.

The next step would be configuring rule details with Rule definition

Finally we will review the Rules Summary

Amazon Lambda Function Code Snippet (Python)

import json

import boto3

print('Loading function')

""" Function to define Lambda Handler """

def lambda_handler(event, context):

    try:

        

        client = boto3.client('cloudtrail')

        if event['detail']['eventName'] == 'StopLogging':

            response = client.start_logging(Name=event['detail']['requestParameters']['name'])

        

    except Exception, e:

        sys.exit();

GitHub Gist URL:  https://github.com/cloud-automaton/automaton/blob/master/aws/events/EnableAWSCloudTrail.py

This post was co authored with Mohan and Ramprasad of 8KMiles

Load Testing tool comparison - JMeter on it's own vs JMeter & BlazeMeter together

Load testing is an important aspect of web applications life cycle on Amazon Cloud. Some of our customers ask us to generate 50000+ RPS to load test the scalability of their application deployed on Amazon cloud. Whenever we used to help such customers and migrate their applications on Amazon cloud for achieving scalability, load testing phase itself becomes a pain. Setting up the Load Testing infrastructure, writing automation around it, Managing, Maintaining and monitoring the load test infrastructure is an headache. Our Load testers and Infrastructure teams were spending considerable time and efforts on the above , instead of focusing only on load testing. We usually work with variety of tools from Grinder, JMeter, HP Load Runner to custom engineered load testing tools during the load testing phase. Some time back , our team started playing around with a SAAS based load testing tool called BlazeMeter. In this article i am going to share our experience in the form comparison between BlazeMeter and JMeter and why BlazeMeter has a bright future.
Blazemeter is a Saas based high scalable load testing tool that handle up to 300,000+ concurrent users. Their load test infrastructure is spread across major AWS regions. Since most of us have been using JMeter for years , the 100 % compatibility  it provides to existing JMeter scripts is a good feature. Blazemeter also provides a Chrome Extension which can record browser actions & convert it to .jmx file.

10 Things i like about BlazeMeter

Point 1) Load Test becomes effective only when the load comes from different IP Addresses similar to real world scenario and not from a single source IP. When multiple virtual user load is generated from the same IP, the router as well as the server tries to cache information and optimize the throughput many times. Hence by using multiple IP addresses for the host, the EC2 server will get an illusion of receiving requests from multiple source IP's.  Also it is better that load is generated from multiple IP's for Amazon ELB to evenly distribute load. Refer URL. BlazeMeter has capability to generate load from IP's which is very important on load testing the cloud applications.

Point 2) Customizing the Network Emulation: Usually online applications will be accessed from multiple devices like PC's, Laptop and mobiles. These devices have multiple network types such as 3G, broad band etc.Also at times times our online application will be accessed from locations which has poor network bandwidth , Both these parameters play an important role in capacity planning and load testing. We can chose the Bandwidth and network type emulation while doing the load test using BlazeMeter. Example we can configure the network type such as Unlimited Internet, 3G, Cable, Wifi etc and. Bandwidth download limit per device can also be set.

Point 3) Controlling the Throughput: Target throughput is a parameter of Apache JMeter that can be used to achieve a required throughput value of the application. A server’s performance need not always satisfy the target throughput value mentioned in JMeter. It could provide more throughput or lesser.The target throughput parameter can be controlled in run time in Blazemeter. Live server monitoring can help us identify if our servers are performing well for say 5000 Hits/sec & change the throughput value in run-time to a higher or lower value based on the server’s performance.

Point 4) Controlling the Agents: Apache JMeter works based on Master-Agent based architecture where the Master controls multiple agents generating the load. The number of agents parameter has to be usually decided before the starting of the test while using JMeter based load testing on Amazon Cloud. Option to dynamically change the throughput value is a very good feature to have while load testing a cloud application requiring thousands of Requests per second. BlazeMeter enables us to add or remove agent instances when a test is running. Any instance can be marked as Master or Slave(Agent) while the test is running.

Point 5) Controlling No. of Simulated Users on Slaves (Agents) : A load test strategy is mainly determined by following parameters like number of concurrent users, ramp up time, no. of test engines and test iterations and the test duration. Apache JMeter allows us to manually configure these values before the test is started. New EC2 instances have to provisioned for the Agents, the IP addresses (Usually Elastic IP) of the slaves/agents has to be manually added to the master. The entire setup has to be maintained, managed and monitored during the test cycles.  This is ok for an load testing environment with few load test agents and low RPS, imagine an environment where you have generate thousands of RPS and having 50+ agents running. This process of managing the EC2 load test infrastructure will become tedious process overall for the load testing teams. In BlazeMeter, once the number of concurrent users is given, the number of test engines, number of threads and engine capacity is chosen automatically. This can be made semi-automatic, where the number of engines & number of threads as well can be selected by user and only engine capacity is chosen by BlazeMeter. Since it is a managed Load Test infrastructure, the Load Testers can concentrate the testing and not managing 100's of EC2 load agents.

Point 6) Integrated Monitoring: 

• BlazeMeter offers live monitoring of essential parameters of test servers when the test is running which enables us to decide on the number & instance type for the test. In the conventional Apache JMeter load test setup in Amazon EC2 we have to observe the Key parameters using AWS Cloudwatch.
• Blaze Meter provides AWS Cloud watch integration.An account with IAM access has to be created and
• AWS Access Key & Secret Key values have to be configured so that the metrics are available in the Blazemeter’s dashboard. This features helps us to understand how the assets in the cloud are reacting to our load tests and help us accordingly tune the infrastructure.
• While performing load testing, it is important not only to monitor your Web Servers & Databases but also the agents from where the load is generated . The New Relic plugin gives us the front end KPIs and back end KPIs. 
• BlazeMeters’s frontend KPIs provide insight on how many users are actually trying to access your website, mobile site or mobile apps.
• BlazeMeters’s backend KPIs show how many users are getting through to your applications.

Point 7) Blazemeter allows us to have a different csv file per load test engine. Though this possible in Apache JMeter, it had to be done manually by copying the files onto the JMeter Agent EC2 instances and have the same filename since the agents refer to the Master’s properties. Blazemeter allows us to parameterize the values of even filenames and have different csv files in each engine without giving us to the trouble of copying files into specific EC2 instances & holds the files in a common repository so that it can be referred from there to each agent.

Point 8) Run the load test using older version of JMeter scripts: Old scripts can be reusable with this feature of BlazeMeter which lets us run the test using any version of Apache JMeter right from version 2.3.2 to 2.10. Some complex scripts prepared some months/years ago can be still be made usable and need not be redone. Saves efforts and costs.

Point 9) Schedule the Test & Stay Relaxed: BlazeMeter as well as JMeter lets you schedule your test duration & test time so that we can run longevity test at any time of the day. Even weekly scheduling is possible in BlazeMeter it is an added advantage, though it is not widely used.

Point 10) Interesting Plug-ins provided by Blazemeter :

• Integration with Google Analytics: At the time of scripting, it is enough if we select the Google Analytics Option & provide account details of Google Analytics. BlazeMeter obtains the last 12 months of data and creates a test with 5 most visited pages and sets up the number of concurrent users based on that record.
• Integration with WordPress: BlazeMeter provides integration with WordPress where WordPress users can test their App by using the BlazeMeter plug-in without any scripting.
• Integration with Drupal & Jenkins: Plugins are available to load test Drupal & Jenkins servers as well. 

Post Co Authored with Harine 8KMiles.

AWS Cost Saving Tip 16: Periodically remove your unwanted AWS resources and save costs

Thanks "@HighScalability.com"  for referencing this post "Stuff The Internet Says On Scalability For October 11th, 2013"

Following are some of the AWS assets that needs to be periodically reviewed for under utilization and should be removed as part of your IT process to avoid cost leakage in AWS. 

Remove unwanted AMI's: Periodically audit for unwanted AMI's in your dev/test/production environment and remove them. Tagging the AMI's with proper identifiers can help you during the cleansing process.

EBS snapshots are incremental in nature and only the changes are moved to them. The unwanted snapshots should be periodically identified (Tagging snapshots will help you) and deleted as part of your IT Infrastructure operations. EBS snapshots are priced $0.095 per GB-month of data stored and you can save few hundred $$$ depending upon the volume of unwanted snapshots stored.To know more about EBS Snapshots

EBS Volumes: In development and testing environments of large enterprises we can often find lots of EBS volumes provisioned and lying unattached or unused. EBS Volumes are priced at $0.10 per GB-month of provisioned storage. Usually 10-30 GB are the most used volume ranges for the dev/test. Imagine a enterprise having few hundred EC2's for their Dev/Test infrastructure and 10-25% of their storage are unused, Periodically identifying , tagging, consolidating and cleansing them will save few hundred to thousands of $$$ for the enterprises.To know more about EBS Volumes

S3 Buckets and Objects: Unused objects in S3 buckets have to be frequently deleted. Objects with a limited life period can be configured to expire dynamically using S3 object expiration policies. Objects that are rarely used can be archived to Amazon Glacier from S3. This exercise is an important candidate for Media and online companies using AWS. To know more about other S3 Cost saving tips 1 | 2.

Elastic IP is a scarce resource in AWS cloud, in event the EIP's attached to your account are not associated with any running EC2 instance, you will be charged $0.005 per hour on a prorata basis.
EIP's can usually be leaked in: 

• Dev/Test environments, where the developer disassociates the EIP and keeps it unused.
• In production, where sometimes EIP are assigned to Auto Scaled EC2 instances using scripts and they are not properly released back after EC2 instances are terminated by AWS.

Though if the EIP is kept unused can lead to around only ~$3.5 leakage per month, if no proper tracking procedures are in place many such EIP's can be residing in your account unused and lead to more such leakages.

ELB: Unused and Unwanted Elastic Load Balancers should be removed periodically to avoid cost leakage. A single ELB can cost around ~216 USD if kept unused a year. Usually ELB leakage is prone to happen in Dev/Test environments compared to production. To know more about Amazon ELB

Monitoring : Detailed monitoring option of CloudWatch is generally not required for Development EC2 instances. On the other hand, they are required on Load Testing and Production environments for monitoring the health. Custom Metrics are also good candidates for Production and Load Test environments. Constant review of the infrastructure in AWS has to be done to identify whether such resources(over provisioning) are associated with Dev/Test env and related cost leakages can be avoided.Detailed monitoring costs $3.50 per instance per month, provided at 1-minute frequency and Custom Metrics costs $0.50 per metric per month.

Glacier archives : Amazon Glacier is designed for use cases where data is retained for months, years, or decades. But there are times where you may find some data life has expired and they can be removed. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. Please ensure it is part of you IT process. To know more about other Cost savings in Glacier 1 | 2 | 3

Other small stuff :

• AWS charges $1.00 per month for inactive data pipelines. They should be periodically reviewed for leakages.  

Note:

• There are times where an AWS asset can be created in a non frequently used AWS Region mistakenly by a developer and it can be lying idle for months. Have an automated or manual IT process to periodically cleanse the above leakages points in "ALL" AWS Regions and not just your most frequently used regions.
• Many times on large AWS deployments/Dev/Test environments, it becomes difficult to track who created and who is currently using the AWS resources. AMI's, EBS snapshots, Volumes and S3 buckets should be tagged for proper identification. Identifiable names should be kept for other assets (wherever applicable) , so that it helps your IT team during the cleansing process. 
• Employ automation using scripts/programs or use governance tools ( which i will be discussing later) to monitor under utilization of resources and there by remove them.

Other Cost Saving Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

Part 1: Understanding Amazon ElastiCache Internals: Connection overheads

Amazon ElastiCache node (MemCached engine) uses a connection buffer per TCP connection to read/write data out over the network. This is not a problem when there are few concurrent connections to an Amazon ElastiCache Node, whereas when you get into hundreds of thousands of connections with hundreds of Amazon ElastiCache nodes, this adds up to connection memory overheads. Example: On each Amazon ElastiCache node, the memory made available for storing cache items is the total available memory on that cache node minus the memory used for connections and other overhead (like TCP connection buffer). This overhead value can be configured using the memcached_connections_overhead parameter in Amazon ElastiCache. For example, a cache node of type cache.m3.2xLarge has a max_cache_memory of 29600 MB. With the default memcached_connections_overhead value of 100 MB, the Memcached process will have 29500 MB available to store cache items. The default value for the memcached_connections_overhead parameter of 100 MB will satisfy most use cases; however, the required amount of allocation for connection overhead can vary depending on multiple factors, including request rate, payload size, and the number of connections. For cache heavy dependent site with high concurrency using multiple nodes of cache.m3.2xlarge instance, an overhead size of just 100 MB might not withstand sometimes and may cause swapping and degrade performance. That’s why Amazon ElastiCache has made this overhead a user configurable property. The configuration change will affect all cache nodes in the cluster. You need to monitor the swap, latency and number of concurrent requests using Amazon CloudWatch periodically and accordingly increase this parameter size (it could be few hundreds of MB increase to GB depending upon the usage readings).

Instead of wasting memory on Connection buffers and overheads it could be better if it is used to store user data. To reclaim this memory for user data some techniques that are employed in the industry:

·                     Implement a per-thread shared connection buffer pool for TCP and UDP sockets. This change can enable you to reclaim multiple gigabytes of memory per server. A patch was provided by Facebook engineering team in their engineering blog on per-thread shared connection buffer pool, we need to check with AWS ElastiCache team whether this patch is applied on or any equivalent tuning is done on Amazon ElastiCache Nodes for reducing this overhead.

·                     Use UDP wherever applicable and feasible. Currently UDP is not supported by AWS ElastiCache. In future it could be a possibility.

Exploring Amazon Availability Zones

Amazon Web Services (AWS) currently serves hundreds of thousands of customers in more than 190 countries. AWS is steadily expanding their global infrastructure to help customers achieve lower latency and higher throughput, and to ensure that customer data resides only in the Region they specify. AWS currently operates at 9 regions around the world and they are constantly expanding their infrastructure as I write this article. Following diagram illustrates their current regional infrastructure distribution:

Note: Image is not the latest

Each Amazon EC2 Region is designed to be completely isolated from the other Amazon EC2 Regions. This infrastructure design achieves the greatest possible failure independence and stability. Also by launching EC2 instances in separate Amazon Regions, we can design our application to be closer to specific customers or to meet legal/compliance or other requirements. 
Every Amazon Region is further sub divided into Availability Zones. By launching EC2 instances in separate Availability Zones (AZ), we can protect our applications from the failure of a single location.

So what is an Amazon Availability Zone?

Amazon operates state-of-the-art, highly available data center facilities. However, failures can occur that affect the availability of EC2 instances that are in the same location. Although this is rare, if you host all your Amazon EC2 instances in a single location that is affected by such a failure, your instances will be unavailable.
So to overcome this every Amazon Region is further sub divided into Availability zones. Amazon Availability Zones are distinct physical locations having Low latency network connectivity between them inside the same region and are engineered to be insulated from failures from other AZ’s. Each availability zone runs on its own physically distinct, independent infrastructure, and is engineered to be highly reliable; they have Independent power, cooling, network and security. Common points of failures like generators and cooling equipment are not shared across Availability Zones. Additionally, they are physically separate; such that even extremely uncommon disasters such as fires, tornados or flooding would only affect a single Availability Zone. The following diagram illustrates availability zone concept: (image Source : AWS)

We can visualize each AZ internally having Single Data Center or Multiple Data centers. The following diagram illustrates the current AZ allocation of AWS infrastructure:(image Source : AWS)

Note: Image is not the latest

• ·         Region :US-EAST( North Virginia) has 5 availability zones
• ·         Region :US-West (California + Oregon) has 6 availability zones
• ·         Region :Europe – West (Dublin) has 3 availability zones
• ·         Region :Asia Pacific (Japan) has 2 availability zones
• ·         Region :Asia Pacific (Singapore) has 2 availability zones
• ·         Region :South America (Sao Paulo) has 2 availability zones 
•       Region :Australia (Sydney) has 2 availability zones
• ·         Region : Gov Cloud

Why we should leverage AWS Availability Zones?

Just moving a system into the cloud doesn’t make it fault-tolerant or highly available; we need to understand the features provided by the cloud provider for availability in depth and then architect our application leveraging the same. 
In AWS infrastructure it is usually recommended to architect applications using multiple availability zones inside a region as best practice. When you design such a system, it is suggested to have a good prior understanding of zone dependencies.
Availability zones (AZs) are distinct geographical locations that are engineered to be insulated from failures in other AZs. Certain services that provide basic infrastructure, such as Amazon Elastic Compute Cloud (EC2) and Amazon Elastic Block Store (EBS) need to be properly architected using this Amazon Multi-AZ’s concept. By placing Amazon EC2 instances in multiple AZs, an application can be protected from failure at a Single EC2, Single location or Single Data center level. It is important to run independent application stacks in more than one AZ, so that if one zone fails, the application in the other zone can continue to run without disruption.
Usually applications which are properly architected for multi-tiered distribution and statelessness in mind can easily leverage AWS High availability features without any code change. Applications which are Statefull can still leverage Amazon Multi-AZ deployment, but it will not be completely fault tolerant because of it is design. Some of the techniques followed in AWS Multi-AZ deployments to avoid Single point of Failure (SPOF) in web/app layer are:

• Since AWS infrastructure does not support Multicast protocol currently, the application layer software should synchronize data using Unicast TCP mechanism. Example: Java based servers can use JGroups or Terracotta NAM to synchronize the cluster information in AWS.
• In case the Web/App servers are written on PHP, .Net, Python etc then all the user and session data can be stored on centralized systems like MemCached EC2 instance or ElastiCache nodes or Amazon DynamoDB. Note: Deploy redundant ElastiCache Clusters in different Availability Zones for HA (or) build cache warming in your design for ElastiCache/MemCached Layers.
• Uploaded User files and documents should be centrally stored on NFS Pool or RedHat Gluster Storage Pool or Amazon S3

The above mentioned points are some of the simple steps that needs to be remembered during the Architecture or Migration phase in AWS.

How AWS Building blocks inherently leverage Availability Zones?

Most of the higher-level AWS, such as Amazon Simple Storage Service (S3), Amazon DynamoDB, Amazon CloudWatch, Amazon Simple Queue Service (SQS), and Amazon Elastic Load Balancing (ELB), have been built with inherent fault tolerance and high availability in mind. They already work across multiple availability zones inside an Amazon Region. Let us see how each of them uses the Availability zone concept;

Amazon ELB:  Amazon ELB can direct and load balance requests to multiple EC2 instances launched across multiple availability zones inside a Region. Load Balancer Instances are spawned in Multiple Availability Zones in the ELB tier as well internally when the traffic increases. 
Amazon RDS MySQL:  Amazon RDS for MySQL and Oracle currently supports Multi-AZ Hot Standby architecture, where the Primary Master will run on one Availability zone and Hot Standby (Secondary) will run on another Availability zone. In event of Primary AZ failure, the Hot Standby will start serving the requests from the alternative AZ. We can also create and launch multiple RDS Read Replica’s in Multiple Availability Zones for Read scaling and HA.
Amazon S3: Any data that is stored on S3 (Standard Storage option) is synchronized at multiple locations/facilities inside an Amazon Region. Amazon S3 Objects are redundantly stored on multiple devices across multiple facilities (AZ’s) in an Amazon S3 Region. To help ensure durability, Amazon S3 PUT and COPY operations synchronously store our data across multiple facilities before returning a successful response.
Amazon DynamoDB: Amazon DynamoDB service replicates data across three facilities/Availability zones in an AWS Region to provide fault tolerance in the event of a server failure or Availability Zone outage. Also to achieve high uptime and durability, Amazon DynamoDB does synchronous replication of data across these (multiple) facilities.
Amazon SQS: Amazon SQS stores all queue and messages redundantly on multiple servers and in multiple data centers/availability zones, which means that no single computer or network failure will render SQS messages inaccessible.
Amazon ElastiCache: Currently Amazon ElastiCache Cluster cannot span multiple Availability Zones inside an AWS Region. Deploy redundant ElastiCache Clusters in different Availability Zones for HA in AWS or alternatively use efficient cache warming techniques in your architecture for fault tolerance in this layer.
Amazon CloudWatch: Amazon CloudWatch can monitor EC2 instances which are deployed at multiple availability zones inside an Amazon Region. Amazon CloudWatch is an AWS building block and it is a Highly Available service by design. 
Amazon AutoScaling: Amazon AutoScaling can scale out/down EC2 instances across multiple Availability zones inside an Amazon Region. Amazon AutoScaling cannot span EC2 scale out across Amazon Regions.

Multiple Availability Zones inside Amazon VPC

Using Amazon Virtual Private Cloud (Amazon VPC) we can provision a private, isolated section of the Amazon Web Services (AWS) Cloud where we can launch AWS resources in a virtual network that we define. We have seen enterprises mostly prefer VPC model of deployment in Amazon cloud.
Inside Amazon VPC, we can define a virtual network topology that closely resembles a traditional network with complete control over our virtual networking environment, including selection of IP address range, creation of subnets, and configuration of route tables and network gateways. Since Availability zone concept works inside VPC, for deploying highly available applications inside VPC it is recommended to run it in multiple Availability Zones.
Create multiple subnets inside a VPC and put each subnet in a distinct Availability Zone for High Availability. Currently a single VPC can span multiple Availability Zones and Multiple VPN connections, but we cannot create subnets inside Amazon VPC that spans multiple availability zones. Amazon VPC is available in all the regions of AWS infrastructure.
VPN Gateways are regional objects, and can be accessed from any of the subnets (subject, of course, to any Network ACLs that you create); Sample architecture using VPC-Multi-AZ is illustrated below:

Let me detail the above diagram:
Point 1) Amazon VPC is created in US-East Region of AWS.
Point 2) Multiple subnets are created inside a VPC and each subnet is put in a distinct Amazon Availability Zone for High Availability. Example: Have your Web, App and DB layer distributed in public/private subnets inside availability zone 1a and keep a similar set in availability zone 1b as well for HA. Since we cannot create subnets inside Amazon VPC that spans multiple availability zones, we need to achieve HA using the above mentioned subnet-AZ network architecture.
Point 3) Multiple VPN connections from the single VPC are attached to multiple customer gateways located in multiple geographies (simulating "branch office" architecture).

Availability Zone Names are logical 

Availability Zones are not the same across AWS accounts. There is a common misconception that an AZ name like "US-east-1a" identifies a specific physical availability zone for everyone.  The fact is that AWS can map/remap the same AZ name to different physical availability zones across multiple accounts. The Availability Zone us-east-1a for account A is not necessarily the same as us-east-1a for account B. Zone assignments are mapped independently for each account. This is important when our infrastructure or use cases spans across multiple accounts. Example: Infrastructure provisioned through Account-A and Load Testing Agents are launched through Account-B, and both pointing to "US-east-1a" may not map to same AZ.

Guidelines for architecting applications across AWS Availability Zones

A typical Web-Application Stack consists of following tiers; DNS tier, Load Balancers, Web Tier, App Tier, Cache, Database and Storage layer. It is important to run all independent application stacks in more than one AZ, so that if one zone fails, the application in the other zone can continue to run without disruption in AWS infrastructure. All the above mentioned tiers can be distributed and deployed to run on at least 2 or more availability zones inside an AWS region. In our experience we have seen the following set of tiers and software’s can be deployed across multiple availability zones with minimal configuration changes:

Sample Web/App Architecture using Multiple Availability zones

Following diagram illustrates a reference architecture using Multiple Availability zones of Amazon. Let us see what the various tiers in this architecture are and how they are using AZ concept.

DNS tier: Route53 
Load balancing Tier: Amazon ELB inbuilt with Multi-AZ
Web /App Tier: EC2 instances launched across Multiple Availability zones using Amazon AutoScaling. Integrated with Amazon ELB and CloudWatch 
Database Tier: RDS MySQL with Hot Standby, Multiple Read Replica’s in multiple AZ’s depending upon the Read Scaling needed 
Caching Tier: Amazon ElastiCache cluster or EC2 MemCached distributed in Multiple –AZ
Search Tier: Since Amazon CloudSearch currently support only single AZ; users can look at deploying Solr replication/Replicated shards across multiple Availability zones
NoSQL Tier: Amazon DynamoDB inherently replicates data in Multiple AZ’s/Facilities
CDN Tier: Amazon CloudFront internally uses global network of edge locations
Monitoring Tier: Amazon CloudWatch is used for monitoring the infrastructure. EC2 instances use detailed monitoring.

AWS Availability Zones: Usage Charges

We are charged a small bandwidth charge (Regional Data Transfer) for data that crosses Availability Zones. Regional Data Transfer rates apply if at least one of the following is true, but is only charged once for a given instance even if both are true:

·   The other instance is in a different availability zone, regardless of which type of address is used.

• Data transferred between instances in the same availability zone on EC2 costs $0.00 per GB.
• Data transferred between instances across different availability zones on EC2 costs $0.01 per GB.

·   Public or Elastic IP addresses are used, regardless of which zone the other instance is in.

Also Data transfer between VPC and non-VPC instances in the same Region, regardless of Availability Zone, is charged at the usual rate of $0.01 per Gigabyte

AWS Availability Zones: Simple latency test

We did a simple test to check the latency between Amazon Availability zones 1a and 1b (our account). 
The Test architecture had:

• Multiple Grinder Load Clients
• Tomcat Application Server and 1-RDS MySQL Database Server
• M1.large instance type was used for the Tomcat and RDS MySQL
• 100,000 txns generated in 60 seconds 
• Region: US-East , Availability Zones : 1A and 1B in our account
• In Practice Majority of Web/App EC2 and Master Database should reside in same AZ. For ease of testing purpose we have followed the above architecture.

Test 1: Observations

·         Java/Tomcat based App EC2 was launched on US-EAST-1A

·         RDS MySQL was launched on US-EAST-1A ( same availability zone)

·         Both were EC2 instances were m1.large capacity

·         100K txns/minute was generated from Tomcat to RDS instance

·         Average latency of 3 ms was observed for 100K txns in a single AZ setup

Test 2: Observations

·         Java/Tomcat based App EC2 was launched on US-EAST-1A

·         RDS MySQL was launched on US-EAST-1B (different availability zone)

·         Both were EC2 instances were m1.large capacity

·         100K txns/minute was generated from Tomcat to RDS instance

·         Average latency of 9 ms was observed for 100K txns in the multi-AZ setup.

·        Though it is roughly ~3X increase of latency from Test1, since the Average latency is in the range of single digit ms, it is suggested to use Multi-AZ deployments for applications requiring high availability.  Note: The observed ~3X latency difference will usually vary depending upon the Availability Zones (AZ origin / AZ destination /AWS Region) and it should not be treated as exact figure. 

Dissecting Amazon ELB : 18 things you should know

While designing highly scalable systems load balancing tier becomes an integral part of any architecture. We have captured some of our prior experiences working with Amazon ELB in this article as points detailed below. Some of the points mentioned here will be encountered only by advanced users in complex use cases. But surely if you/your team have noted some of these points, I feel it might shorten your efforts while debugging a problem or designing a solution and not go through the same effort cycle and pain as our team.

In AWS, there are wide variety of solution choices for the Load balancing layer like Amazon Elastic Load Balancing (ELB) , EC2 AMI’s like HAProxy , Nginx , Zeus , Citrix NetScaler. In this article we are going to dissect our experience with Amazon ELB layer as X points which you will not frequently encounter in Amazon documents or blogosphere. 

To know more about Configuring Amazon ELB in 4 Easy Steps, Refer article:   http://harish11g.blogspot.in/2012/02/configuring-aws-elb-console.html

Currently there are 18 points in this article and i am having plans to add some more in coming days . So if you are an advanced user of Amazon ELB , please watch this article closely. 

Some of the points are:

Point 1) Algorithms supported by Amazon ELB

Currently Amazon ELB only supports Round Robin(RR) and Session Sticky Algorithms.

Round Robin algorithm can be used for load balancing traffic between

•  Web/App EC2 instances which are designed stateless
•  Web/App EC2 instances which synchronizes the state between them
•  Web/App EC2 instances which synchronizes the state using common data stores like MemCached , ElastiCache , Database etc.

Session Sticky algorithm can be used for load balancing traffic between

• Web/App EC2 instances which are designed to be statefull

Current Version of ELB does not support Weighted or Least Connection algorithms like other Reverse proxies. We can probably expect these algorithms to be supported in future.

Point 2) Amazon ELB is not a PAGE CACHE

Amazon ELB is just a load balancer and not to be confused with Page Cache Server or Web Accelerator.  Web Accelerators like Varnish can cache pages, Static assets etc and also do RR load balancing to backend EC2 servers. Amazon ELB is designed to do just Load balancing efficiently and elastically. If you need page accelerators + LB you can use Varnish or NetScaler in your LB Tier. Refer Article Varnish or NetScaler. Amazon ELB can be used with Amazon CloudFront to deliver the static assets and dynamic assets that can be page cached at edge location itself to reduce latency for above use cases.

Point 3) Amazon ELB can be pre warmed on request basis

Amazon ELB can be pre warmed by raising a request to Amazon Web Service Support Team. Amazon team will pre warm the Load Balancers in the ELB tier to handle the sudden load/flash traffic. This is advisable for scenarios like Quarterly sales/launch campaigns, promotions etc which follow flash traffic pattern. AWS team would require details like estimated Request per second, average request size in bytes, average response size in bytes, what percentage of traffic is SSL/ Non SSL, whether HTTP/1.1 keep alive is enabled ? etc from your team. Once provided, it will be activated by them. Amazon ELB pre warm cannot be done on hourly/daily basis (i think). It will be a cool feature if Amazon team can get these details and offer ELB Pre warming as a configurable feature into the AWS console (like Amazon DynamoDB console)

Point 4) Amazon ELB is not designed for sudden load spikes /Flash traffic

Amazon ELB is designed to handle unlimited concurrent requests per second with “gradually increasing” load pattern.  It is not designed to handle heavy sudden spike of load or flash traffic. For example: Imagine an e-commerce website whose traffic increases gradually to thousands of concurrent requests/sec in hours, Amazon ELB can easily handle this traffic pattern. According to RightScale benchmark, Amazon ELB was easily able to handle 20K+ requests/sec and more in such patterns. Whereas imagine use cases like Mass Online Exam or GILT load pattern or 3-Hrs Sales/launch campaign sites expecting 20K+ concurrent requests/sec spike suddenly in few minutes, Amazon ELB will struggle to handle this load pattern. If this sudden spike pattern is not a frequent occurrence then we can pre warm ELB, else we need to look for alternative Load balancers in AWS infrastructure. 

Comparison analysis of HAProxy vs Amazon ELB, Refer article:   http://harish11g.blogspot.in/2012/11/amazon-elb-vs-haproxy-ec2-analysis.html

Point 5) Protocols supported by Amazon ELB

Currently Amazon ELB only supports following protocols: HTTP, HTTPS (Secure HTTP), SSL (Secure TCP) and TCP protocols. ELB supports load balancing for the following TCP ports: 25, 80, 443, and 1024-65535. In case RTMP or HTTP Streaming protocol is needed, we need to use Amazon CloudFront CDN in your architecture.

Point 6) Amazon ELB timeouts at 60 seconds (kept idle)

Amazon ELB currently timeouts persistent socket connections @ 60 seconds if it is kept idle. This condition will be a problem for use cases which generates large files (PDF, reports etc) at backend EC2, sends them as response back and keeps connection idle during entire generation process. To avoid this you'll have to send something on the socket every 40 or so seconds to keep the connection active in Amazon ELB. Note: I heard we can extend this value after explaining the case to AWS support team.

Point 7) Amazon ELB does not provide Permanent or Fixed IP for its load Balancers

Currently Amazon ELB does not provide fixed or permanent IP address for the Load balancing instances that are launched in its tier. This will be a bottleneck for enterprises which have compulsion to whitelist their Load balancer IP’s in external firewalls/gateways. For such use cases, currently we can use HAProxy, NginX, NetScaler over EC2 attached with Elastic IPs as load balancers in AWS infrastructure.

Designing High Availability @ HAProxy / ELB Layer http://harish11g.blogspot.in/2012/10/high-availability-haproxy-amazon-ec2.html

Point 8) Amazon ELB cannot do Multi AWS Region Load Balancing

Amazon ELB can be used to Load balance

• Multiple EC2 instances launched inside a Single Amazon Availability Zone
• Multiple EC2 instances launched inside Multiple Availability Zones inside a Single Region

Amazon ELB cannot load Balance between EC2 instances launched on Multiple AWS regions . Use Route53 DNS RR / LBR / Failover configurations for Load balancing at DNS level between ELB, EC2 etc launched at multiple AWS Regions

To know more about DNS Load Balancing : http://harish11g.blogspot.in/2012/06/aws-high-availability-dns-load.html To know more about Geo Distributed Load Balancing using Amazon Route 53 : http://harish11g.blogspot.in/2012/09/geo-distributed-route53-lbr-latency.html

Point 9) Amazon ELB sticks request when traffic is generated from Single IP

This point comes as a surprise to many users using Amazon ELB. Amazon ELB behaves little strange when incoming traffic is originated from Single or  Specific IP ranges, it does not efficiently do round robin and sticks the request.  Amazon ELB starts favoring a single EC2 or EC2’s in Single Availability zones alone in Multi-AZ deployments during such conditions. For example: If you have application A(customer company) and Application B, and Application B is deployed inside AWS infrastructure with ELB front end. All the traffic generated from Application A(single host) is sent to Application B in AWS, in this case ELB of Application B will not efficiently Round Robin the traffic to Web/App EC2 instances deployed under it. This is because the entire incoming traffic from application A will be from a Single Firewall/ NAT or Specific IP range servers and ELB will start unevenly sticking the requests to Single EC2 or EC2’s in Single AZ.

Note: Users encounter this usually during load test, so it is ideal to load test AWS Infra from multiple distributed agents. 

Point 10)Too long Load Balancer CNAMES causes issues in some firewalls /ISP

Some ISP's do not allow Amazon ELB CNAMES that exceeds 32 characters and some firewalls versions/models (like Cisco PIX) will not allow larger CNAMES , in such cases try to have shorter name.

Point 11) Amazon ELB cannot Load Balance based on URL patterns

Amazon ELB cannot Load Balance based on URL patterns like other Reverse proxies. Example Amazon ELB cannot direct and load balance between request URLs  www.xyz.com/URL1 and www.xyz.com/URL2. Currently for such use cases you can use HAProxy in EC2.

Point 12) Amazon ELB can easily support more than 20K+ Concurrent reqs/sec

Amazon ELB is designed to handle unlimited concurrent requests per second. ELB is inherently scalable and it can elastically increase /decrease its capacity depending upon the traffic. According to a benchmark done by RightScale, Amazon ELB was easily able to scale out and handle 20K or more concurrent requests /sec. Refer URL: http://blog.rightscale.com/2010/04/01/benchmarking-load-balancers-in-the-cloud/

Point 13) Amazon ELB does not provide logs

Amazon ELB currently does not provide access to its log files for analysis. We cannot debug load balancing problems , analyze the traffic and access patterns; categorize bots / visitors etc currently because we do not have access to the ELB logs.This will also be a bottleneck for some organizations which has strong audit/compliance requirements to be met at all layers of their infrastructure. Amazon ELB can generate the logs and put in Amazon S3 buckets– (feature request to Amazon ELB product team)

Point 14) Monitoring Amazon ELB

Amazon ELB is an AWS building block and it does not currently provide access to its logs or Stats files for monitoring. Secondly, we cannot get full access to the Load Balancers launched inside the ELB tier and install any monitoring agents in it. This closed model of ELB makes us rely only on CloudWatch metrics for monitoring. Refer this URL for ELB metrics that can be currently monitored: http://harish11g.blogspot.in/2012/02/cloudwatch-elastic-load-balancing.html

Point 15) Amazon ELB and Compliance requirements

SSL Termination can be done at 2 levels using Amazon ELB in your application architecture .They are

• SSL termination can be done at Amazon ELB Tier, which means connection is encrypted between Client(browser etc) and Amazon ELB, but connection between ELB and Web/App EC2 is clear. This configuration may not be acceptable in strictly secure environments and will not pass through compliance requirements.
• SSL termination can be done at Backend with End to End encryption, which means connection is encrypted between Client and Amazon ELB, and connection between ELB and Web/App EC2 backed is also encrypted. This is the recommended ELB configuration for meeting the compliance requirements at LB level. 
• Important ELB-SSL Reference URLs 
• Refer this URL to understand how to configure SSL Offloading in Amazon ELB : http://harish11g.blogspot.in/2012/03/ssl-offloading-elastic-load-balancing.html
• How Amazon ELB-SSL support/offloading helps Cloud Admins ? :http://harish11g.blogspot.in/2010/10/amazon-elastic-load-balancing-support.html

Point 16) Amazon ELB and X.X.X.255 IP address

Sometimes ELB assigns its load balancers with IP address ending with X.X.X.255. Though it is technically fine, there are certain networks that will not properly route to an IP address ending in X.X.255 series. Unfortunately, it is not possible to exclude an IP address ending in .255 from ELB currently. It is possible, in such circumstances, some requests from certain users may face issues. Note this when you are debugging ELB for missing requests. 

Point 17) Amazon ELB inherently fault tolerant and Scalable service

Elastic Load Balancer does not cap the number of connections that it can attempt to establish with the load balanced Amazon EC2 instances. We can expect this number to scale with the number of concurrent HTTP, HTTPS, or SSL requests or the number of concurrent TCP connections that the Elastic Load Balancer receives. Since multiple Load balancers are launched in ELB tier, it is inherently fault tolerant as well. If you need a Scalable and and elastic LB layer , then ELB comes highly recommended. Amazon ELB can be deployed to support following HA architectures in AWS : http://harish11g.blogspot.in/2012/02/elastic-load-balancing-aws-deployment.html

Point 18) Amazon ELB + Amazon AutoScaling : No graceful connection termination

Amazon ELB can be configured with work seamlessly with Amazon AutoScaling and Amazon CloudWatch. The New EC2 instances launched by AutoScaling are added to the ELB for Load balancing automatically and whenever load drops; existing EC2 instances can be removed by Auto Scaling from ELB. Both Auto Scaling and ELB use CloudWatch Monitoring for enabling this functionality. The important point to remember while using this kind of integration is Amazon AutoScaling does not gracefully (without interruption to existing connections) remove Web/App EC2 from Amazon ELB. The connections are instantly dropped when the Web/App EC2 is removed and no grace period is given by ELB or AutoScaling. This behavior of Auto scaling can make dozens or hundreds of users to get error pages when they are using the application when such an event occurs in the backend infrastructure.

To know more about Amazon Auto Scaling : http://harish11g.blogspot.in/2011/04/auto-scaling-in-aws-cloud.html Harish Presentation @ AWS Summit on Amazon Auto Scaling http://harish11g.blogspot.in/2011/11/scale-new-business-peaks-with-amazon.html How Auto Scaling can save costs ? http://harish11g.blogspot.in/2013/04/Amazon-Web-Services-AWS-Cost-Saving-Tips-how-Amazon-AutoScaling-can-reduce-leakage-save-costs.html

Article under progress

Other Load Balancing Articles

Monitoring ELB with Amazon CloudWatch

Configuring Amazon ELB using AWS Management Console

Amazon ELB Implementation Architectures

Amazon Elastic Load balancing benefits

Why do we need Elastic Load Balancing ?

Offloading SSL in Amazon ELB