25 Best Practice Tips for architecting your Amazon VPC

According to me Amazon VPC is one of the most important feature introduced by AWS. We have been using AWS from 2008 and Amazon VPC from the day it was introduced and i strongly feel the customer adoption towards AWS cloud gained real momentum only after the introduction of VPC into the market. 

Amazon VPC comes with lots of advantages over the limitations faced in Amazon Classic cloud like: Static private IP address , Elastic Network Interfaces :  possible to bind multiple Elastic Network Interfaces to a single instance, Internal Elastic Load Balancers, Advanced Network Access Control ,Setup a secure bastion host , DHCP options , Predictable internal IP ranges , Moving NICs and internal IPs between instances, VPN connectivity, Heightened security etc. Each and everything is a interesting topic on its own and i will be discussing them in detail in future. 

Today i am sharing some of our implementation experience on working with hundreds of Amazon VPC deployments as best practice tips for the AWS user community. You can apply some of the relevant ones in your existing VPC or use these points as part of your migration approach to Amazon VPC.

Please also look into 27 Best practices on AWS Security Groups to understand the relation with Amazon VPC. 
http://harish11g.blogspot.com/2015/06/best-practices-tips-on-amazon-web-services-security-groups-aws-security-managed-services.html

Practice 1) Get your Amazon VPC combination right: Select the right Amazon VPC architecture first.  You need to decide the right Amazon VPC & VPN setup combination based on your current and future requirements. It is tough to modify/re-design the Amazon VPC at later stage, so it is better to design it taking into consideration your NW and expansion needs for next ~2 years. Currently different types of Amazon VPC setups are available; Like Public facing VPC, Public and Private setup VPC, Amazon VPC with Public and Private Subnets and Hardware VPN Access, Amazon VPC with Private Subnets and Hardware VPN Access, Software based VPN access etc. Choose the one which you feel you will be in next 1-2 years.

Practice 2) Choose your CIDR Blocks: While designing your Amazon VPC, the CIDR block should be chosen in consideration with the number of IP addresses needed and whether we are going to establish connectivity with our data center. The allowed block size is between a /28 netmask and /16 netmask. Amazon VPC can have contain from 16 to 65536 IP addresses. Currently Amazon VPC once created can't be modified, so it is best to choose the CIDR block which has more IP addresses usually. Also when you design the Amazon VPC architecture to communicate with the on premise/data center ensure your CIDR range used in Amazon VPC does not overlaps or conflicts with the CIDR blocks in your On premise/Data center. Note: If you are using same CIDR blocks while configuring the customer gateway it may conflict.
E.g., Your VPC CIDR block is 10.0.0.0/16 and if you have 10.0.25.0/24 subnet in a data center the communication from instances in VPC to data center will not happen since the subnet is the part of the VPC CIDR. In order to avoid these consequences it is good to have the IP ranges in different class. Example., Amazon VPC is in 10.0.0.0/16 and data center is in 172.16.0.0/24 series.

Practice 3) Isolate according to your Use case: Create separate Amazon VPC for Development , Staging and Production environment (or) Create one Amazon VPC with Separate Subnets/Security/isolated NW groups for Production , Staging and development. We have observed 60% of the customer preferring the second choice. You chose the right one according to your use case. 

Practice 4) Securing Amazon VPC : If you are running a machine critical workload demanding complex security needs you can secure the Amazon VPC like your on-premise data center or more sometimes. Some of the tips to secure your VPC are: 

• Secure your Amazon VPC using Firewall virtual appliance, Web application firewall available from Amazon Web Services Marketplace. You can use check point, Sophos etc for this
• You can configure Intrusion Prevention or Intrusion Detection virtual appliances and secure the protocols and take preventive/corrective actions in your VPC
• Configure VM encryption tools which encrypts your root and additional EBS volumes. The Key can be stored inside AWS (or) in your Data center outside Amazon Web Services depending on your compliance needs. http://harish11g.blogspot.in/2013/04/understanding-Amazon-Elastic-Block-Store-Securing-EBS-TrendMicro-SecureCloud.html
• Configure Privileged Identity access management solutions on your Amazon VPC to monitor and audit the access of Administrators of your VPC.
• Enable the cloud trail to audit in the VPC environments  ACL policy's. Enable cloud trail : http://harish11g.blogspot.in/2014/01/Integrating-AWS-CloudTrail-with-Splunk-for-managed-services-monitoring-audit-compliance.html
• Apply anti virus for cleansing specific EC2 instances inside VPC. Trend micro has very good product for this.
• Configure Site to Site VPN for securely transferring information between Amazon VPC in different regions or between Amazon VPC to your On premise Data center
• Follow the Security Groups and NW ACL's best practices listed below

Practice 5) Understand Amazon VPC Limits: Always design the VPC subnets in consideration with the expansion in the future. Also understand the Amazon VPC's limits before using the same. AWS has various limitations on the VPC components like Rules per security group, No of route tables and Subnets etc. Some of them may be increased after providing the request to the Amazon support team while few components cannot be increased. Ensure the limitations are not affecting your overall design. Refer URL:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html

Practice 6) IAM your Amazon VPC: When you are going to assign people to maintain your Amazon VPC you can create Amazon IAM account with the fine grained permissions (or) use Sophisticated Privileged identity Management solutions available on AWS marketplace to IAM your VPC.

Practice 7) Disaster Recovery or Geo Distributed Amazon VPC Setup : When you are designing a Disaster Recovery Setup plan using VPC or expanding to another Amazon VPC region you can follow these simple rules. Create your Production site VPC CIDR : 10.0.0.0/16 and your DR region VPC CIDR:  172.16.0.0/16. Make sure they do not conflict with on premises subnet CIDR block in event both needs to be integrated to on premise DC as well. After CIDR blocks creation , setup a VPC tunnel between regions and to your on premise DC. This will help to replicate your data using private IP’s. 

Practice 8) Use security groups and Network ACLs wisely:  It is advisable to use security groups over Network ACLs inside Amazon VPC wherever applicable for better control. Security groups are applicable on EC2 instance level while network ACL is applicable on Subnet level.  Security groups are used for White list mostly. To blacklist IPs, one can use Network ACLs.

Practice 9) Tier your Security Groups : Create different security groups for different tiers of your infrastructure architecture inside your VPC. If you have Web, App, DB tiers create different security group for each of them. Creating tier wise security groups will increase the infrastructure security inside Amazon VPC.  EC2 instances in each tier can talk only on application specified ports and not at all ports. If you create Amazon VPC security groups for each and every tier/service separately it will be easier to open a port to a particular service. Don't use same security group for multiple tiers of instances, this is a bad practice. 

Example: Open ports for security group instead of IP ranges : For example : People have tendency to open for port 8080 to 10.10.0.0/24 (web layer) range. Instead of that, open port 8080 to web-security-group. This will make sure only web security group instances will be able to contact on port 8080. If someone launches NAT instance with NAT-Security-Group in 10.10.0.0/24, he won't be able to contact on port 8080 as it allows access from only web security group. 

Practice 10 ) Standardize your Security Group Naming conventions : Following a security group naming conventions inside Amazon VPC will improve operations/management for large scale deployments inside VPC. It also avoids manual errors, leaks and saves cost and time overall.

For example: Simple ones like Prod_DMZ_Web_SG or Dev_MGMT_Utility_SG (or) complex coded ones for large scale deployments like

USVA5LXWEBP001- US East Virginia AZ 5 Linux Web Server Production 001

This helps in better management of security groups. 

Practice 11) ELB on Amazon VPC:  When using Amazon ELB for Web Applications, put all other EC2 instances( Tiers like App,cache,DB,BG etc)  in private subnets as much possible. Unless there is a specific requirement where instances need outside world access and EIP attached, put all instances in private subnet only. Only ELBs should be provisioned in Public Subnet as secure practice in Amazon VPC environment.

Practice 12) Control your outgoing traffic in Amazon VPC: If you are looking for better security, for the traffic going to internet gateway use Software's like Squid or Sophos to restrict the ports,URL,Domains etc so that all traffic go through the proxy tier controlled and it also gets logged. Using these proxy/security systems we can also restrict the unwanted ports, by doing so,  if there is any security compromise to the application running inside Amazon VPC they can be detected by auditing the restricted connections captured from the logs. This helps in corrective security measure. 

Practice 13) Plan your NAT Instance Type: Whenever your Application EC2 instances residing inside private subnet of Amazon VPC are making Web Service/HTTP/S3/SQS calls they go through NAT instance. If you have designed Auto scaling for your application tier and there are chances ten's of app EC2 instances are going to make lots of web calls concurrently, NAT instance will become a performance bottleneck at this juncture. Size your NAT instance capacity depending upon application needs for avoiding performance bottlenecks. Using the NAT instances provides us with advantages of saving cost of Elastic IP and provides extra security by not exposing the instances to outside world for accessing the internet. 

Practice 14) Spread your NAT instance with Multiple Subnets: What if you have hundreds of EC2 instances inside your Amazon VPC and they are making lots of heavy web service/HTTP calls concurrently. A single NAT instance with even largest EC2 size cannot handle that bandwidth sometimes and may become performance bottleneck. In Such scenarios, span your EC2 across multiple subnets and create NAT's for each subnet. This way you can spread your out going bandwidth and improve the performance in your VPC based deployments.

Practice 15) Use EIP when needed: At times you may need to keep a part of your application services to be kept in Public subnet for external communication. It is recommended practice to associate them with Amazon Elastic IP and white list these IP address in the target services used by them 

Practice 16) NAT instance practices : If needed, enable Multi factor authentication on NAT instance. SSH and RDP ports are open only on sources and destination IP's, not global network (0.0.0.0/0). SSH / RDP ports are opened only on static exit IP's not dynamic exit IP's.  

Multi factor authentication ref link : Linux: http://www.howtogeek.com/121650/how-to-secure-ssh-with-google-authenticators-two-factor-authentication/ 

Windows : http://www.rohos.com/2013/02/google-authenticator-windows-login/

Practice 17) Plan your Tunnel between On-Premise DC to Amazon VPC: 

Select the right mechanism to connect your on premises DC to Amazon VPC. This will help you to connect the EC2 instance via private IP's in a secure manner. 

• Option 1: Secure IPSec tunnel to connect a corporate network with Amazon VPC (http://aws.amazon.com/articles/8800869755706543) 
• Option 2 : Secure communication between sites using the AWS VPN CloudHub (http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPN_CloudHub.html)
• Option 3: Use Direct connect between Amazon VPC and on premise when you have lots of data to be transferred with reduced latency (or) you have spread your mission critical workloads across cloud and on premise. Example: Oracle RAC in your DC and Web/App tier in your Amazon VPC. Contact us if you need help on setting up direct connect between Amazon VPC and DC.

Practice 18) Always span your Amazon VPC across multiple subnets in Multiple Availability zones inside a Region. This helps is architecting high availability inside your Amazon VPC properly. Example: Classification of the VPC subnet : WEB Tier Subnet : 10.0.10.0/24 in Az1 and 10.0.11.0/24 in Az2, Application Tier Subnet :  10.0.12.0/24 and 10.0.13.0/24, DB Tier Subnet :  10.0.14.0/24 and 10.0.15.0/24, Cache Tier Subnet : 10.0.16.0/24 and 10.0.17.0/24 etc

Practice 19) Good security practice is that to have only public subnet with route table which carries route to internet gateway. Apply this wherever applicable.

Practice 20) Keep your Data closer : For small scale deployments in AWS where cost is critical than high availability, It is better to keep the Web/App in same availability zone as of ElastiCache , RDS etc inside your Amazon VPC. Design your subnets accordingly to suit this. This is not a recommended architecture for applications demanding High Availability.

Practice 21) Allow and Deny Network ACL : Create Internet outbound allow and deny network ACL in your VPC. 

First network ACL: Allow all the HTTP and HTTPS outbound traffic on public internet facing subnet. 

Second network ACL: Deny all the HTTP/HTTPS traffic. Allow all the traffic to Squid proxy server or any virtual appliance. 

http://techlib.barracuda.com/display/BNGv54/How+to+Deploy+the+Barracuda+NG+Firewall+in+an+Amazon+Virtual+Private+Cloud 

Practice 22 ) Restricting Network ACL : Block all the inbound and outbound ports. Only allow application request ports. These are stateless traffic filters that apply to all traffic inbound or outbound from a Subnet within VPC. AWS recommended Outbound rules : http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html

Practice 23) Create route tables only when needed and use the Associations option to map subnets to the route table in your Amazon VPC

Practice 24) Use Amazon VPC Peering (new) : Amazon Web Services has introduced VPC peering feature which is quite useful one. AWS VPC peering connection is a networking connection between two Amazon VPCs that enables you to route traffic between them using private IP addresses. Currently it can be in same AWS region, Instances in either VPC can communicate with each other as if they are within the same network. Since AWS uses the existing infrastructure of a VPC to create a VPC peering connection; it is neither a gateway nor a VPN connection, and does not rely on a separate piece of physical hardware (which essentially means there is no single point of failure for communication or a bandwidth bottleneck).

We have seen it is useful in following scenarios :

1. Large Enterprises usually run Multiple Amazon VPC in single region and some of their applications are so interconnected that they may need to access them privately + securely inside AWS. Example Active Directory, Exchange, Common business services will be usually interconnected.
2. Large Enterprise have different AWS accounts for different business units/teams/departments , at times systems deployed by some business units in different AWS accounts need to be shared or need to consume a shared resource privately. Example: CRM , HRMS ,File Sharing etc can be internal and shared. In such scenarios VPC peering comes very useful.
3. Customer can peer their VPC with their core suppliers to have tighter integrated access of their systems.
4. Companies offering Infra/Application Managed Services on AWS can now safely peer into customer Amazon VPC and provide monitoring and management of AWS resources.

Practice 25) Use Amazon VPC: It is highly recommended that migrate all your new workloads inside Amazon VPC rather than Amazon Classic Cloud. I also strongly recommend to migrate your existing workloads from Amazon Classic cloud to Amazon VPC in phases or one shot which ever is feasible. In addition to the benefits of the VPC that is detailed in the start of the article, AWS has started introducing lots of features which are compatible only inside VPC and in the AWS marketplace as well there are lots of products which are compatible only with Amazon VPC.  So make sure you leverage this strength of VPC. If you require any help for this migration please contact me. 

readers feel free to suggest more.. I will link relevant ones in this article

MySQL Amazon EC2 Cross region Master-Slave Replication using SSL

Well before Amazon VPC many customers had deployed their application on AWS classic public cloud. Still many customers are inside AWS classic cloud and have not yet migrated to Amazon VPC. Keeping them in mind, This post explains the configuration steps to enable database replication between MySQL Master and Slave on two different Amazon EC2 regions (on AWS Classic cloud). 

• MySQL Master is setup in US-East and Slave is setup in APAC. 
• Asynchronous replication is enabled between Master and Slave over internet. To protect the data in transit SSL encryption is used. It is bad idea to transfer DB data over Amazon EC2 regions in clear without protection.
• This technique is useful only for MySQL on EC2 on AWS Classic cloud. For replication of MySQL in Amazon VPC, i recommend VPN between the Amazon EC2 regions. To know more about this refer Amazon VPC- VPN article.
• Since the Data is encrypted and decrypted by MySQL EC2 instances itself, it adds additional burden on EC2 instance. Usually MySQL is Memory/IO intensive, but adding SSL enc/dec function to it, we need to power it with more CPU as well. 
• It is recommended to use larger EC2 instances with very good CPU,Memory and IO bandwidth for such cases. CPU is needed for SSL, Memory for MySQL and IO for replication over internet. Thought it is possible to set this up with small/medium EC2, it will not give good performance.
• Elastic IP is a mandatory recommendation for this replication process
• This setup is usually good for warm Disaster Recovery mode, Where MySQL is already up to date on alternate Amazon EC2 region and you need to bring the Web/App/LB tier using CloudFormation templates in event of primary location outage.
• RDS MySQL does not offer Inter Region DR currently, so it is out of scope of this article.
• Since the replication happens over internet between MySQL Master and Slave EC2, X amount(secs -->mins) of replication lag is natural. Please factor this in your RPO/RTO objectives of your DR.
• This is not a replacement to backup policy because an accidentally issued DELETE command will also be carried out on the slave; but replication can help protect against hardware failures though. So it is recommended to take regular backups and snapshots and transfer them to alternate amazon ec2 region.For copying AMI,Snapshots and Security groups to Alternate Amazon EC2 region, Refer articles:

1. http://harish11g.blogspot.com/2013/05/Amazon-EC2-AMI-Copy-between-Multiple-Amazon-EC2-regions-performance-tips-how-to.html
2. http://harish11g.blogspot.com/2013/05/Amazon-EBS-Snapshot-Copy-Multiple-Amazon-EC2-regions-howto-performance-tips-practices.html
3. http://harish11g.blogspot.com/2013/05/Migrating-Amazon-Security-groups-to-Multi-Region-setup.html

Step 1: Setup MySQL Master/Slave replication with SSL:

Launch Two RHEL 64 bit EC2 instance with MySQL Database on EC2. The entire setup is done inside AWS Classic cloud and not Amazon VPC. Master MySQL Amazon EC2 is in US-EAST (Master US-East-1a) region and One Amazon EC2 for MySQL Slave in Asia Pacific Singapore region (ap-southeast-1a)

Install MySQL on EC2 and configure some basic steps:

yum update

rpm -Uvh http://mirror.nus.edu.sg/Fedora/epel/6/x86_64/epel-release-6-7.noarch.rpm

rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm

yum --enablerepo=remi -y install mysql mysql-server

service mysqld start

chkconfig --levels 235 mysqld on

mkdir /var/log/mysql

touch /var/log/mysql/mysql-bin.log

chown -R mysql:mysql /var/log/mysql

Disable IPTABLES:

service iptables stop

service ip6tables stop

Step 2: Configure the following steps on MySQL Master & Slave Amazon EC2's 

To enable SSL, Edit the my.cnf file

vim /etc/my.cnf

Add a line with the word ssl to the [mysqld] section:  

ssl

Restart MySQL

mysql> show variables like '%ssl%';

+---------------+-------+

| Variable_name | Value |

+---------------+-------+

| have_openssl  | YES   |

| have_ssl      | YES   |

| ssl_ca        |       |

| ssl_capath    |       |

| ssl_cert      |       |

| ssl_cipher    |       |

| ssl_key       |       |

+---------------+-------+

7 rows in set (0.00 sec)

The response shows that now SSL is enabled.

Step 3: Steps to create SSL Certificates on MySQL EC2

Now we need to create the CA, server and client certificates that we need for the SSL connections.  Create these certificates in the directory /etc/mysql/certs/

Step 3.1 : On MySQL Master EC2 - create CA/Server/Client Certificate:

mkdir -p /etc/mysql/certs

cd /etc/mysql/certs

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 1000 -key ca-key.pem > ca-cert.pem

Create server certificate:

openssl req -newkey rsa:2048 -days 1000 -nodes -keyout server-key.pem > server-req.pem

openssl x509 -req -in server-req.pem -days 1000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > server-cert.pem

Create client certificate:

openssl req -newkey rsa:2048 -days 1000 -nodes -keyout client-key.pem > client-req.pem

openssl x509 -req -in client-req.pem -days 1000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > client-cert.pem

NOTE: Please use “Unique Common Names” while creating certificates otherwise SSL will not work.

Add the following ssl-ca, ssl-cert, and ssl-key lines under [mysqld] section in /etc/my.cnf

[mysqld]

ssl
ssl-ca=/etc/mysql/certs/ca-cert.pem
ssl-cert=/etc/mysql/certs/server-cert.pem
ssl-key=/etc/mysql/certs/server-key.pem

Restart MySQL to load the SSL configurations

Step 3.2 : Configure the certificates on MySQL Slave EC2 

Copy the following certificate files to MySQL Slave EC2 using SCP to a new directory called /etc/mysql/certs/

ca-cert.pem,client-cert.pem,client-key.pem,server-cert.pem,server-key.pem

Add the following in MySQL Slave /etc/my.cnf under [mysqld]

[mysqld]

ssl

ssl-ca=/etc/mysql/certs/ca-cert.pem

ssl-cert=/etc/mysql/certs/client-cert.pem

ssl-key=/etc/mysql/certs/client-key.pem

Restart MySQL to load the SSL configuration.

Step 4: Configuring replication settings

Step 4.1: On MySQL Master Amazon EC2 

Add the following in /etc/my.cnf under [mysqld]

server_id               = 1

log_bin                 = /var/log/mysql/mysql-bin.log

#log_bin_index      = /var/log/mysql/mysql-bin.log.index

max_binlog_size    = 100M

expire_logs_days   = 1

Restart MySQL

service mysqld restart

Execute the following in MySQL EC2 shell terminal of Master:     

Note:The REQUIRE SSL string is optional; if you leave it out, slave_user will be allowed to connect through encrypted and also non encrypted connections. If you use REQUIRE SSL, then only encrypted connections are allowed between slave and master.

$ mysql -u root -p

mysql> GRANT REPLICATION SLAVE ON *.* TO 'slave_user1'@'%' IDENTIFIED BY 'mysql_mslave1' REQUIRE SSL;

mysql> FLUSH PRIVILEGES;

mysql> FLUSH TABLES WITH READ LOCK;

mysql> show master status;

+------------------+----------+--------------+------------------+

| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |

+------------------+----------+--------------+------------------+

| mysql-bin.000001 |      335  |              |                  |

+------------------+----------+--------------+------------------+

1 row in set (0.00 sec)

mysql> unlock tables;

mysql> quit

            

Step 4.2:On MySQL Slave EC2 instance :

Edit the my.cnf file

server-id                = 2

log_bin                  = /var/log/mysql/mysql-bin.log

#log_bin_index       = /var/log/mysql/mysql-bin.log.index

max_binlog_size     = 100M

expire_logs_days    = 1

Restart MySQL

service mysqld restart

Execute the following in MySQL EC2 shell terminal of Slave:     

mysql -u root -p

mysql> slave stop;

mysql> CHANGE MASTER TO MASTER_HOST='ec2-IP-49.compute-1.amazonaws.com', MASTER_USER='slave_user1', MASTER_PASSWORD='mysql_mslave1', MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=107,MASTER_SSL=1, MASTER_SSL_CA = '/etc/mysql/certs/ca-cert.pem', MASTER_SSL_CERT = '/etc/mysql/certs/client-cert.pem', MASTER_SSL_KEY = '/etc/mysql/certs/client-key.pem';

mysql> START SLAVE;

mysql> SHOW SLAVE STATUS;

 ** there should be a message "Waiting for master to send event"

mysql>quit

Step 5: Test for MySQL AWS Inter Region Replication between MySQL Master & Slave:

On MySQL Master EC2 instance

mysql -u root -pmysql> create database rep_test_US_APAC;

mysql> use rep_test_US_APAC;

mysql> CREATE TABLE t1(c1 int,c2 var(100));

Note: In MySQL Slave Amazon EC2 in APAC You can find that database and the table is created. Next step, Insert data into the Master MySQL EC2, in few seconds/minutes replication lag check you can find that the data is present on the slave. Stop inserting the records to master and verify that the slave is up to date, Check the replication lag status on Slave EC2 by applying following command

$ mysql -u root -p -e 'show slave status\G' | grep -i seconds

Seconds_Behind_Master: 0

SHOW SLAVE STATUS \G

Note:It is important that both Slave_IO_Running and Slave_SQL_Running have the value "Yes" in the output (otherwise something went wrong, and you should check your configuration steps again and /var/log/syslog to find out about any errors); as you're using an SSL connection now, you should also find values in the fields Master_SSL_Allowed, Master_SSL_CA_File, Master_SSL_Cert, and Master_SSL_Key.

This article was co-authored with ramprasad. His linkedin handle in.linkedin.com/in/ramprasadguru

Amazon Elastic IP - Inter Region Migration Best Practices

Elastic IP (EIP) addresses are static IP addresses designed for Amazon Cloud infrastructure. Elastic IP addresses allow you to mask an Amazon EC2 instance or Amazon availability zone failures by programmatically remapping your public IP addresses to any EC2 instance associated with your account. Rather than waiting on a technician to reconfigure or replace your faulty EC2's, or waiting for DNS to propagate to all of your customers, EIP enables you to design applications with high availability in AWS infrastructure.
Though Amazon Elastic IP’s are associated with your account, they operate in Amazon EC2 regional scope meaning Elastic IP’s cannot be migrated or reused between AWS regions. This is normal because you cannot use the IP address assigned by your USA Hosting provider inside Singapore DC; you can only access it from internet if it is public facing. This applies to EIPs in Amazon EC2 regions as well. Let us explore some best practices and tips for migrating applications using Elastic IP’s between regions:

• Elastic IP are usually associated to EC2’s in the Front facing tiers like Web Servers, Load Balancers (Nginx, HAProxy) and Firewalls (Checkpoint, Sophos) and inturn are pointed in DNS. When you are migrating the traffic to different Amazon EC2 region, then these Elastic IP’s cannot be migrated instead your DNS entries need to re pointed to new set of EIP associated in the target region.
• Some practices that need to be followed during the migration of this scenario are:
• Update your time-to-live (TTL) value on your DNS that is pointing to the Elastic IP address with a suitable number such as 300 seconds or less for the DNS cache to expire. This way re association can be faster.This number can be increased to normal TTL values post the migration.
• If you use sophisticated Managed DNS providers like Ultra DNS, DynDNS etc you can pre-configure the ElasticIP’s of the target region LB or Firewall in their DNS entries and direct the traffic to target region within 30 seconds in event of DR scenarios or migration.
• In both the scenarios it is recommended to keep the TTL at suitable level for your use case, Since you will be charged based on number DNS requests by some DNS providers, any decrease in DNS TTL could result in an increase in number of DNS requests, increase load on your current DNS service, and increase your bill from your DNS service provider.
• Sometimes Elastic IP’s are associated to common services like Zookeeper, Chef Server, Nagios Monitoring Server, Discovery service EC2’s etc.  Since these common services need to be accessed from other EC2’s in several tiers of the application stack they need to be either associated with Elastic IP or internal DNS names for easy discovery. Some recommended practices that need to be followed during the migration of these scenarios are:
• It is recommended to use DNS host names instead of EIP for most of the internal node communications. You can discover common services using the DNS names instead of EIP. This way during migration or in event of DR you need not change configuration files and properties of all calling nodes with target EIP's.
• In case EIP is used for these services, it is recommended prior to migration from source region, you should ensure that all inter node communication using EIP’s are identified for remapping to EIP’s of the target region. This has to be done in CloudFormation templates, scripts, security groups,configuration files and all assets which are pointing to Source EIP.

High Availability @ Load Balancing Layer-HAProxy / ELB

Architecting High Availability at the Load Balancing layer is one of the important aspects in the web scale systems in AWS. We can follow multiple strategies for achieving the same. I am listing some of the designs for achieving the same.

Pattern 1: Route 53 DNS RR + HAProxy

Route53 is a Managed DNS service provided by Amazon Web Services. Route 53 supports Round robin and weighted algorithms. If the Route53 DNS server has several entries for a given hostname, it will return all of them in a rotating order. This way, various users will see different addresses for the same name and will be able to reach different EC2 instances in LB Tier.

$ host -t a HAProxyTestXYZ.com

HAProxyTestXYZ.com. has address 50.19.82.183 (Primary EIP)

HAProxyTestXYZ.com. has address 23.23.174.254 (Secondary EIP)

Example, if we attach the Elastic IP’s of 2 HAProxy EC2 instances under the Route 53, both the IP’s are sent to the user browsers by the Route 53 DNS. In case, the algorithm configured is Round Robin at the Route 53 level, then browser- 1 will get EIP-1(50.19.82.183) of HAProxy-1 as the primary IP and browser -2 will get EIP-2 (23.23.174.254) of HAProxy-2 as the primary IP in rotation basis.  The browser- 1 will contact the HAProxy-1 and in case HAProxy-1 is not reachable it will contact the secondary EIP which is HAproxy-2 and so forth. This is an age old technique generally used by search engines, content servers (or) web scale systems for achieving scalability in LB layer. But this method does not provide any means of High availability @ LB layer. It requires additional measures to permanently check the HAProxy EC2 LB instances status and switch a failed EC2 instance EIP to another HAProxy EC2 LB. For this reason, this pattern is generally used as a complementary solution in High Availability, not as a primary one.  For achieving better stability at this layer in AWS, I usually recommend having 2 or more HAProxies distributed on multiple AZ’s inside the Amazon EC2 region. This way if one of the HAProxy is down, the website still functions with the help of other HAProxies and even if the entire Amazon EC2 AZ is down still the HAProxies in the other AZ can handle the requests and keep the website active. Some load tests have proven that HAProxy on m1.large EC2 instance can handle close to ~4500+ HTTP requests/second. So depending upon the number of concurrent requests/sec needed on your application you can go ahead and attach multiple HAProxy EC2 instances to the Route 53. Now that we achieved availability horizontally using the Route53 DNS Round Robin in HAProxy layer let us try to understand the intricacies behind this architecture.  Since we now have 2 or more HAProxies what will happen to the contextual web sessions data that resides in the application servers. HAProxies need to know in which application server the session data of the user resides else the requests will have authorization failures.

There are 2 architecture designs we can follow for solving this contextual problem they are:

Stateless Application design:  This is the recommended and widely used design. The web session data is separated out from the Web/App server memory and they are kept in common cache stores like MemCacheD, TerraCotta etc. Refer this URL to understand common patterns followed in AWS: http://harish11g.blogspot.in/2012/10/web-session-synchronization-aws-ec2.html

Since the session data is now kept in a common store like MemCacheD, HAProxies can direct their requests to any of the web/app servers attached under it without knowing where the session state is mapped. Whenever any web/app server receives the request from any of the HAProxies, it will validate and authorize the session data from the common store.  In event of any HAProxy or Web/App EC2 failure still the website functions without problems because other HAProxies and Web/App servers are still able to handle the subsequent requests. Thus we achieve availability and scalability on the HAProxy/Load Balancing layer following this model.

Sticky Application design: Things are usually not ideal and the way we assume to be in real world. Some applications are still designed with stateful nature and they store the session data, cache data etc. in their web/app server memory.  We can always recommend the application teams to re-architect this model to stateless, but not always this suggestion works for short term migrations, inter dependencies etc.  So as architects we need to find way to live and cope up with this design and still try to achieve availability on the load balancing layer. HAProxy follows a technique called as “Cookie Learning” and “Cookie Insertion” to help state full applications. HAProxy can be configured to learn the application cookie ("JSESSIONID"), when HAProxy receives the user's request, it checks if it contains this particular cookie and a known value. If this is not the case, it will direct the request to any Web/App EC2 server, according to the load balancing algorithm configured. HAProxy will then extract the cookie value from the response and add it along with the server's identifier to a local table. When the user request comes back again, the load balancer sees the cookie, lookups the table and finds the Web/App EC2 server to which it forwards the request. Let me detail this important flow a little bit;

HAProxy-1 EC2 instance will receive client’s requests from the browser. If a request does not contain a cookie, it will be forwarded to a valid Web/App EC2 Instance Apache-A. In return, if a JESSIONID cookie is seen, the Web/App EC2 Instance name (Example “A”)will be prefixed into it, followed by a delimiter ('~') like "JSESSIONID=A~xxx".When the browser client requests again with the cookie  JSESSIONID=A~xxx", HAProxy-1 will know that it must be forwarded to Web/App Instance Apache-A. The EC2 Instance name ”A” will then be extracted from cookie before it is sent to the Web/App EC2 Instance Apache-A.

If Web/App EC2 Instance Apache-A dies, then requests will be sent to another valid server Web/App EC2 Instance Apache-B by LB and the cookie will be reassigned.

If HAProxy-1 itself dies, then requests will be sent to HAProxy-2 which will identify the Web/App EC2 instance to forward the request. This way even if the subsequent requests moves from HAProxy-1 to HAProxy-2 in event of HAProxy-1 failure, still the requests are sent to the same Web/App instance Apache-A by the cookie learning/insertion mechanism of HAProxy.

Sample HAProxy Settings to achieve this

listen webfarm 192.168.1.1:80

       mode http

       balance roundrobin

       cookie JSESSIONID prefix

       option httpclose

       option forwardfor

       option httpchk HEAD /index.html HTTP/1.0

       server Apache-A 192.168.1.11:80 cookie A check

       server Apache-B 192.168.1.12:80 cookie B check

Note: You can use more sophisticated DNS services like UltraDNS , DNSMadeEasy etc also in this architecture to better control the Load balancing and traffic direction at the DNS level. 

Pattern 2: Route 53 DNS RR + HAProxy in Active-Passive mode

This is an extension of the Route53 DNS RR pattern and everything discussed in the previous pattern still applies to this context. In addition to associating HAProxies horizontally under Route53, we will build availability for every HAProxy vertically as well in this pattern. High Availability is built taking into consideration HAProxy process failure and HAProxy EC2 instance failure.

2 or more HAProxies from multiple AZ’s are taken and they are attached with Amazon Elastic IP’s. These Elastic IP’s are then associated in Route 53 with DNS RR. These HAProxies are now “Active” and are ready to handle the user requests. For HA, another equivalent set of HAProxies are launched in the respective AZ’s as “Standby”. In event of the “Active” HAProxy failure, the Standby HAProxy remaps to the same Amazon Elastic IP takes over the subsequent requests from the client.

In the above diagram, there are 2 HAProxies in “Active” state with Elastic IP’s 50.19.82.183 & 23.23.174.254. They are deployed across Multiple Availability Zones inside an Amazon EC2 region. Another 2 HAProxies are launched in respective AZ’s, but they are kept idle in “Standby” state. In event of HAProxy-1 (EIP: 50.19.82.183) failure the Elastic IP is remapped to Standby HAProxy-3 in the same AZ. The remapping takes ~60 seconds and the HAPorxy-3 will be handling the subsequent requests directed by the browsers to the 50.19.82.183 IP.

Broadly there are 2 levels of failure in this pattern;

Failure @ HAProxy Process level

Failure @ HAProxy EC2 instance level

Failure @HAProxy Process level:  When HAProxy Process at the “Active” server fails; we can detect this using KeepAliveD and switch the Elastic IP from Active -> Standby. We have observed it takes ~60-120 seconds for the standby to takeover. During this time the particular HAProxy alone will be unreachable. KeepAliveD script is configured in both the Active and standby HAProxy EC2 instance. KeepAliveD implements a set of checkers to dynamically and adaptively maintain and manage load balanced server pool according to their health. High availability is achieved by Virtual Router Redundancy Protocol VRRP protocol of the KeepAliveD. Since Amazon EC2 currently does not support Multicast protocol we need to configure KeepAliveD with Unicast TCP in this scenario.  For more details refer http://www.keepalived.org/. Mean time manually we can bring the failed HAProxy Process up and make this as the new standby.

Script Name: “/etc/keepalived/keepalived.conf”

vrrp_script chk_haproxy {           # Requires keepalived-1.1.13 script "killall -0 haproxy"     # cheaper than pidof interval 20                      # check every 2 seconds weight 20                        # add 2 points of prio if OK } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 ipriority 101                    # 101 on master, 100 on backup vrrp_unicast_bind 10.215.31.4       #internal IP address of EC2 instance 01 vrrp_unicast_peer 10.85.110.252   #internal IP address of EC2 instance 02 notify_master "/etc/keepalived/vrrp.sh" track_script { chk_haproxy weight 20 } }

Script Name: /etc/keepalived/vrrp.sh

#vrrp.sh #!/bin/bash cd /opt/aws/apitools/ec2/bin #DisAssociate EIP from this instance. ./ec2-disassociate-address --aws-access-key XXXXXXX –aws-secret-key XXXXXXX  [EIP] #Mapping EIP to secondary server ./ec2-associate-address --aws-access-key XXXXXXX  –aws-secret-key XXXXXXX  [EIP] -i [ec2_instance_id_of_primary_or_secondary]

Failure @ HAProxy EC2 instance level: When the Active HAProxy EC2 instance itself fails; we can detect this using Heartbeat and switch the Elastic IP from Active -> Standby. "Heartbeat" tool connects two servers and checks the regular "pulse" or "heartbeat" between them. The standby server takes over the work of the “Active” as soon as it detects an alteration in the "heartbeat" of the former. We have observed it takes ~120+ seconds for the standby to takeover. During this time the particular HAProxy alone will be unreachable. Heartbeat has to be configured in both the Active and standby HAProxy EC2 instance. Since Amazon EC2 currently does not support Multicast protocol we need to configure Heartbeat with Unicast UDP in this scenario. Mean time manually we can bring the failed HAProxy EC2 instance up and make this as the new standby.

Script Name: /etc/ha.d/ha.cf

logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 30 initdead 120 udpport 694 ucast eth0 xx.xxx.xxx.xxa #Internal IP of EC2 instance 01 ucast eth0 xx.xxx.xxx.xxb #Internal IP of EC2 instance 02 auto_failback off

Script Name: Create a script named “elastic_ip” in both the servers.

#!/bin/bash I_ID="[ec2_instance_id" # different for each EC2 servers. ELASTIC_IP="X.X.X.X" case $1 in     start) ec2-associate-address --aws-access-key XXXXX --aws-secret-key XXXXX "$ELASTIC_IP" -i "$I_ID" > /dev/null        echo $0 started        ;;     stop) ec2-disassociate-address --aws-access-key XXXXX --aws-secret-key XXXXX "$ELASTIC_IP" > /dev/null     echo $0 stopped        ;;     status) ec2-describe-addresses --aws-access-key XXXXX --aws-secret-key XXXXX | grep "$ELASTIC_IP" | grep "$I_ID" > /dev/null     # grep will return true if this ip is mapped to this instance     [ $? -eq 0 ] && echo $0 OK || echo $0 FAIL     ;; esac

Why do we need this redundancy in the HAProxy layer?

Not all the times the DNS RR with LB Cookie Insertion alone is enough for ensuring availability;

Case 1: Imagine you have not automated the scalability @ Load Balancing Layer and one of your Load balancer is down. You do not want to be waked up in the middle of the night rather it is better to have a standby Load Balancer automatically replacing the failed one. Manually you can replace the faulty LB next day.

Case 2: You have a gaming site where long running TCP sockets are established from flash gaming clients to the LB layer. You have planned the capacity of Front end Load Balancers with concurrent connections/sec. Now couple of your Load balancers are down, the new connections will be established to other running LB, but overall your site will now start performing poorly and chances are new connections are exhausted after few hours of heavy traffic. It is better to automatically detect and replace the faulty LB EC2 instance with the standby.

Case 3: Some clients cache the IP address of the Load Balancer, Some of them have long running sticky sessions with web/app, Some hardware devices can take only IP address to push data into the Server infrastructure. Though it is suggested to resolve the IP using DNS, still in reality some use cases does not work the same way.

Pattern 3: Use ELB

Do not worry about all the above patterns, just go and configure Amazon Elastic Load Balancing (ELB). For most of the use cases ELB is more than sufficient.

Amazon Elastic Load Balancer can distribute incoming traffic across your Amazon EC2 instances in a single Availability Zone or multiple Availability Zones. Amazon Elastic Load Balancing automatically scales its request handling capacity in response to incoming application traffic. It can handle 20k+ concurrent requests/sec with ease. It enables you to achieve even greater fault tolerance in your applications, seamlessly providing the amount of load balancing capacity needed in response to incoming application traffic. Elastic Load Balancing detects unhealthy instances within a pool and automatically reroutes traffic to healthy instances until the unhealthy instances have been restored. Any faulty Load balancers in the ELB tier are automatically replaced.

Though for most of the common use cases ELB is more than sufficient in AWS. There are some unique cases which demand the use of Load balancers like HAProxy, Nginx and NetScaler in our architecture in the AWS infrastructure. I will be writing some articles in coming months comparing ELB and others in use case perspectives and context. 

Why do we need Elastic Load Balancing ?

Dissecting Amazon ELB: 18 things you should know

Monitoring ELB with Amazon CloudWatch

Configuring Amazon ELB using AWS Management Console

Amazon ELB Implementation Architectures

Amazon Elastic Load balancing benefits

Offloading SSL in Amazon ELB

Amazon ELB and Flash Traffic

Amazon ELB Supported Algorithms and Protocols

I would like to thank Ramprasad for co-authoring some portions of this article with me.