Showing posts with label Managed Services. Show all posts
Showing posts with label Managed Services. Show all posts

Friday, June 12, 2015

27 Best Practice Tips on Amazon Web Services Security Groups

AWS Security Groups are one of the most used and abused configurations inside an AWS environment if you are using them on cloud quite long. Since AWS security groups are simple to configure, users many times ignore the importance of it and do not follow best practices relating to it. In reality, operating on AWS security groups every day is much more intensive and complex than configuring them once. Actually, nobody talks about it! So in this article, I am going to share our experience in dealing with AWS Security groups since 2008 as a set of best practice pointers relating to configuration and day to day operations perspective.
In the world of security, proactive and reactive speed determines the winner. So a lot of these best practices should be automated in reality. In case your organizations' Dev/Ops/Devops teams needs help with security group best practices automation, feel free to contact me.

AWS released so many features in the last few years relating to Security, that we should not visualize Security groups in isolation, It just does not make sense anymore. The Security Group should always be seen in the overall security context, with this I start the pointers.

Practice 1:  Enable AWS VPC Flow Logs for your VPC or Subnet or ENI level. AWS VPC flow logs can be configured to capture both accept and reject entries flowing through the ENI and Security groups of the EC2, ELB + some more services. This VPC Flow log entries can be scanned to detect attack patterns,alert abnormal activities and information flow inside the VPC and provide valuable insights to the SOC/MS team operations.

Practice 2: Use AWS Identity and Access Management (IAM) to control who in your organization has permission to create and manage security groups and network ACLs (NACL). Isolate the responsibilities and roles for better defense. For example, you can give only your network administrators or security admin the permission to manage the security groups and restrict other roles.

Practice 3: Enable AWS Cloud Trail logs for your account. The AWS Cloud Trail will log all the security group events and it is needed for management and operations of security groups. Event streams can be created from AWS Cloud Trail logs and it can be processed using AWS Lambda. For example : whenever a Security Group is deleted , this event will be captured with details on the AWS Cloud Trail logs. Events can be triggered in AWS Lamdba which can process this SG change and alert the MS/SOC on the dashboard or email as per your workflow. This is a very powerful way of reacting to events within span of <7 minutes. Alternatively, you can process the AWS Cloud Trail logs stored in your S3 every X frequency as a batch and achieve the above. But the Operation teams reaction time can vary depending on generation and polling frequency of the AWS Cloud Trail logs. This activity is a must for your operations team.

Practice 4: Enable AWS App Config for your AWS account. App records all events related to your security group changes and can even send emails.

Practice 5: Have proper naming conventions for the Amazon Web Services security group. The naming convention should follow a enterprise standards. For example it can follow the notation: "AWS Region+ Environment Code+ OS Type+Tier+Application Code"
Security Group Name - EU-P-LWA001
AWS Region ( 2 char ) = EU, VA, CA etc
Environment Code (1 Char)  = P-Production , Q-QA, T-testing, D-Development etc
OS Type (1 Char)= L -Linux, W-Windows etc
Tier (1 Char)= W-Web, A-App, C-Cache, D-DB etc
Application Code ( 4 Chars) = A001
We have been using Amazon Web Services from 2008 and found over the years managing the security groups in multiple environments is itself a huge task. Proper naming conventions from beginning is a simple practice, but will make your AWS journey manageable.

Practice 6: For security in depth, make sure your Amazon Web Services security groups naming convention is not self explanatory also make sure your naming standards stays internal. Example : AWS security group named UbuntuWebCRMProd is self explanatory for hackers that it is a Production CRM web tier running on ubuntu OS. Have an automated program detecting AWS security groups with Regex Pattern scanning of AWS SG assets periodically for information revealing names and alert the SOC/Managed service teams.

Practice 7: Periodically detect, alert or delete AWS Security groups not following the organization naming standards strictly. Also have an automated program doing this as part of your SOC/Managed service operations.  Once you have this stricter control implemented then things will fall in line automatically.

Practice 8: Have automation in place to detect all EC2,ELB and other AWS assets associated with Security groups. This automation will help us to periodically detect Amazon Web Services Security groups lying idle with no associations, alert the MS team and cleanse them. Unwanted security groups accumulated over time will create unwanted confusion.

Practice 9: In your AWS account, when you create a VPC, AWS automatically creates a default security group for the VPC. If you don't specify a different security group when you launch an instance, the instance is automatically associated with the appropriate default security group. It will
allow inbound traffic only from other instances associated with the "default" security group and allow all outbound traffic from the instance. The default security group specifies itself as a source security group in its inbound rules. This is what allows instances associated with the default security group to communicate with other instances associated with the default security group. This is not a good security practice. If you don't want all your instances to use the default security group, you can create your own security groups and specify them when you launch your instances. This is applicable to EC2 , RDS , ElastiCache and some more services in AWS. So detect "default" security groups periodically and alert to the SOC/MS.

Practice 10: Alerts by email and cloud management dash board should be triggered whenever critical security groups or rules are added/modified/deleted in production.  This is important for reactive action of your managed services/security operations team and audit purpose.

Practice 11 : When you associate multiple security groups with an Amazon EC2 instance, the rules from each security group are effectively aggregated to create one set of rules. AWS uses this set of rules to determine whether to allow access or not. If there is more than one SG rule for a specific port, AWS applies the most permissive rule. For example, if you have a rule that allows access to TCP port 22 (SSH) from IP address 203.0.113.10 and another rule that allows access to TCP port 22 for everyone, then everyone will have access to TCP port 22 because permissive takes precedence.
Practice X.1 : Have automated programs detecting EC2 associated with multiple SG/rules and alert the SOC/MS periodically. Condense the same manually to 1-3 rules max as part of your operations.
Practice X.1 : Have automated programs detecting conflicting SG/rules like restrictive+permissive rules together and alert the SOC/MS periodically.

Practice 12 : Do not create least restrictive security groups like 0.0.0.0/0 which is open to every one.
Since web servers can receive HTTP and HTTPS traffic open, only their SG can be permissive like
0.0.0.0/0,TCP, 80, Allow inbound HTTP access from anywhere
0.0.0.0/0,TCP, 443, Allow inbound HTTPS access from anywhere
All least restrictive SG created in your account should be alerted to SOC/MS teams immediately.

Practice 13: Have a security policy not to launch servers with default ports like 3306, 1630, 1433, 11211, 6379 etc. If the policy has to be accepted, then security groups also have to be created on the new hidden listening ports instead of the default ports. This provides a small layer of defense, since one cannot infer the information from the security group port on the EC2 service it is protecting. Automated detection and alerts should be created for SOC/MS, if security groups are created with default ports.

Practice 14: Applications which require stricter compliance requirements like HIPAA, PCI etc to be met need end to end transport encryption to be implemented on server back end in AWS. The communication from ELB to Web->App->DB->Others tiers need to be encrypted using SSL or HTTPS. This means only secured ports like 443, 465, 22 are permitted in corresponding EC2 security groups. Automated detection and alerts should be created for SOC/MS if security groups are created on secure ports for regulated applications.

Practice 15: Detection , alert and actions can be taken by parsing the AWS Cloud Trail logs based on usual patterns observed in your production environment
Example:
15.1 :If a port was opened and closed in <30 or X mins in production can be a candidate for suspicious activity if it is not normal pattern for your production
15.2 :If a permissive Security Group was created and closed in <30 or X mins can be a candidate for suspicious activity if it is not the normal pattern for your production
Detect anomalies on how long a change effected and reverted in security groups in production.

Practice 16: In case ports have to be opened in Amazon Web Services security groups or a permissive AWS security group needs to be applied, Automate this entire process as part of your operations such that a security group is open for X agreed minutes and will be automatically closed aligning with your change management. Reducing manual intervention avoids operational errors and adds security.

Practice 17: Make sure SSH/RDP connection is open in AWS Security Group only for jump box/bastion hosts for your VPC/subnets. Have stricter controls/policies avoid opening SSH/RDP to other instances of production environment. Periodically check , alert and close for this loop hole as part of your operations.

Practice 18: It is a bad practice to have SSH open to the entire Internet for emergency or remote support. By allowing the entire Internet access to your SSH port there is nothing stopping an attacker from exploiting your EC2 instance. The best practice is to allow very specific IP address in your security groups, this restriction improves the protection. This could be your office or on premise or DC through which you connect your jump box.

Practice 19: Too much or Too less: How many security groups for a usual multi tiered web app is preferred is a frequently asked question ?
Option 1 : One security group cutting across multiple tiers is easy to configure, but it is not a recommended for secure production applications.
Option 2: One Security group for every instance is too much protection and tough to manage operationally on longer term
Option 3: Individual Security group for different tiers of the application, For example : Have separate security groups for ELB, Web , App, DB and Cache tiers of your application stack.
Periodically check whether Option 1 type rule is being created in your production and alert the SOC/MS.

Practice 20: Avoid allowing  UDP or ICMP for private instances in Security groups. Not a good practice unless specifically needed.

Practice 21: Open only specific ports, Opening range of ports in a security group is not a good practice. In the security group you can add many inbound ingress rules, While opening the ports it is always advised to open for specific ports like 80,443, etc rather than range of ports like 200-300.


Add rules for communication between associated instances

Practice 22: Private Subnet instances can be accessed only from the VPC CIDR IP range. Opening instances to the public IP ranges is a possibility , but it does not make any sense. E.g., Opening HTTP to 0.0.0.0/0 in the SG of the private subnet instance does not make any sense. So detect and cleanse such rules.

Practice 23: AWS CloudTrail log captures the events related security. AWS lambda events or automated programs should trigger alerts to operations when abnormal activities are detected. For example:
23.1:Alert when X number of SG were added/deleted at "Y" Hours or Day by IAM user / Account
23.2:Alert when X number of SG Rules were added/deleted at "Y" Hours or Day by IAM user / Account

Practice 24: In case you are an enterprise make sure all security groups related activities of your production are part of your change management process. Security Group actions can be manual or automated with your change management in an enterprise.
In case you are an agile Startup or SMB and do not have complicated Change management process, then automate most of the security group related tasks and events as illustrated above on various best practices. This will bring immense efficiency into your operations

Practice 25: Use outbound/egress security groups wherever applicable within your VPC. Restrict FTP connection to any server on the Internet from your VPC. This way you can avoid data dumps and important files getting transferred out from your VPC. Defend harder and make it tougher !

Practice 26: For some tiers of your application, use ELB in front your instance as a security proxy with restrictive security groups - restrictive ports and IP ranges. This doubles your defense but increases the latency.

Practice 27: Some of the tools we use in conjunction to automate and meet above best practices are ServiceNow, Amazon CFT, AWS API'S, Rundeck, Puppet, Chef, Python , .Net and Java automated programs.

Note : In case your organizations Dev/Ops/Devops teams needs help on security group best practices automation on points listed above, feel free to contact me harish11g.aws@gmail.com

Thursday, January 23, 2014

AWS Cost Saving Tip 18: Using AWS Tagging with Graffiti Monkey to Saves costs

Tagging AWS assets and resources is a simple and an important feature that helps IT teams streamline their process of managing the environments. We can categorize AWS resources in variety of ways using the AWS Tagging feature. For example: 

  • Environment based Tags: Environment based tags can be used by customers who have Larger AWS footprints. These tags will help you to distinguish between production, development, and staging infrastructure deployed on AWS
  • Application based Tags: Used to describe the set of disparate resources (or clusters) that work together to deliver a product or service to a customer. Can be applied on Small to Big AWS Production environments to identify the resources used
  • Role based Tags– Used to describe the function of a particular node (web server, database server, load balancer, etc.). Owner – Used to identify the individual who is responsible for the instance. Both these cases are usable for small AWS foot prints.

How it helps ?
Tags can be very useful in understanding our AWS Billing. When you run large foot prints on AWS, constantly we need to monitor and manage our resource usage. Any IT operations team which manages AWS resources, constantly need to work on the following questions to optimize and manage their operations efficiently. Which AWS resources are used ? Which resources are lying idle (volumes, snapshots) in detached state ? Which Tiers or environments are consuming more billing than others? Which owner is exceeding the AWS cost allocated ? In short by starting to tag the AWS resources we will know how those assets are translating to monthly cost and take optimization decisions. Tagging is an simple but an essential feature that helps us to reduce and keep costs in control by improving our management efficiency in AWS.

AWS Tags in Detail:
Each tag consists of a key and a value, both of which the user can define. Currently AWS does not provide facility to TAG all AWS resources. Some of them can be Tagged only using API actions or the Command Line. The following Screen Shot shows all AWS resources and the restrictions that apply to them. Resources with tagging restrictions of None can be tagged with API actions, the CLI, and the console.
The following diagram illustrates how tagging works. In this example, i am assigning two tags to each of my EC2 instances, one called Owner and another called Stack. Each of the tags also has an associated value.
Owner = 8KMiles , Stack = Development, Stack = Production as illustrated in the below diagram



You can assign tags using the AWS Management Console, the Amazon EC2 command line interface (CLI), and the Amazon EC2 API.You can define tags from the EC2 console by selecting the relevant instance and selecting the “Tags” option.  Alternatively, you can select the instance, click “Actions” and select “Add/Edit Tags.” Below Screenshot shows how tags can be assigned using AWS Management Console

We have assigned Name, Owner , Environment and Contact for the 8KMiles Production instance using the AWS Tagging feature. The above is not the usual EC2 Managed Service naming standard , but just an simple illustration of the tag use case.
AWS Tagging Constraints and the Cost Leakage:
Currently AWS Tagging has some limitations as well. For example each tag associated with a single AWS resource, applies only to that particular resource and it is not propagated to the dependent resources attached to it. Example: Imagine if you have an EC2 instance, most of the times this EC2 instance is attached with dependent resources like EBS volumes, Snapshots etc. Usually there will be 1-2 EBS Volumes on normal scenarios and up to 8 EBS Volumes or more on RAID Scenarios. Number of snapshots are more in number and usually becomes unmanageable. Imagine how this problem will be propagated if you are running a Large AWS foot print with thousands of EC2 instances. 
You have to find each EBS volumes associated with a particular EC2 and Tag it separately.Managing this is easier said than done and usually slightest mis- management of these resources will lead to cost leakage. What if i can use a tool to automate tagging for dependent AWS resources. It will surely reduce the manual labor associated and help us remove all the dependent resources when we are removing the EC2. This avoids leakages and reduces cost of Cloud Operations in very dynamic AWS environments.

Welcome to Graffiti Monkey. 
The Graffiti Monkey goes around tagging things automatically. By looking at the tags of an EC2 instance, it copies those tags to the EBS Volumes that are attached to it, and then copies those tags to the EBS Snapshots as well. This small automation reduces hours of Manual labor on large AWS footprints, helps us efficiently manage the cloud infrastructure and reduce the cost leakages.

Step 1: Installation  of Graffiti Monkey 

Easiest way to install Graffiti Monkey using the usual PyPI channels.Please refer the below screenshot
link for installing pip.http://www.pip-installer.org/en/latest/installing.html

Step 2: Using Graffiti Monkey

It will identify the tags associated with EC2 instance and applies them to dependent resources like EBS volumes and snapshots saving hours of manual labor.

This Post is co authored with Nitheesh 8Kmiles


Monday, January 13, 2014

AWS CloudTrail and Splunk for better Cloud IT Operations

In any Cloud IT operations, gaining visibility into user activities is important for achieving compliance and meeting the regulatory standards of the enterprise. For years in AWS, we used to solve this with a combination of tools for log collection, alerts and privileged Identity access management. At times, it becomes cumbersome to install and manage these tools and overall operation cost effectively on Amazon Cloud. Like any other company pioneering managed services in AWS infrastructure, Our Cloud operations was in touch with Amazon Cloud product team for years giving input in this area. As an outcome of this, we see a new product line called as AWS Cloud Trail launched last year.

AWS Cloud Trail provides a record of AWS API calls made on your AWS account. AWS CloudTrail can be used for Resource Life Cycle Tracking, Operational Troubleshooting, Compliance Aid, and Security Analytics on AWS cloud infrastructure by integrating with the analytics tools like Splunk, Loggly, Sumo logic and etc.

In this article we are going to explore topics like :

  • How AWS CloudTrail can be integrated with Splunk App for AWS ?
  • How can we use it to operate our managed services efficiently ( with sample use cases answering following questions)
    • How do i get to know when security groups or Network details are changed ?
    • How do i know when EC2 instances are launched, terminated etc ?
    • How do i know what activities are carried on the Amazon Elastic IP's ?
    • What activities and changes happened in the IAM policies over the period ?
    • What are the Rare errors related to AWS account activities that can be tracked ?
How AWS CloudTrail + Splunk Integration works ?



Currently CloudTrail logs API calls on the following AWS services and in future you can expect more services to be launched. The current list is as follows :
·      Elastic Compute Cloud (EC2)
·      Elastic Block Store (EBS)
·      Virtual Private Cloud (VPC)
·      Relational Database Service (RDS)
·      Identity and Access Management (IAM)
·      Security Token Service (STS)
·      Redshift
·      CloudTrail

AWS CloudTrail captures AWS API calls made by or on behalf of an AWS account. Once the API calls are logged on files, it delivers these log files to an Amazon S3 bucket that you specify. Each call detail is logged in JSON format. Usually the JSON file is zipped and delivered to the bucket. SNS notification can be enabled to operations team when the log file is delivered.
Sample format of the log file nomenclature is shown below:
https://s3-us-west-2.amazonaws.com/<bucket_name>/<prefix_name>/AWSLogs/<AccountID>/CloudTrail/
us-west-2/2013/10/25/<AccountID>_CloudTrail_us-west-2_2014-11-25T12:25Z_bPlKAoJtDqSWWlTT.json.gz
How to activate AWS CloudTrail ?
By default CloudTrail is not enabled in your AWS account. To activate follow the below steps
1. Open the CloudTrail console in the AWS Management console
2. Click Get Started
3. On the Turn on CloudTrail page, you will be presented with the option to create a new Amazon S3 bucket or using an existing Amazon S3 bucket. In this article, I have selected the option to create new S3 bucket and provided the Bucket name as cloudtraillogs-splunk as illustrated below.

4. If you need to configure SNS notification and other settings, Click the option “Advanced”. Once clicked, below options will be displayed and these options are optional and not mandatory. I have enabled the option of SNS notification for every log delivery to our operations team.
Log file prefix: Enter a prefix for the Amazon S3 bucket. It is usually not needed since AWS automatically prefixes path automatically. You can see that path when you select the “View log file location”. But if you want Pattern for identification you can go ahead with the prefix.
Select Yes or No to “Include global services”?  IAM and AWS STS are the global services so if you want to log activities of those services as well, please select “Yes”. It is recommend to Select them because you can review the actions of the IAM.
SNS notification for every log files delivery?  If you want delivery, please enter the SNS topic name. The entered topic will be created in the SNS service and you must subscribe to it. After Saving, Click Subscribe. Once it is done, the service is activated and the log files are generated.

5. You can also edit the CloudTrail configuration and Turn off CloudTrail using the console. The console is illustrated in the below screen Shot.

What is Splunk App for AWS ?
Splunk App for AWS is designed to take the data from the AWS CloudTrail and provide the results using built-in dashboards. It is provided as a plug-in. Operations / Managed Services team can view this app and monitor and review some activities efficiently.

What is interesting with Splunk App for AWS?
  • It gives important insights into security-related activity such as unauthorized access attempts, simultaneous logins from geographically disparate locations and frequent changes to access control privileges. You can ensure security and compliance with continuous monitoring and a full audit trail of user activity using the Splunk App.
  • It provides option to configure email alerts based on the schedule
  • It provides easy option to monitor events, generate reports and export the reports and Search for events.

Steps to Integrate Splunk with AWS Cloud Trail :

Stage 1: Installing Splunk on AWS:
In order to use the Splunk App for AWS for this article, we are using the Enterprise Splunk AMI for illustrating the use cases.
1. Launch an EC2 Linux instance and based on the choice of Operating system download the Enterprise Splunk from the link http://www.splunk.com/download?r=header
2. Install the Splunk Enterprise



3. Export the following environmental values and then start the Splunk service.
# export SPLUNK_HOME=/opt/splunk
# export PATH=$SPLUNK_HOME/bin:$PATH$
$ splunk start
Commands to Restart and Check Status are below.
$ splunk restart
$ splunk status
4. On successful start the Spunk will be listening on the port 8000. Access the Splunk using the link http://<EC2 Public DNS>:8000. The below screen will be presented

Stage 2: Installing and Configuring Splunk App for AWS:
Splunk App for AWS fetches the data from the Amazon S3 bucket and provides the report in the Splunk GUI. Following are the steps to integrate Splunk app plugin with AWS Cloud Trail
1. Download the Splunk App for AWS from the link http://apps.splunk.com/app/1274/
2. Log into Splunk Enterprise and select the Manage Apps from the App menu
3. Click Install app from the file button.
4. On the Upload a file screen and Browse to locate the downloaded [app].tar.gz file, click Open and Upload
5. Click Restart Splunk, confirm to restart.
6. After Splunk restarted, login again, you can find the below screen.
7.From Settings in the top right corner of the page Select Data input, In that you can see an option to add AWS CloudTrail Log settings.
8. Click Add new and you will be asked for the following mandatory details.
  • AWS Access key
  • AWS Secret key
  • AWS SQS name
  • AWS SQS region
SQS Creation for topics
In order to use Splunk App, CloudTrail logging should be configured to send SNS topic and that topic should point to the Amazon SQS queue. You can see cloudtrail-spunkalert SNS topic is subscribed to the SQS queue Cloud Trail’s endpoint.
SQS queue should be created manually with the proper permissions to process messages. We have created a queue named CloudTrail in SQS. Below screenshot details the CloudTrail SQS queue:
The above step completes the configuration.

What we can monitor and review using CloudTrail + Splunk ?
The Splunk App for AWS has built in dashboard with lot of options to review Network Actions, IAM Actions, EC2 options etc. Now lets go through some of the common review activities the IT operations/ Managed Services team can perform using the AWS CloudTrail and Splunk Integration:
Case 1: Reviewing AWS Security Group related activities using AWS CloudTrail and Splunk:
From the above Screen shots , we can infer that a new Security group was created and authorized. The Audit details pertaining to Security group + additional details like IAM User, AWS Account, Source IP address and Time stamp of this activity is captured by AWS CloudTrail and presented in Splunk. The Operations/ Managed Services team can constantly monitor for such changes in the infrastructure and take actions accordingly.

Case 2: Reviewing Amazon EC2 related activities using AWS CloudTrail and Splunk:

From the above Screen shots , we can infer that Start, Stop and Terminate EC2 Instance activities are carried out in the AWS Account in the specified period. The EC2 Instance actions are noted with IAM User name and time stamp for monitoring purpose.
Case 3: Rare errors pertaining the AWS account can be reviewed using AWS CloudTrail and Splunk:

Case 4: Reviewing AWS IAM related activities using AWS CloudTrail and Splunk:
AWS IAM Activities like CreateUser, DeleteUser, DeleteLoginProfile etc are tracked by AWS CloudTrail as logs. These activities are presented to the IT Ops/MS team on the IAM actions module of Splunk. The diagrams representing the same is listed below:


Case 5: Reviewing Amazon ElasticIP related activities using AWS CloudTrail and Splunk:
From the above screen shot, We can infer that One EIP Allocate Address activity and One EIP Release Address 107.20.189.161 is carried out during the specified period. The Action, Time stamp , IAM user Name and AWS Account details are captured and can be used for audit purpose.

Case 6: View all the events occurred on your AWS Account as Graph



Case 7: Query your AWS CloudTrail logs for viewing activities and patterns

Splunk app provides the capability to query the AWS CloudTrail logs for viewing specific or overall actions carried out during specified period. This is a powerful feature used by operations/managed services team to view the events and take actions accordingly. 

Please talk to us to understand more about our Managed Services offerings and how we can help your cloud infrastructure. Contact

This article was co authored with Senthil.

Need Consulting help ?

Name

Email *

Message *

DISCLAIMER
All posts, comments, views expressed in this blog are my own and does not represent the positions or views of my past, present or future employers. The intention of this blog is to share my experience and views. Content is subject to change without any notice. While I would do my best to quote the original author or copyright owners wherever I reference them, if you find any of the content / images violating copyright, please let me know and I will act upon it immediately. Lastly, I encourage you to share the content of this blog in general with other online communities for non-commercial and educational purposes.

Followers