AWS CloudTrail and Splunk for better Cloud IT Operations

In any Cloud IT operations, gaining visibility into user activities is important for achieving compliance and meeting the regulatory standards of the enterprise. For years in AWS, we used to solve this with a combination of tools for log collection, alerts and privileged Identity access management. At times, it becomes cumbersome to install and manage these tools and overall operation cost effectively on Amazon Cloud. Like any other company pioneering managed services in AWS infrastructure, Our Cloud operations was in touch with Amazon Cloud product team for years giving input in this area. As an outcome of this, we see a new product line called as AWS Cloud Trail launched last year.

AWS Cloud Trail provides a record of AWS API calls made on your AWS account. AWS CloudTrail can be used for Resource Life Cycle Tracking, Operational Troubleshooting, Compliance Aid, and Security Analytics on AWS cloud infrastructure by integrating with the analytics tools like Splunk, Loggly, Sumo logic and etc.

In this article we are going to explore topics like :

• How AWS CloudTrail can be integrated with Splunk App for AWS ?
• How can we use it to operate our managed services efficiently ( with sample use cases answering following questions)
• How do i get to know when security groups or Network details are changed ?
• How do i know when EC2 instances are launched, terminated etc ?
• How do i know what activities are carried on the Amazon Elastic IP's ?
• What activities and changes happened in the IAM policies over the period ?
• What are the Rare errors related to AWS account activities that can be tracked ?

How AWS CloudTrail + Splunk Integration works ?

Currently CloudTrail logs API calls on the following AWS services and in future you can expect more services to be launched. The current list is as follows :

·      Elastic Compute Cloud (EC2)

·      Elastic Block Store (EBS)

·      Virtual Private Cloud (VPC)

·      Relational Database Service (RDS)

·      Identity and Access Management (IAM)

·      Security Token Service (STS)

·      Redshift

·      CloudTrail

AWS CloudTrail captures AWS API calls made by or on behalf of an AWS account. Once the API calls are logged on files, it delivers these log files to an Amazon S3 bucket that you specify. Each call detail is logged in JSON format. Usually the JSON file is zipped and delivered to the bucket. SNS notification can be enabled to operations team when the log file is delivered.

Sample format of the log file nomenclature is shown below:

https://s3-us-west-2.amazonaws.com/<bucket_name>/<prefix_name>/AWSLogs/<AccountID>/CloudTrail/

us-west-2/2013/10/25/<AccountID>_CloudTrail_us-west-2_2014-11-25T12:25Z_bPlKAoJtDqSWWlTT.json.gz

How to activate AWS CloudTrail ?

By default CloudTrail is not enabled in your AWS account. To activate follow the below steps

1. Open the CloudTrail console in the AWS Management console

2. Click Get Started

3. On the Turn on CloudTrail page, you will be presented with the option to create a new Amazon S3 bucket or using an existing Amazon S3 bucket. In this article, I have selected the option to create new S3 bucket and provided the Bucket name as cloudtraillogs-splunk as illustrated below.

4. If you need to configure SNS notification and other settings, Click the option “Advanced”. Once clicked, below options will be displayed and these options are optional and not mandatory. I have enabled the option of SNS notification for every log delivery to our operations team.

Log file prefix: Enter a prefix for the Amazon S3 bucket. It is usually not needed since AWS automatically prefixes path automatically. You can see that path when you select the “View log file location”. But if you want Pattern for identification you can go ahead with the prefix.

Select Yes or No to “Include global services”?  IAM and AWS STS are the global services so if you want to log activities of those services as well, please select “Yes”. It is recommend to Select them because you can review the actions of the IAM.

SNS notification for every log files delivery?  If you want delivery, please enter the SNS topic name. The entered topic will be created in the SNS service and you must subscribe to it. After Saving, Click Subscribe. Once it is done, the service is activated and the log files are generated.

5. You can also edit the CloudTrail configuration and Turn off CloudTrail using the console. The console is illustrated in the below screen Shot.

What is Splunk App for AWS ?

Splunk App for AWS is designed to take the data from the AWS CloudTrail and provide the results using built-in dashboards. It is provided as a plug-in. Operations / Managed Services team can view this app and monitor and review some activities efficiently.

What is interesting with Splunk App for AWS?

• It gives important insights into security-related activity such as unauthorized access attempts, simultaneous logins from geographically disparate locations and frequent changes to access control privileges. You can ensure security and compliance with continuous monitoring and a full audit trail of user activity using the Splunk App.
• It provides option to configure email alerts based on the schedule
• It provides easy option to monitor events, generate reports and export the reports and Search for events.

Steps to Integrate Splunk with AWS Cloud Trail :

Stage 1: Installing Splunk on AWS:

In order to use the Splunk App for AWS for this article, we are using the Enterprise Splunk AMI for illustrating the use cases.

1. Launch an EC2 Linux instance and based on the choice of Operating system download the Enterprise Splunk from the link http://www.splunk.com/download?r=header

2. Install the Splunk Enterprise

3. Export the following environmental values and then start the Splunk service.

# export SPLUNK_HOME=/opt/splunk

# export PATH=$SPLUNK_HOME/bin:$PATH$

$ splunk start

Commands to Restart and Check Status are below.

$ splunk restart

$ splunk status

4. On successful start the Spunk will be listening on the port 8000. Access the Splunk using the link http://<EC2 Public DNS>:8000. The below screen will be presented

Stage 2: Installing and Configuring Splunk App for AWS:

Splunk App for AWS fetches the data from the Amazon S3 bucket and provides the report in the Splunk GUI. Following are the steps to integrate Splunk app plugin with AWS Cloud Trail

1. Download the Splunk App for AWS from the link http://apps.splunk.com/app/1274/
2. Log into Splunk Enterprise and select the Manage Apps from the App menu
3. Click Install app from the file button.
4. On the Upload a file screen and Browse to locate the downloaded [app].tar.gz file, click Open and Upload
5. Click Restart Splunk, confirm to restart.
6. After Splunk restarted, login again, you can find the below screen.

7.From Settings in the top right corner of the page Select Data input, In that you can see an option to add AWS CloudTrail Log settings.
8. Click Add new and you will be asked for the following mandatory details.

• AWS Access key
• AWS Secret key
• AWS SQS name
• AWS SQS region

SQS Creation for topics
In order to use Splunk App, CloudTrail logging should be configured to send SNS topic and that topic should point to the Amazon SQS queue. You can see cloudtrail-spunkalert SNS topic is subscribed to the SQS queue Cloud Trail’s endpoint.

SQS queue should be created manually with the proper permissions to process messages. We have created a queue named CloudTrail in SQS. Below screenshot details the CloudTrail SQS queue:

The above step completes the configuration.

What we can monitor and review using CloudTrail + Splunk ?
The Splunk App for AWS has built in dashboard with lot of options to review Network Actions, IAM Actions, EC2 options etc. Now lets go through some of the common review activities the IT operations/ Managed Services team can perform using the AWS CloudTrail and Splunk Integration:

Case 1: Reviewing AWS Security Group related activities using AWS CloudTrail and Splunk:

From the above Screen shots , we can infer that a new Security group was created and authorized. The Audit details pertaining to Security group + additional details like IAM User, AWS Account, Source IP address and Time stamp of this activity is captured by AWS CloudTrail and presented in Splunk. The Operations/ Managed Services team can constantly monitor for such changes in the infrastructure and take actions accordingly.

Case 2: Reviewing Amazon EC2 related activities using AWS CloudTrail and Splunk:

From the above Screen shots , we can infer that Start, Stop and Terminate EC2 Instance activities are carried out in the AWS Account in the specified period. The EC2 Instance actions are noted with IAM User name and time stamp for monitoring purpose.
Case 3: Rare errors pertaining the AWS account can be reviewed using AWS CloudTrail and Splunk:

Case 4: Reviewing AWS IAM related activities using AWS CloudTrail and Splunk:
AWS IAM Activities like CreateUser, DeleteUser, DeleteLoginProfile etc are tracked by AWS CloudTrail as logs. These activities are presented to the IT Ops/MS team on the IAM actions module of Splunk. The diagrams representing the same is listed below:

Case 5: Reviewing Amazon ElasticIP related activities using AWS CloudTrail and Splunk:

From the above screen shot, We can infer that One EIP Allocate Address activity and One EIP Release Address 107.20.189.161 is carried out during the specified period. The Action, Time stamp , IAM user Name and AWS Account details are captured and can be used for audit purpose.

Case 6: View all the events occurred on your AWS Account as Graph

Case 7: Query your AWS CloudTrail logs for viewing activities and patterns

Splunk app provides the capability to query the AWS CloudTrail logs for viewing specific or overall actions carried out during specified period. This is a powerful feature used by operations/managed services team to view the events and take actions accordingly. 

Please talk to us to understand more about our Managed Services offerings and how we can help your cloud infrastructure. Contact

This article was co authored with Senthil.