AWS Cost Saving Tip 16: Periodically remove your unwanted AWS resources and save costs

Thanks "@HighScalability.com"  for referencing this post "Stuff The Internet Says On Scalability For October 11th, 2013"

Following are some of the AWS assets that needs to be periodically reviewed for under utilization and should be removed as part of your IT process to avoid cost leakage in AWS. 

Remove unwanted AMI's: Periodically audit for unwanted AMI's in your dev/test/production environment and remove them. Tagging the AMI's with proper identifiers can help you during the cleansing process.

EBS snapshots are incremental in nature and only the changes are moved to them. The unwanted snapshots should be periodically identified (Tagging snapshots will help you) and deleted as part of your IT Infrastructure operations. EBS snapshots are priced $0.095 per GB-month of data stored and you can save few hundred $$$ depending upon the volume of unwanted snapshots stored.To know more about EBS Snapshots

EBS Volumes: In development and testing environments of large enterprises we can often find lots of EBS volumes provisioned and lying unattached or unused. EBS Volumes are priced at $0.10 per GB-month of provisioned storage. Usually 10-30 GB are the most used volume ranges for the dev/test. Imagine a enterprise having few hundred EC2's for their Dev/Test infrastructure and 10-25% of their storage are unused, Periodically identifying , tagging, consolidating and cleansing them will save few hundred to thousands of $$$ for the enterprises.To know more about EBS Volumes

S3 Buckets and Objects: Unused objects in S3 buckets have to be frequently deleted. Objects with a limited life period can be configured to expire dynamically using S3 object expiration policies. Objects that are rarely used can be archived to Amazon Glacier from S3. This exercise is an important candidate for Media and online companies using AWS. To know more about other S3 Cost saving tips 1 | 2.

Elastic IP is a scarce resource in AWS cloud, in event the EIP's attached to your account are not associated with any running EC2 instance, you will be charged $0.005 per hour on a prorata basis.
EIP's can usually be leaked in: 

• Dev/Test environments, where the developer disassociates the EIP and keeps it unused.
• In production, where sometimes EIP are assigned to Auto Scaled EC2 instances using scripts and they are not properly released back after EC2 instances are terminated by AWS.

Though if the EIP is kept unused can lead to around only ~$3.5 leakage per month, if no proper tracking procedures are in place many such EIP's can be residing in your account unused and lead to more such leakages.

ELB: Unused and Unwanted Elastic Load Balancers should be removed periodically to avoid cost leakage. A single ELB can cost around ~216 USD if kept unused a year. Usually ELB leakage is prone to happen in Dev/Test environments compared to production. To know more about Amazon ELB

Monitoring : Detailed monitoring option of CloudWatch is generally not required for Development EC2 instances. On the other hand, they are required on Load Testing and Production environments for monitoring the health. Custom Metrics are also good candidates for Production and Load Test environments. Constant review of the infrastructure in AWS has to be done to identify whether such resources(over provisioning) are associated with Dev/Test env and related cost leakages can be avoided.Detailed monitoring costs $3.50 per instance per month, provided at 1-minute frequency and Custom Metrics costs $0.50 per metric per month.

Glacier archives : Amazon Glacier is designed for use cases where data is retained for months, years, or decades. But there are times where you may find some data life has expired and they can be removed. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. Please ensure it is part of you IT process. To know more about other Cost savings in Glacier 1 | 2 | 3

Other small stuff :

• AWS charges $1.00 per month for inactive data pipelines. They should be periodically reviewed for leakages.  

Note:

• There are times where an AWS asset can be created in a non frequently used AWS Region mistakenly by a developer and it can be lying idle for months. Have an automated or manual IT process to periodically cleanse the above leakages points in "ALL" AWS Regions and not just your most frequently used regions.
• Many times on large AWS deployments/Dev/Test environments, it becomes difficult to track who created and who is currently using the AWS resources. AMI's, EBS snapshots, Volumes and S3 buckets should be tagged for proper identification. Identifiable names should be kept for other assets (wherever applicable) , so that it helps your IT team during the cleansing process. 
• Employ automation using scripts/programs or use governance tools ( which i will be discussing later) to monitor under utilization of resources and there by remove them.

Other Cost Saving Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

AWS Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage

Tip 1: Use Multi-part upload and range retrieval:
You store data in Amazon Glacier as an archive. An archive is any object, such as a photo, video, or document that you store and usually represents a single file or several files. It is a base unit of storage in Amazon Glacier. You can upload an archive in a single request. For large archives, Amazon Glacier provides a multipart upload API that enables you to upload an archive in parts and enables range retrieval in future.
In reality many times, you may need to retrieve a small selection of those files from a archive, in which case you could retrieve only the ranges of the archive that contained the required files.You may also choose to perform range retrievals in order to reduce or eliminate your retrieval fees. If you exceed your free retrieval allowance, you pay a retrieval fee that is based on your peak retrieval rate. Spreading out a retrieval of an archive in smaller parts could therefore allow you reduce your retrieval fees, by reducing your peak retrieval rate.

Tip 2: Spread your retrievals and avoid cost leakage:
Let’s assume you are storing 75 TB of data in Amazon Glacier and you would like to retrieve 140 GB. The amount you pay is determined by how fast you retrieve the data. 
Option 1: Storing 75 TB and retrieving 140 GB in 4 hours costs $21.60.  
Option 2: Storing 75 TB and retrieving 140 GB in 8 hours costs $10.80 (half the cost of option 1)
Option 3: Storing 75 TB and retrieving 140 GB evenly over 28 hours costs NIL because you would no longer exceed your daily free retrieval allowance and would therefore not be charged a Retrieval Fee. Imagine how much cost is leaked if option 1 is followed frequently instead option 3 where the billing is lower or free most times. This can be avoided by better understanding of Amazon Glacier, planning your retrieval in advance with proper spread.

In Amazon Glacier, the suggested practice to lower costs is to plan your retrieval strategy in advance by spreading out the retrievals over longer periods of time. This will reduce or eliminate the retrieval fees and lowers your billing.
Use the aggressive retrieval mechanisms only on use cases where timing is important, Example : You have an old archived news footage in Amazon Glacier, and suddenly based on the current event this old new footage needs to be retrieved, related and delivered. In this case you can use peak retrieval rate and other times you can spread your retrieval strategy.

To explore Log Analysis and Archive with Amazon S3 and Glacier. Refer this detailed article series.

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

AWS Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage

Amazon Glacier is designed for use cases where data is retained for months, years, or decades. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee.

Example : In the US East Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months. So if you deleted 1 gigabyte of data 1 month after uploading it, you would be charged a $0.02 early deletion fee. If, instead you deleted 1 gigabyte after 2 months, you would be charged a $0.01 early deletion fee.

Avoid Cost leakage:
Imagine your IT process does not have this important point considered and assume they delete ~10 TB within 3 months every quarter from Amazon Glacier Archives, they are unnecessarily leaking ~2000 USD cost to AWS every quarter. Instead plan your deletion strategy well in advance build your IT process around this Cost constraint and delete items only after 3 months and avoid this unnecessary leakage.
(Note: Storing rarely accessed archives in Amazon Glacier and deleting early is still cheap compared to storing 10 TB in Amazon S3 and paying ~8K USD bill per month.)

To explore Log Analysis and Archive with Amazon S3 and Glacier. Refer this detailed article series.

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

AWS Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs

Amazon S3 is a storage service that is widely used for storing frequently accessible data like documents, images, videos, log files etc. It is designed for use cases demanding low latency and frequent access. Once the assets grow over a period of time, S3 becomes a costly storage option to the enterprises for rarely accessed items. Such rarely accessed items can be moved to a durable, stable archive systems for saving costs. Some of the use cases for archives are :

• Media assets like news footage, movies, HD content can grow to tens or hundreds of petabytes over years.  Old Archived footage sometimes can become valuable based on current global events and access is needed only during that time.
• Enterprises need to archive data like email, legal records, financial documents etc for complying to their their regulatory and business needs. They are needed only during audits. 
• Organizations like libraries, historical societies, non-profits, governments are increasing their efforts to preserve valuable but aging digital content and artifacts that are no longer readily available. These archive efforts can grow to petabytes over time.

Since Archives do not require frequent access or low latency,for savings costs in Amazon Web Services Infrastructure, you can access real time data from S3 and move rarely retrieved data to Cold storage data archival solutions like Amazon Glacier. 

To explore Log Analysis and Archive with Amazon S3 and Glacier. Refer this detailed article series.

How does it work ?
You can automatically move the objects from S3 to Amazon Glacier using S3 object life cycle rules. Objects which needs to archived to Glacier are configured with object prefixes, relative or absolute time specifier and time periods. Amazon S3 will go through the life cycle policies every day and moves these objects which are qualified to Amazon Glacier. Since the objects are moved from S3 to Glacier, it will be deleted in S3, but the index entry will remain as-is.

What tools can be used ?
You can additionally use the following tools/methodologies to take advantage of this functionality:

• Latest release of CloudBerry Explorer freeware comes with the feature: http://www.cloudberrylab.com/blog/how-to-archive-data-from-amazon-s3-to-glacier-with-cloudberry-explorer/
• Use AWS SDK (Java/.Net based) to access the Amazon Glacier.
• Use Fast Glacier is free for personal use while a commercial licence is also available.  http://fastglacier.com/

Cost Savings ?

<content under progress>

Other Tips

Cost Saving Tip 1: Amazon SQS Long Polling and Batch requests
Cost Saving Tip 2: How right search technology choice saves cost in AWS ?
Cost Saving Tip 3: Using Amazon CloudFront Price Class to minimize costs
Cost Saving Tip 4 : Right Sizing Amazon ElastiCache Cluster
Cost Saving Tip 5: How Amazon Auto Scaling can save costs ?
Cost Saving Tip 6: Amazon Auto Scaling Termination policy and savings
Cost Saving Tip 7: Use Amazon S3 Object Expiration
Cost Saving Tip 8: Use Amazon S3 Reduced Redundancy Storage  
Cost Saving Tip 9: Have efficient EBS Snapshots Retention strategy in place 
Cost Saving Tip 10: Make right choice between PIOPS vs Std EBS volumes and save costs 
Cost Saving Tip 11: How elastic thinking saves cost in Amazon EMR Clusters ? 
Cost Saving Tip 12: Add Spot Instances with Amazon EMR 
Cost Saving Tip 13: Use Amazon Glacier for archive data and save costs (new)
Cost Saving Tip 14: Plan your deletion in Amazon Glacier and avoid cost leakage (new)

Cost Saving Tip 15: Plan your Amazon Glacier Retrieval strategy and avoid cost leakage (new)

Log Analysis and Archive with Amazon S3 and Glacier - Full Summary

Logging is an essential component of any system and helps you understand what's really going on in the system. Just like how you build systems that can scale, perform performance tweaks, design caching architecture, logging is an area that requires that special care to effectively collect logs and make some meaning out of it.

In the Cloud, and more specifically in AWS, there are numerous options and considerations with respect to logging such as

• What are the different sources from where you can collect logs
• How do you collect logs from a dynamic infrastrucuture
• How effectively logs can be collected without affecting the performance of the system
• What are the different storage options available
• And most importantly how one can do it cost effectively

When I set to write on this, I understood that it is going to be a lengthy article with many areas being covered. And logging is an area whose importance is understood only when things go wrong. Otherwise it is pretty boring :) So I decided to split my thoughts in to multiple posts and had been writing about it for the past one month. So this post is a summary of all those different posts.

The Introduction - this is the introductory post setting the context of the different areas that we are going to cover as part of this multi-part post

Part I - in this part, we define the log structure and look at how to collect logs from Amazon CloudFront, the Content Distribution Network service from AWS

Part II - this post describes on how to use the local storage of the EC2 Instance for logging

Part III - part III discusses on how to collect from multiple instances that are dynamically provisioned, how to rotate the log files and store them in a centralized log storage

Part IV - In this final post, we look at what different storage options are available  for cost effective logging, how one can use Glacier, the archival service from AWS, the best practices that one needs to remember and a list of third party / commercial log management solutions available in the market

I hope this of some use to you and provides some insights on logging in AWS. I would definitely like to hear any comments and alternative approaches towards this.

This blog article is syndicated from original article written at http://www.raghuramanb.com/2013/02/log-analysis-and-archive-with-amazon-s3-aws-glacier.html

You can connect with author@ in.linkedin.com/in/raghuramanbalachandran

Part 4: Log Archival & Analysis with Amazon S3 & Amazon Glacier

Log archive & analysis with Amazon S3 and Glacier - Part IV

This blog article is syndicated from original article written at http://www.raghuramanb.com/2013/01/aws-log-archive-amazon-s3-glacier-part-4.html

You can connect with author@ in.linkedin.com/in/raghuramanbalachandran

We now have the logs coming from CloudFront, Web/App and Search tier to the centralized log storage in Amazon S3. In this final post of this series, let's now see what are the options at storage level from cost point of view and what to do with mountains of logs.

Using Reduced Redundancy Storage
Amazon S3 has different storage class - Standard, Reduced Redundancy Storage (RRS) and Glacier. By default when we create store any Object in Amazon S3, it is stored under the Standard storage class. Under "Standard" storage class, all Objects have 99.999999999% durability and 99.99% availability of objects over a given year. With RRS, the Objects that are stored in S3 are replicated at fewer locations to give 99.99% durability and 99.99% availability of objects over a given year. RRS comes cheaper than Standard storage. If we are storing 1TB of log files under Standard storage, it would cost about $95/month (in US-Standard region). Under RRS the same 1TB of storage would be $76/month.

The RRS option cannot be enabled at bucket level but rather at individual Objects level. We can enable RRS for the logs folders that we created through the Object properties

Enable S3 Reduced Redundancy Storage

Log Analysis

We can now initiate Elastic Map Reduce jobs to process these log files and produce log analytics. Elastic Map Reduce takes a S3 bucket as the input source location. We can point the "log" bucket as the input source and supply a Map-Reduce implementation to EMR to crunch the logs.

Yearly Analysis / Multi Year Analysis

Certain requirements want an on-off analysis to be performed at the end of an year. For example, we may perform monthly or on-demand analysis of the log data regularly. And at the end of an year we may require an analysis against the entire year's data and compare it with previous years. For such cases, if we maintain multi-year log files in S3, the cost of storage might be very high. And previous year log files will be accessed only once in an year. For such reasons, we can archive the older log data to Amazon Glacier. Amazon Glacier provides low cost archival service for $0.01 per GB per month.

Archiving to Glacier
We will not be storing the log files forever. Typically any application will have a requirement to store log files for certain period of time beyond which they can be deleted. Let's say that we are interested in retaining only last 6 month's log files. And occasionally we might be doing one year or three years analysis. In such cases, we can use set Lifecycle policies in S3 to automatically archive to Glacier beyond a certain period of time. We can also instruct S3 to automatically deleted Objects beyond a certain period of time.

• Click on the bucket properties and navigate to the "Lifecycle" tab
• Click on "Add Rule" to create a new "Lifecycle Rule"
• Specify that the rule needs to apply for the entire bucket and create a "Transition" and "Expiration" rule
• Create a "Transition" rule specifying "180" days. This will automatically move files from the S3 bucket to Glacier after 180 days
• Create an "Expiration" rule specifying "1095" days. This will delete the log files automatically from S3 or Glacier after 3 years

Lifecycle Rules to Archive to Glacier and Delete Log Files

With that the log files will get automatically archived to Glacier after 6 months (from creation) and will be deleted after 3 years. Once the log files are archived to Glacier, the storage class of these log files (objects) in S3 changes to Glacier indicating that they are being stored in Glacier.

S3 Storage Class for archived files

Restoring from Glacier

For our year end analysis, we will need the archived data in Glacier back in Amazon S3 so that we can run Elastic Map Reduce jobs against them to produce our year-end / multi-year analytics information.  We can do this through the AWS Management Console by

• Right clicking the particular object (log file) whose storage class is Glacier (meaning it is archived) and "Initiate Restore"
• Specify, how long we require the Object in S3 for us to perform the analysis and complete the request

Once this request is initiated, it normally takes around 3-4 hours for AWS to restore the object from Glacier to S3. The "Object Restoration" process can be done only at an individual object level. We will normally have large number of log files at the end of an year, and doing this way is not practically possible.

Restoring from Glacier programmatically

Restoring from Glacier is essentially a S3 operation and not a Glacier operation as it seems to be. We need to use the S3 API to initiate restoration

AmazonS3 s3Client = new AmazonS3Client(new BasicAWSCredentials("aws-access-key", "aws-secret-access-key"));
ObjectListing listing = s3Client.listObjects(new ListObjectsRequest()
             .withBucketName("my-global-logs")
             .withPrefix("web-logs/")); 

The first step is to list all the keys of the Objects that we want to restore. To do this, use the S3 ListObjects API call to list all the Objects. Few pointers while using this API

• Specify the bucket name that we want to list. Also include a prefix if we are interested in restoring only a specific directory within that bucket. For example, if we are interested only in performing analysis against the web-logs and not others, we can specify as indicated above
• Since a bucket can contain 1000's of Objects, S3's API does pagination when sending the response. Hence use the "isTruncated" method in the response "ObjectListing" to check if there are more Objects. If so, initiate further API calls to list till the end
• Since we are listing the entire bucket, the call will result in keys for the directory also. Something like the following. Hence check for the key containing a file instead of a directory and keep adding such keys to a list (like performing a simple 'contains(".log")' check)

          web-logs/2012/

          web-logs/2012/12/

          web-logs/2012/12/10/

          web-logs/2012/12/10/i-7a3flcd3/

          web-logs/2012/12/10/i-7a3flcd3/tomcat.log

          web-logs/2012/12/10/i-9d9dedf2/

          web-logs/2012/12/10/i-9d9dedf2/01/

          web-logs/2012/12/10/i-9d9dedf2/02/

          web-logs/2012/12/10/i-9d9dedf2/03/

Once we have entire list of Object Keys to restore, the next step is to initate the restore process for all the Objects

RestoreObjectRequest requestRestore = new RestoreObjectRequest("my-global-logs", "<object-key>", <restoration-period>);
s3Client.restoreObject(requestRestore);

Once the above request is initiated for all the Objects, Amazon Glacier takes about 3-5 hours to restore the Objects and make it available in Amazon S3. We can then run Elastic Map Reduce jobs with all the required data.

Things to remember / consider

• Archiving and Restoring are S3 operations and hence are part of S3 API
• If you have data stored in Glacier that weren't archived from S3, then to restore them, you should use the Glacier API to initiate downloads. See the steps outlined in AWS documentation fordownloading an archive
• Restored objects by default are stored under "Reduced Redundancy Storage"
• If you have millions of Objects in S3 that has to be transitioned to Glacier, be aware of the cost of restore requests. Eric Hammond has put across a very detailed analysis here
• Glacier is designed for Archival Storage. Meaning, you do not access the data frequently and can wait for accessing the data. Any download request from Glacier, will take 3-5 hours before it is available. Hence carefully choose the archival policy. If you plan to retrieve the log data frequently, Glacier will not be right choice and will prove to be very expensive (since it is not designed for frequent retrieval)

Log Management Solutions

There are plenty for log management solutions that are available as a service and can be plugged in to existing applications and cloud environment.

• Splunk is a widely used log management and monitoring solution. Splunk can be setup on a server and can be easily configured to start collecting data from web servers. A SaaS version is also available where the service is completely managed by Splunk
• Loggly is another cloud based log management solution that is available as a service
• There are also open source solutions available such as LogStashthat can be customized for our needs

That brings to the end of this series on what I wanted to cover as part of log analysis using S3 and Glacier. Logging is an essential component in any system and in the era of Cloud Computing, a good log management solution will prove handy. Once the problems of scale and performance gets sorted out with the help of Cloud Computing, the immediate next need of any system would be to have an effective way to look at the system and analyse at scale. A log management solution will definitely prove handy.

Part 3: Log Archival with Amazon S3 and Amazon Glacier

Log archive & analysis with Amazon S3 and Glacier - Part III

This blog article is syndicated from original article written at http://www.raghuramanb.com/2012/12/aws-log-archive-amazon-s3-glacier-part-3.html

You can connect with author@ in.linkedin.com/in/raghuramanbalachandran

A recap from the previous posts:

• We had seen how to configure CloudFront to enable access logs. CloudFront will directly push all access logs to a configured S3 bucket
• We saw what the considerations are for logging at the web tier - such as local log storage, dynamic environment. We configured the EC2 Instances to use the Ephemeral Storage for storing the log files locally

In this post, we will see how to push logs from the local storage to the central log storage - Amazon S3 and what the considerations are.

Amazon S3 folder structure

The logs are going to be generated throughout the day and hence we need to have a proper folder structure to store them in S3. Logs will be particularly useful to perform analysis such as a production issue or to find a usage pattern such as feature adoption by users/month. Hence it will make sense to store them by year/month/day/hour_of_the_day structure.

Multiple Instances
The web tier will be automatically scaled and we will have multiple Instances always running for High Availability and Scalability. So, even if we have logs stored hourly basis, we will be having log files with similar names from multiple Instances. Hence the folder structure needs to factor multiple Instances as well. The resultant folder structure in S3 will look something like this

Amazon S3 Log Folder Structure

Note in the above picture (as encircled) that we are storing "Instance" wise logs for every hour.

Log Rotation
Every logging framework will have an option to rotate the log files on size, date, etc...We will be periodically pushing the log files to Amazon S3 and hence it might make sense to say, rotate the log file every hour and push it to S3. But the downside to that is, we cannot anticipate the traffic to the web tier and that's the reason we have the web tier scaling automatically on demand. If there is a sudden surge in the traffic which may result in large log files generated, it will start filling up the file system eventually making the Instance unavailable. Hence it is better to rotate the log files by size.

Linux-logrotate
You can use the default logrotate available in Linux systems to rotate the log files on size. Logrotate can be configured to call a post script after the rotation to enable us push the newly rotated files to S3. A sample logrotate implementation will look like this:

Note: If you are using logrotate, make sure your logging framework isn't configured to rotate

/var/log/applogs/httpd/web {
        missingok
        rotate 52
        size 50M
        copytruncate
        notifempty
        create 644 root root
        sharedscripts
        postrotate
                /usr/local/admintools/compress-and-upload.sh web &> /var/log/custom/web_logrotate.log
        endscript
}

The above set of commands rotate the "httpd" log files whenever the size reaches 50M. It also calls a "postrotate" script to compress the rotated file and upload it to S3.

Upload to S3
The next step is to upload the rotated log file to S3.

• We need a mechanism to access the S3 API from the shell to upload the files. S3cmd is a command line tool that is widely used and recommended for accessing all S3 APIs through the command line. We need to setup the CLI in the Instance
• We are rotating by size but we will be uploading to a folder structure that maintains log files by the hour
• We will also be uploading from multiple Instances and hence we need to fetch the Instance Id to store in the corresponding folder. Within EC2, there is an easy way to get Instance meta data. If we "wget" "http://169.254.169.254/latest/meta-data/" it will provide the Instance meta-data such as InstanceId, public DNS, etc.. For example if we "wget" "http://169.254.169.254/latest/meta-data/instance-id" we will get the current Instance Id

The following set of commands will compress the rotated file and upload them into the corresponding S3 bucket

# Perform Rotated Log File Compression
tar -czPf /var/log/httpd/"$1".1.gz /var/log/httpd/"$1".1

# Fetch the instance id from the instance
EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`"
if [ -z $EC2_INSTANCE_ID ]; then
echo "Error: Couldn't fetch Instance ID .. Exiting .."
exit;
else
        # Upload log file to Amazon S3 bucket
        /usr/bin/s3cmd -c /.s3cfg put /var/log/httpd/"$1".1.gz s3://$BUCKET_NAME/$(date +%Y)/$(date +%m)/$(date +%d)/$EC2_INSTANCE_ID/"$1"/$(hostname -f)-$(date +%H%M%S)-"$1".gz
fi
# Removing Rotated Compressed Log File
rm -f /var/log/httpd/"$1".1.gz

Now that the files are automatically getting rotated, compressed and uploaded to S3 there is one last thing to be taken care of.

Run While Shutdown

Since the web tier will automatically scale depending upon the load, Instances can be pulled off (terminated) when load decreases. During such scenarios, we might be still left with some log files (maximum of 50MB) that didn't get rotated and uploaded. During shutdown, we can have a small script, that will forcefully call the logrotate to rotate the final set of files, compress and upload.

stop() {
echo -n "Force rotation of log files and upload to s3 intitiated"
/usr/sbin/logrotate -f /etc/logrotate.d/web
exit 0
}

Use IAM

We need to provide Access Key and Secret Access Key to the S3cmd utility for S3 API access. Do NOT provide the AWS account's Access Key and Secret Access Key. Create an IAM user who has access to only the specific S3 bucket where we are uploading the files and use the IAM user's Access Key and Secret Access Key. A sample policy allowing access for the IAM user to the S3 log bucket would be

{
  "Statement": [
    {
      "Sid": "Stmt1355302111002",
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::"
      ]
    }
  ]
}

Note:

• The above policy allows the IAM user to perform all actions on the S3 bucket. The user will not have permission to access any other buckets or services
• If you want to restrict further, instead of allowing all actions on the S3 bucket, we can allow only PutObject (s3:PutObject) for uploading the files
• Through the above approach, you will be storing the IAM credentials on the EC2 Instance itself. An alternative approach is to use IAM Roles so that the EC2 Instance will obtain the API credentials at runtime

With that we have the web tier log files automatically getting rotated, compressed and uploaded to Amazon S3 and stored in a central location. We have access to log information by the year/month/day/hour and Instance-wise.

Part 2: Log archival with Amazon S3 and Amazon Glacier

Log archive & analysis with Amazon S3 and Glacier - Part II

This blog article is syndicated from original article written at http://www.raghuramanb.com/2012/12/aws-log-archive-amazon-s3-glacier-part-2.html

You can connect with author@ in.linkedin.com/in/raghuramanbalachandran

In the previous post, we saw how to configure logging in AWS CloudFront and start collecting access logs. Let's now move on to the next tier in the architecture - web/app. The following are the key considerations for  logging in web/app layer:

• Local Log Storage - choosing the local storage for logging. Using a storage option that is sufficient, cost-effective and meets the performance requirement for logging
• Central Log Storage - how do we centrally store log files for log analysis in future
• Dynamic Infrastructure - how do we collect logs from multiple servers that are provisioned on demand

Local Log Storage

Except very few cases, EBS-backed Instances are the most sought after Instance type. They launch quickly and easier to build Images out of them. But they come with couple of limitations from logging perspective

• Limited storage - EBS-backed AMIs that are provided by AWS or third-party providers come with limited storage. For example, a typical RHEL AMI comes with around 6GB of EBS attached as the root partition. Similarly Windows AMI come with 30GB EBS attached as C:\
• Growing EBS - log files tend to grow faster. And it becomes difficult to grow the root EBS (or an additional EBS) as the log file sizes grow
• Performance - any I/O operation on an EBS Volume is over the network. And it tends to be slower than local disk writes. Specifically for logging, it is always better to remove the I/O bottleneck. Otherwise lot of system resources could be spent towards logging

Every EC2 Instance comes with ephemeral storage. These are local storage directly attached to the host on which the Instance is running. Ephemeral storage do not persist between stop-start cycles of an Instance (EBS-backed) but they are available when the Instance is running and persist during reboots. There are couple of advantages of Ephemeral storage:

• They are locally attached on the physical host on which the Instance runs and hence have better I/O throughput when compared to EBS Volumes
• They come in pretty good size - for example a m1.large Instance comes with 850GB of ephemeral storage
• And it comes free of cost - you aren't charged per GB or for any I/O operations on the ephemeral storage unlinke EBS

This makes ephemeral storage the ideal candidate for storing log files. For an EBS-backed Instance, the ephemeral storage is not mounted and readily available. Hence one needs to follow the following steps to start using the ephemeral storage for storing log files

• The logging framework usually comes with a configuration file to configure logging parameters. The log file path needs to be configured to point to the ephemeral storage mount directory that we create below
• All application related files (such as binaries, configuration files, web/app server) will be installed on the root EBS. Before the final AMI is created, the ephemeral storage needs to be setup and configured
• Run fdisk to list all the storage devices that are mounted

fdisk -l

• Created a directory such as "/applogs". This is the directory where the ephemeral storage will be mounted

mkdir /var/log/applogs

• Mount the storage device in this directory using the "mount" command

mount /dev/xvdj /var/log/applogs

• Add "fstab" entries so that the ephemeral storage is mounted in the same directory after stop/start or when new Instances are launched out of this AMI

/dev/xvdj  /var/log/applogs xfs defaults,noatime,nobarrier,allocsize=64k,logbufs=8 0 2
/dev/xvdj /var/log/applogs    ext3    defaults        0   0

The last step is essential especially from AutoScaling point of view. When AutoScaling launches new Instances, the ephemeral storage needs to be automatically mounted in the directory so that the application can start logging. Now, we can go ahead create the final AMI and launch Instances from them. The new Instances will have the ephemeral storage automatically mounted in the "/var/log/applogs" directory and applications can start storing the log files in them.