The long waited AWS CloudWatch Custom Metrics feature has arrived

Amazon Web Services introduced Custom metrics 2 weeks back, according to me it is one of the important features released by them in the recent times. I had earlier written a blog post last September 2010 wishing for a similar feature in CloudWatch and have requested AWS Technical evangelist team about this whenever I met them last year.

Refer URL: http://harish11g.blogspot.com/2010/09/i-wish-it-was-there-in-amazon.html

« If the Amazon CloudWatch can monitor the number of Messages on Amazon SQS and give this input to Amazon AutoScaling , We can configure the Auto Scale to create/remove Amazon EC2 instances according the number of queue messages. I feel it will be very good use case for architects designing solutions around SQS »

From this release, it is again evident that AWS Evangelist and technology team are very receptive to the comments from Infra Architects like us.

Now coming back to monitoring context, let us explore what is Custom CloudWatch Metric and how it can help us?

Custom Metrics feature of CloudWatch allows DEVOPS to customize or create new metrics (both business and application metrics) in AWS CloudWatch. We can extend the monitoring of existing AWS technologies like Amazon EC2, EBS, ELB and RDS by adding new metrics (or) we can add totally new set of business or technical metrics to SW/HS systems in our architecture.

By leveraging the Custom metrics option we can use CloudWatch as a single aggregated storage for all of the metrics that we want to watch and to monitor in the AWS environment. We can use any suitable graphing and visualization tools to consume and present this information. In addition, we can create our own CloudWatch Alarms for the metrics. For example, we can set up an SNS notification to email us, when the number of user registration count has touched 100,000 per day using Custom metrics.

Top 4 Things to remember before using AWS CloudWatch Custom metrics:

1. AWS CloudWatch Custom Metrics is not an installable monitoring product. I feel it is currently suitable for a company which has enough DEVOPs bandwidth to develop and present their metrics. Others can rely on traditional Hyperic/Nagios stack or can take help from AWS System Integrators to have such custom metrics developed.
2. Only .Net, Java, and PHP AWS SDKs are currently supported with Custom Metrics
3. We can store only metrics with following standard units - Seconds, Bits, Bytes, Percent, Count, Bytes/Second, Bits/Second, and Count/Second. Metrics with different units will not be included in CloudWatch statistical calculations.
4. All metrics will be retained only for a period of two weeks in CloudWatch Storage. If we need to analyze historical data for a longer period, we need to move the collected information to other analytics storage systems. This is an added headache.

Some Use Cases relating to CloudWatch custom metrics that i can immediately think of are

• Increase / Decrease the Amazon EC2 processing nodes elastically depending upon the count of messages in SQS
• Increase / Decrease the Amazon EC2 processing nodes elastically depending upon the count of media objects in S3
• Increase / Decrease the Session Sticky Amazon EC2 web/app instances based on the number of Active session in Web/App
• Add more Media or Data File upload / download processing instances based on the upload /download count volatility
• Automatically Resize the RDS storage based on the custom file system usage metric
• Automatically Resize the EBS storage of the Amazon EC2 instances based on the custom file system usage metric
• Generate SNS email alarms on increase in User Registrations count from predicted value
• Generate SNS email alarms for abnormal counts/usage of Database connections, Cache miss etc