Part 10: Performance Tuning Amazon Elastic Block Store - IO Block Size

The IOPS rate you get usually depends on the I/O size of your applications’ reads and writes. It is very important to know the I/O size your application operates and accordingly expect the performance from EBS Volumes. Provisioned IOPS volumes process your applications’ reads and writes in I/O sizes of 16KB blocks or less. Every increase in I/O size above 16KB will linearly increase the resources you need to achieve the same IOPS rate. For example, if you have provisioned a volume with 2000 IOPS, that means that it can handle 2000 16KB writes per second, or 1000 32KB writes per second, or 500 64KB writes per second, and so on. You can use Amazon CloudWatch to monitor your EBS Volumes throughput and I/O size for a while and decide what best suits for your case. 

Since PIOPS is especially designed for Database context, let us explore how it pans out in Amazon RDS. 

Following are some of the important points to note on PIOPS + RDS context.

P1) The theoretical maximum IOPS rate is also a function of database I/O page size and available channel bandwidth. Each page read or write constitutes one I/O operation. Database operations that read or write more than a single page will use multiple I/O operations for each database operation. For example, for a page size of 16 KB, if a read operation requires 32 KB, you would need 2000 IOPS to achieve 1000 read operations per second.

P2) If you provision an IOPS rate that is higher than the maximum or that is higher than your realized IOPS rate, you may still benefit from reduced latency and improvements in overall throughput.

Applicability of the following points and IOPS rates are based on the m2.4xlarge instance class with full duplex (1000 Mbit/second) and a workload that is perfectly balanced between reads and writes.

·         MySQL uses a page size of 16 KB and can do 12,500 IOPS. On a DB instance with a full duplex I/O channel bandwidth of 1000 megabits per second (Mbps), the maximum IOPS for page I/O is about 6,250 IOPS in each direction for 16 KB I/O. This is for a workload consisting of 50% reads and 50% writes and could reach 12,500 IOPS with 16 KB I/O.

·         Oracle uses a page size of 8KB and can do 25,000 IOPS. On a DB instance with a full duplex I/O channel bandwidth of 1000 megabits per second (Mbps), the maximum IOPS for page I/O is about 12,500 IOPS in each direction for 8 KB I/O. This is for a workload consisting of 50% reads and 50% writes and could reach 25,000 IOPS with 8 KB I/O.

·         SQL Server uses a page size of 8KB and can do 10,000 IOPS. On a DB instance with a full duplex I/O channel bandwidth of 1000 megabits per second (Mbps), the maximum IOPS for page I/O is about 5,000 IOPS in each direction for 8 KB I/O. This is for a workload consisting of 50% reads and 50% writes and could reach 10,000 IOPS with 8 KB I/O.

Note : Please refer the AWS Documentation for latest PIOPS 

EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store

Part 2: Understanding Standard EBS Volumes

Part 3: Understanding EBS PIOPS Volumes

Part 4: Understanding EBS-Optimized Instances

Part 5: Understanding Latency in EBS

Part 7: 10% of your provisioned IOPS 99.9% of the time

Part 8: Performance Tuning - Pre Warming the EBS volume

Part 9: Performance Tuning - EBS Striping

Part 10: Performance Tuning - IO Block Size

Part 11: Understanding Amazon EBS Snapshots

Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 

Part 13: Amazon EBS Security Best practices and tips

Part 9: Performance Tuning Amazon Elastic Block Store - EBS Striping

Software RAID systems takes number of EBS Volumes and present them as one volume to the user. A RAID 0 (known as a striped volume) splits data evenly across two or more volumes (striped) without parity information. Example: The chunks of data from the same file are distributed across the disks that form the RAID array as illustrated in the below diagram (coming soon)

By writing data in small chunks across several EBS volumes, the performance of those volumes can be aggregated. Since I/O is distributed across the volumes in a stripe manner, if you add a volume in the array, you get the straight addition of throughput as well (most times). For example, where a single PIOPS volume is restricted to its individual I/O performance and disk RPM’s, a write to an array made up of several volumes combines the I/O of all those PIOPS Volumes. So, a stripe made up of six PIOPS volumes with a throughput of 1800 IOPS each would together have a total I/O performance of up to 10800 IOPS. You can create EBS stripes on both Standard and PIOPS EBS Volumes.  Note: Performance of the stripe is limited to the worst performing volume in the set. Usually EBS Striping is chosen in cases where I/O performance is more important than fault tolerance (such as a heavily used database, Search engine servers) and where data replication is already set up separately.

To create a six-volume stripe set on Linux, use a command like the following.

$ sudo mdadm --create /dev/md0 --level=0 --chunk=64 --raid-devices=6 /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk

Why 64 KB RAID chunk size?

The RAID chunk size should suit the I/O characteristics of the data you are working with. Important factor here is the size of the average I/O request you're going to place on the RAID Volume; as general rule; if you want big I/O requests, you should opt for smaller RAID chunk sizes, and if I/O will be small, you should go for larger chunks sizes.

Let me illustrate the chunk sizing with following scenarios:

Scenario 1:  Let us imagine your application works with lots of video or large image files, which means, you will need to spread data across individual volumes as much as possible. For this scenario use smaller RAID chunk sizes (for example, Block size 512 bytes-> 8 KB) because you want to take data from one volume while the others seek the next chunks to be read.

Scenario 2: Let us imagine you are running a database in which the amount of data read on each operation is small (4KB ->16KB). Here you want a single I/O to deal with one volume (drive) with one seek rather than the action is split between more than one volume and multiple seeks. So, for use cases such as databases and exchange servers, you should go for a bigger RAID chunk size say 64 KB or larger.



EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store
Part 2: Understanding Standard EBS Volumes
Part 3: Understanding EBS PIOPS Volumes
Part 4: Understanding EBS-Optimized Instances
Part 5: Understanding Latency in EBS
Part 7: 10% of your provisioned IOPS 99.9% of the time
Part 8: Performance Tuning - Pre Warming the EBS volume
Part 9: Performance Tuning - EBS Striping
Part 10: Performance Tuning - IO Block Size
Part 11: Understanding Amazon EBS Snapshots
Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 
Part 13: Amazon EBS Security Best practices and tips

Part 8: Performance Tuning Amazon Elastic Block Store - Pre Warming the EBS volume

New EBS Volumes always experience a “First Use Penalty” in AWS; means there will be a performance hit on the first write to an unused block on the EBS Volume and will perform slower than subsequent writes. During this time you can experience spike in volume metrics like service times, I/O latency and then subsequently leading to normalization. It is observed that 5 to 50 percent reduction in IOPS when you first access the data on a volume. Performance is restored after the data is accessed once. Therefore, it is recommended that you read or write to all the blocks on your volume before you use it.

$ dd if=/dev/md0 of=/dev/null

In Linux, the above command reads from all blocks on a volume and pre-warms the volume. On Windows, formatting the new EBS volume pre-warms it.

“dd” is not verbose by default so download and use the following script from the github, it shows the status of the pre warming of EBS Volumes, URL:  https://gist.github.com/muhqu/3293988

#!/bin/bash if [ -z "$1" ];   echo "Usage: sudo $0 /dev/sdh1"   exit 1; fi  dd if=$1 of=/dev/null & pid=$! while true; do   ps -p$pid --no-heading || break;   echo "-- $(date) ------------------";   kill -USR1 $pid;   sleep 60s; done echo "-- $(date) ------------------"; echo "DONE \o/"

EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store
Part 2: Understanding Standard EBS Volumes
Part 3: Understanding EBS PIOPS Volumes
Part 4: Understanding EBS-Optimized Instances
Part 5: Understanding Latency in EBS
Part 7: 10% of your provisioned IOPS 99.9% of the time
Part 8: Performance Tuning - Pre Warming the EBS volume
Part 9: Performance Tuning - EBS Striping
Part 10: Performance Tuning - IO Block Size
Part 11: Understanding Amazon EBS Snapshots
Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 
Part 13: Amazon EBS Security Best practices and tips

Part 7: Understanding Amazon Elastic Block Store Performance: 10% of your provisioned IOPS 99.9% of the time

Understanding the 10% statement of PIOPS

“Provisioned IOPS volumes are designed to deliver within 10% of your provisioned IOPS 99.9% of the time”. In simpler terms, it means for a volume provisioned with 2000 IOPS, your volume should deliver at least 1800 IOPS 99.9% of the time. (This is measured based on one minute intervals). When you design EBS Striping (RAID 0) with 6 disks (example 1 TB each with 2000 IOPS) you need to factor 6 X 1800 = 10800 IOPS as the bottom line and not 12000 IOPS.


EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store
Part 2: Understanding Standard EBS Volumes
Part 3: Understanding EBS PIOPS Volumes
Part 4: Understanding EBS-Optimized Instances
Part 5: Understanding Latency in EBS
Part 7: 10% of your provisioned IOPS 99.9% of the time
Part 8: Performance Tuning - Pre Warming the EBS volume
Part 9: Performance Tuning - EBS Striping
Part 10: Performance Tuning - IO Block Size
Part 11: Understanding Amazon EBS Snapshots
Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 
Part 13: Amazon EBS Security Best practices and tips

Part 5: Understanding Amazon Elastic Block Store Performance: Latency

Understanding the latencies

In general, it is impossible for a storage system to sustain the same peak IOPS number when presented with different I/O types and latency requirements.  Only IOPS numbers alone are meaningless without considering additional metrics such as latency, read/write % and I/O block size etc.

Latency is a measure of how long it takes for a single I/O request to happen from the apps perspective. High Latency and variance is not preferred when it comes to database applications. Database makes variety of requests and ideally IO latency needs to be under 10ms and write especially under 5ms. Example: In Oracle DB with for heavy writes, the redo log under 1ms is preferred. On the other hand, applications doing sequential, throughput-driven I/O (like backup or archival) typically don’t need high IOPS, but rather need high MB/s.

PIOPS Volumes delivers consistent level of latency and performance than standard EBS Volumes on random reads and random read/writes.  In the benchmark conducted by parse, PIOPS showed latency between 0-0.6 seconds and Standard EBS volumes showed 0.0 to 2.5 seconds. Refer URL:

http://blog.parse.com/2012/09/17/parse-databases-upgraded-to-amazon-provisioned-iops/

On the other hand, for random writes you should expect to see roughly similar latencies between Standard and Provisioned IOPS volumes, whereas on sequential writes and reads you can observe Standard EBS performing better than PIOPS sometimes. 


EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store
Part 2: Understanding Standard EBS Volumes
Part 3: Understanding EBS PIOPS Volumes
Part 4: Understanding EBS-Optimized Instances
Part 5: Understanding Latency in EBS
Part 7: 10% of your provisioned IOPS 99.9% of the time
Part 8: Performance Tuning - Pre Warming the EBS volume
Part 9: Performance Tuning - EBS Striping
Part 10: Performance Tuning - IO Block Size
Part 11: Understanding Amazon EBS Snapshots
Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 
Part 13: Amazon EBS Security Best practices and tips

Part 4: Understanding Amazon Elastic Block Store Performance: EBS-Optimized Instances

Performance tuning Amazon EBS

The performance of a block storage device is commonly measured and quoted in a unit called IOPS, short for Input/Output Operations per Second (IOPS).

To put the numbers in this post into perspective, a disk drive spinning at 7,200 RPM can perform at 75 to 100 IOPS whereas a drive spinning at 15,000 RPM will deliver 175 to 210. The exact number will depend on a number of factors including the access pattern (random or sequential) and the amount of data transferred per read or write operation etc

Some of the key factors that affect EBS QoS are:

• Bandwidth – The rate of data throughput available
• Latency – The time delay for a read/write operation to execute
• Queue depth – The number of outstanding operations waiting to execute to the underlying disks (Traditional or SSD).

Now let us explore those factors and some solutions for the same.

Bandwidth: Use EBS Optimized Instances:

Usually the Network traffic and EBS traffic is shared in the Amazon EC2 instance, which means consistent EBS performance depends on amount of non-EBS related network you send to and from your instance.  Since you cannot predict this there will be contention between EBS and internet traffic and at times, traffic to EBS can suffer a lot. To solve this problem and improve the performance, AWS introduced EBS-optimized instances.  It is an optimized configuration stack that provides additional and dedicated capacity between EC2 and EBS IO. This optimization minimizes the contention between EBS I/O and other traffic from your Amazon EC2 instance and thereby gives you best and consistent performance. EBS-optimized instances deliver dedicated throughput to EBS, with options between 500 Mbps and 1,000 Mbps, depending on the instance type you use and enables you to effectively use PIOPS volumes (if attached). This is additional throughput, and doesn't affect other general purpose network throughput already available on the instance. EBS-Optimized instances can be used with both standard and PIOPS volumes.

Standard EBS Volumes + EBS-OPT Instances:  Scaling throughput from a number of volume stripes where I/O performance consistency isn't critical.

PIOPS Volumes + EBS-OPT Instances: Performance-sensitive production databases that require dedicated EBS traffic and high IOPS performance. It is recommended to always use PIOPS with EBS-Optimized instances to get the full performance benefits.

Things to note:

• Not all Amazon EC2 instances are EBS-Optimized.  Only the following list is currently supported, please refer AWS documentation for the latest supported types. High-CPU Extra Large (c1.xlarge) ,M1 Large (m1.large), M1 Extra Large (m1.xlarge), High-Memory Double Extra Large (m2.2xlarge), High-Memory Quadruple Extra Large (m2.4xlarge), M3 Extra Large (m3.xlarge) ,M3 Double Extra Large (m3.2xlarge).
• Provisioned IOPS volumes may not be available in every AZ inside a Region where EBS-OPT is available (currently). AWS will sort this going forward. In mean time, you can use an EBS-Optimized instance with Standard EBS volumes as well to get dedicated throughput between your instance and volumes(in such AZ).

EBS Article Series (continued..)

Part 1: Understanding Amazon Elastic Block Store
Part 2: Understanding Standard EBS Volumes
Part 3: Understanding EBS PIOPS Volumes
Part 4: Understanding EBS-Optimized Instances
Part 5: Understanding Latency in EBS
Part 7: 10% of your provisioned IOPS 99.9% of the time
Part 8: Performance Tuning - Pre Warming the EBS volume
Part 9: Performance Tuning - EBS Striping
Part 10: Performance Tuning - IO Block Size
Part 11: Understanding Amazon EBS Snapshots
Part 12: Securing Amazon EBS volumes - EBS Encryption using SecureCloud 
Part 13: Amazon EBS Security Best practices and tips