Welcome to the SoftNAS Forums.
Results 1 to 2 of 2
  1. #1
    Administrator
    Join Date
    Apr 2012
    Location
    Texas
    Posts
    153
    Blog Entries
    23

    SoftNAS on EC2 - up to 9,873 IOPS, 154 MB/sec throughput and 0.1ms/op response time

    The following benchmarks of SoftNAS on Amazon EC2 show the benefits of combining EC2's latest "Provisioned IOPS" and SoftNAS' ZFS caching and RAID striping - enough IOPS to run demanding SQL, VDI and other small-block random I/O applications in the cloud. Provisioned IOPS provide maximum sustained read/write IOPS with EBS volumes, ensuring you get quality of service and highest possible IOPS and throughput.

    The benchmark shows how random reads and writes of a 5 GB file perform using the "filebench" benchmarking tool. This benchmark is without write-caching (no ZFS write log) or additional SSD caching (L2ARC).

    According to Amazon, "Provisioned IOPS are a new EBS volume type designed to deliver predictable, high performance for I/O intensive workloads, such as database applications, that rely on consistent and fast response times. With EBS Provisioned IOPS, customers can flexibly specify both volume size and volume performance, and Amazon EBS will consistently deliver the desired performance over the lifetime of the volume. Customers can then attach multiple volumes to an Amazon EC2 instance and stripe across them to deliver thousands of IOPS to their application."

    Amazon recently announced last week an increase in the available IOPS... "We are excited to announce support for up to 4,000 IOPS per Amazon EBS Provisioned IOPS volume. This represents a fourfold increase from the original Provisioned IOPS volume performance since its launch last year...You can get up to 4,000 IOPS from one volume. For performance beyond 4,000 IOPS, you can attach and stripe multiple volumes to deliver thousands of IOPS to your application. You can set the level of performance you need and EBS will consistently deliver it over the lifetime of the volume"

    SoftNAS was configured as follows:

    - Installed on EC2 High-Memory Quadruple Extra Large Instance (8 cores and 64 GB of RAM)
    - 2.5 TB storage pool created from five 500 GB EBS volumes with 2,000 provisioned IOPS each
    - Thin-provisioned volume
    - Compression and deduplication disabled
    - 5 x 500 GB EBS volumes configured for RAIDZ1 striping - each volume set for 2,000 provisioned IOPS

    RAID striping across multiple EBS volumes provides the storage system with access to more read IOPS. The RAM cache (ZFS ARC) was approximately 36 GB in size, providing extremely low-latency, high-speed access for random reads. Caching is important, especially on EC2 with EBS volumes, which are backed by 7.2K RPM SATA drive SAN, accessed across a 1 GbE link. The RAIDZ1 configuration optimizes IOPS for reads and increased data integrity (recoverability in case an EBS volume fails to prevent data loss).

    Benchmark results summary and more details below.

    Benchmark Results Summary

    Random reads: rand-read1 592457ops 9873ops/s 154.2mb/s 0.1ms/op 37543us/op-cpu [0ms - 44ms]

    Random read/write:

    rand-write1 45043ops 751 ops/s 11.7mb/s 135.3ms/op 327721us/op-cpu [0ms - 1304ms]
    rand-read1 400845ops 6,680 ops/s 104.3mb/s 0.2ms/op 20785us/op-cpu [0ms - 1255ms]
    Summary: 445888 ops, 7,430 ops/s, (6680/751 r/w), 116.0mb/s, 295us cpu/op, 13.9ms latency

    Sequential read:

    seqread 125975ops 2,099 ops/s 2,097.1mb/s 4.2ms/op 169208us/op-cpu [0ms - 847ms]
    Summary: 125975 ops, 2,099.269 ops/s, (2099/0 r/w), 2097.1mb/s, 2412us cpu/op, 4.2ms latency


    Test Environment:

    High-Memory Quadruple Extra Large Instance

    - 68.4 GiB of memory

    - 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)

    - 64-bit platform

    - I/O Performance: High

    - EBS-Optimized: 1000 Mbps (1 GbE)




    Test result details:

    RANDOM READ (256 threads)

    [root@ip-10-233-25-98 filebench-1.4.9.1]# filebench
    Filebench Version 1.4.9.1
    7966: 0.000: Allocated 170MB of shared memory
    filebench> load randomread
    7966: 7.312: Random Read Version 3.0 personality successfully loaded
    7966: 7.312: Usage: set $dir=<dir> defaults to /tmp
    7966: 7.312: set $filesize=<size> defaults to 5368709120
    7966: 7.312: set $iosize=<value> defaults to 8192
    7966: 7.312: set $nthreads=<value> defaults to 1
    7966: 7.312: set $workingset=<value> defaults to 0
    7966: 7.312: set $directio=<bool> defaults to 0
    7966: 7.312: run runtime (e.g. run 60)
    filebench> set $dir=/naspool1/vol1
    filebench> set $iosize=16384
    filebench> set $nthreads=256
    filebench> run 60
    7966: 43.561: Creating/pre-allocating files and filesets
    7966: 43.561: File largefile1: 5120.000MB
    7966: 43.561: Re-using file largefile1.
    7966: 43.561: Creating file largefile1...
    7966: 43.562: Preallocated 1 of 1 of file largefile1 in 1 seconds
    7966: 43.562: waiting for fileset pre-allocation to finish
    8044: 43.562: Starting 1 rand-read instances
    8045: 43.564: Starting 256 rand-thread threads
    7966: 44.893: Running...
    7966: 104.899: Run took 60 seconds...
    7966: 104.900: Per-Operation Breakdown
    rand-read1 592457ops 9873ops/s 154.2mb/s 0.1ms/op 37543us/op-cpu [0ms - 44ms]
    7966: 104.900: IO Summary: 592457 ops, 9873.407 ops/s, (9873/0 r/w), 154.2mb/s, 239us cpu/op, 0.1ms latency
    7966: 104.900: Shutting down processes
    [root@ip-10-233-25-98 filebench-1.4.9.1]#



    RANDOM READ (128 threads)

    [root@ip-10-233-25-98 filebench-1.4.9.1]# filebench
    Filebench Version 1.4.9.1
    7515: 0.000: Allocated 170MB of shared memory
    filebench> load randomread
    7515: 6.192: Random Read Version 3.0 personality successfully loaded
    7515: 6.192: Usage: set $dir=<dir> defaults to /tmp
    7515: 6.192: set $filesize=<size> defaults to 5368709120
    7515: 6.192: set $iosize=<value> defaults to 8192
    7515: 6.192: set $nthreads=<value> defaults to 1
    7515: 6.192: set $workingset=<value> defaults to 0
    7515: 6.192: set $directio=<bool> defaults to 0
    7515: 6.192: run runtime (e.g. run 60)
    filebench> set $dir=/naspool1/vol1
    filebench> set $iosize=16384
    filebench> set $nthreads=128
    filebench> run 60
    7515: 37.738: Creating/pre-allocating files and filesets
    7515: 37.739: File largefile1: 5120.000MB
    7515: 37.739: Re-using file largefile1.
    7515: 37.739: Creating file largefile1...
    7515: 37.739: Preallocated 1 of 1 of file largefile1 in 1 seconds
    7515: 37.739: waiting for fileset pre-allocation to finish
    7631: 37.739: Starting 1 rand-read instances
    7632: 37.742: Starting 128 rand-thread threads
    7515: 38.915: Running...
    7515: 98.921: Run took 60 seconds...
    7515: 98.921: Per-Operation Breakdown
    rand-read1 562265ops 9370ops/s 146.4mb/s 0.0ms/op 14813us/op-cpu [0ms - 51ms]
    7515: 98.921: IO Summary: 562265 ops, 9370.263 ops/s, (9370/0 r/w), 146.4mb/s, 209us cpu/op, 0.0ms latency
    7515: 98.922: Shutting down processes
    [root@ip-10-233-25-98 filebench-1.4.9.1]#




    RANDOM R/W
    Filebench Version 1.4.9.1
    6856: 0.000: Allocated 170MB of shared memory
    filebench> load randomrw
    6856: 13.371: Random RW Version 3.0 personality successfully loaded
    6856: 13.371: Usage: set $dir=<dir> defaults to /tmp
    6856: 13.371: set $filesize=<size> defaults to 5368709120
    6856: 13.371: set $iosize=<value> defaults to 8192
    6856: 13.371: set $nthreads=<value> defaults to 1
    6856: 13.371: set $workingset=<value> defaults to 0
    6856: 13.371: set $directio=<bool> defaults to 0
    6856: 13.371: run runtime (e.g. run 60)
    filebench> set $dir=/naspool1/vol1
    filebench> set $iosize=16384
    filebench> set $nthreads=128
    filebench> run 60
    6856: 47.134: Creating/pre-allocating files and filesets
    6856: 47.134: File largefile1: 5120.000MB
    6856: 47.134: Re-using file largefile1.
    6856: 47.134: Creating file largefile1...
    6856: 47.135: Preallocated 1 of 1 of file largefile1 in 1 seconds
    6856: 47.135: waiting for fileset pre-allocation to finish
    6940: 47.135: Starting 1 rand-rw instances
    6941: 47.137: Starting 128 rand-w-thread threads
    6941: 47.302: Starting 128 rand-r-thread threads
    6856: 48.467: Running...
    6856: 108.473: Run took 60 seconds...
    6856: 108.474: Per-Operation Breakdown
    rand-write1 45043ops 751ops/s 11.7mb/s 135.3ms/op 327721us/op-cpu [0ms - 1304ms]
    rand-read1 400845ops 6680ops/s 104.3mb/s 0.2ms/op 20785us/op-cpu [0ms - 1255ms]
    6856: 108.474: IO Summary: 445888 ops, 7430.814 ops/s, (6680/751 r/w), 116.0mb/s, 295us cpu/op, 13.9ms latency
    6856: 108.474: Shutting down processes
    [root@ip-10-233-25-98 filebench-1.4.9.1]#


    SEQUENTIAL READ

    filebench> set $dir=/naspool1/vol1
    filebench> set $nthreads=128
    filebench> run 60
    6250: 27.591: Creating/pre-allocating files and filesets
    6250: 27.591: File largefile1: 5120.000MB
    6250: 27.591: Re-using file largefile1.
    6250: 27.591: Creating file largefile1...
    6250: 79.807: Preallocated 1 of 1 of file largefile1 in 53 seconds
    6250: 79.807: waiting for fileset pre-allocation to finish
    6386: 79.807: Starting 1 seqread instances
    6387: 79.810: Starting 128 seqread threads
    6250: 81.146: Running...
    6250: 141.156: Run took 60 seconds...
    6250: 141.156: Per-Operation Breakdown
    seqread 125975ops 2,099ops/s 2,097.1mb/s 4.2ms/op 169208us/op-cpu [0ms - 847ms]
    6250: 141.156: IO Summary: 125975 ops, 2099.269 ops/s, (2099/0 r/w), 2097.1mb/s, 2412us cpu/op, 4.2ms latency
    6250: 141.156: Shutting down processes


    References

    SoftNAS VSA

    Amazon EC2 Provisioned IOPS

    Benchmark: filebench
    Last edited by rbraddy; 05-11-2013 at 11:08 AM.

  2. #2
    Administrator
    Join Date
    Apr 2012
    Location
    Texas
    Posts
    153
    Blog Entries
    23
    I decided to go back and re-run the tests using RAID 10 instead of RAIDZ1 (RAID5), which did improve the write IOPS.

    Second benchmark... same setup, except this time with ten 500 GB 1,000 IOPS EBS volumes configured in a RAID 10 configuration, as shown below (this results in five RAID 1 mirrors striped as RAID0)

    The write IOPS are much better with RAID10:

    rand-write1 388955ops 2561ops/s 40.0mb/s 34.1ms/op 201124us/op-cpu [0ms - 801ms]
    rand-read1 693099ops 4564ops/s 71.3mb/s 0.0ms/op 78205us/op-cpu [0ms - 80ms]
    6559: 194.671: IO Summary: 1082054 ops, 7125.146 ops/s, (4564/2561 r/w), 111.3mb/s, 350us cpu/op, 12.3ms latency

    Here's a screenshot of the pool and device configuration in SoftNAS (note that the poolname says "ssdpool1", but it's actually not ssd - just a naming habit)

    EC2-RAID10-ProvisionedIOPSTest.jpg

    This reinforces the well-known fact that RAID10 is a better configuration for write-intensive workloads than RAIDZ1.
    Last edited by rbraddy; 05-11-2013 at 06:10 PM.

 

 

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •