i recently got new toy for tests – brand new dell powervault md1000.
what's this, you ask? basically – a rather nice das (direct attached storage) from dell.
the box i got had 15 sas discs, each disc being 72gb, 15krpm.
since this will be used as database storage, i wanted to make some performance tests.
the box was connected to a nice server, i had handy (also dell), specs:
- 4, dual core, xeon processors (3.4ghz)
- 32gb ram
- 2 internal, 15k rpm, 72g, sas discs for system in raid1 setup
for making tests we used bonnie++. test procedure was quite simple:
- setup new raid
- mke2fs -j (ext3)
- mount using noatime and nodiratime options
- run bonnie using these parameters: “-u nobody:nobody -f -s 65000,8192 -n 0 -x 3" (8192 because this is page size in postgresql)
- results (-x 3!) were averaged
i knew we will be going for raid10, so this is what we tested at first.
2 separate series of tests were made:
- raid 10, purely hardware, using 2, 4, 6, 8, 10, 12, and 14 discs
- raid 10, mixed software/hardware – hardware was used to built 7 separate logical devices, each using 2 discs in raid1, then we combined them in software linux raid0 to create 4, 6, 8, 10, 12 and 14 discs raid10 setups
(all tests were done using no read ahead, and write-back using controller's battery backed up memory).
results are somewhat strange (red line – pure hardware, green line – hardware/software):
if you want the results as table:
name column means:
- (\d+)xraid(\d+) – $1 discs in pure hardware raid $2. for example – 6xraid10 means 6 discs in pure hardware raid10
- (\d+)s0@2h1 – $1 logical drives (where each logical drive is 2 discs in hardware raid1) connected using software raid0. for example 5s0@2h1 – 5 logical drives (each of 2 discs in raid1) connected giving in total raid10 over 10 discs in mixed hardware/software setting
- 2s0@6h10 – 2 logical drives, each being 6-disk, hardware raid 10, connected using software raid 0
- 2s1@2h1 – 2 logical drives, each being 2-disk, hardware raid 1, connected using software raid 1 – effectively giving size of 1 disc but using 4 drives
as you can see we tested more than i showed in graphs above, but the rest of tests were only for “fun" of testing :).
strange thing – purely hardware raid has very visible “steps" in write/rewrite performance. speed was gained only when total number of used discs was power of 2. i don't know why, but this is how it worked. in mixed software/hardware raid – there was no such situation.
since we have 15 discs, we decided to use layout of:
- 1 disc for global hot-spare
- 2 disc raid1 (hardware) for pg_xlog
- 8 disc raid10 (hardware) for primary tablespace
- 4 disc raid10 (hardware) for secondary tablespace
theoretically this layout should give best results.
after creation of the arrays and mkfs i decided to make a concurrent test of all 3 arrays at the same time.
i will not graph the results as this is not the point, but numbers from the results are:
(i show here all 3 results for each array for comparison purposes).
as you can see the numbers are lower than expected. it has to be noted that 8xraid10 disc finished tests as first, 4xraid10 as second and 2xraid1 as last.
in general i think that the layout we've chosen is the best possible on this array, but i can't help wondering why is this controller so performance-bound to powers of 2.
hope you'll find the numbers useful for your own purposes.