792 MB/s read speed at 12% cpu use in the directio case looks a bit promising :)
# dd if=/dev/fioa2 of=/dev/null bs=1M
26723+1 records in
26723+1 records out
28021615104 bytes (28 GB) copied, 201.797 s, 139 MB/s
# dd if=/dev/fioa2 of=/dev/null bs=1M iflag=direct
19792+0 records in
19791+0 records out
20752367616 bytes (21 GB) copied, 26.2142 s, 792 MB/s
# dd if=/dev/sda4 of=/dev/null bs=1M
5157+0 records in
5156+0 records out
5406457856 bytes (5.4 GB) copied, 25.136 s, 215 MB/s
# dd if=/dev/sda4 of=/dev/null bs=1M iflag=direct
6040+0 records in
6039+0 records out
6332350464 bytes (6.3 GB) copied, 26.1701 s, 242 MB/s
Stay tuned, more to come...