Tuesday 14th August 2012: 3.35pm. My new mirrored ZFS pool has finished synchronising, so I thought I'd try a quick performance test as supposedly ZFS knows it can pluck every second block from each mirror and therefore double read performance, and I was keen to see ~240Mb/sec reads. Here's what I got instead:
[root@freenas]/mnt/IcyBoxZ# dd if=Windows\ Easy\ Transfer\ -\ Items\ from\ old\ computer.MIG of=/dev/null bs=1M
65669+1 records in
65669+1 records out
68859136109 bytes transferred in 893.325946 secs (77081760 bytes/sec)
About 73Mb/sec, or 60% of gigabit LAN speed, and a very long way short of 240Mb/sec. Interestingly, that's almost exactly what read performance was before I added the mirror with a single drive, one which can easily break 100Mb/sec. Perversely write speed is unaffected, and sustains 130Mb/sec without issue.
Cause? Almost certainly the lousy virtio driver support in FreeBSD before v9.0 where they completely replaced the storage abstraction layer in the kernel. My FreeNAS v8.3 (based on FreeBSD v8.3) does support virtio, and it's bug free at last unlike earlier FreeBSD's where it would corrupt your data slightly. virtio support was only added at all in FreeBSD v8.2 and won't become mainline until FreeBSD v10. So I guess I'll be waiting for FreeNAS v9.1 which is certainly six months away. No rush though. ~70Mb/sec reads are more than plenty on a mostly 11n Wifi based network, it won't be till we find a home in Canada that I can try to sneak in CAT6 ethernet cabling without Megan noticing :)