Investigate fsync performance

[behlendorf]

There are reports that fsync performance, as measured by postgresql's pg_test_fsync test, are significantly slower in ZoL compared to FreeBSD 10. The following test results are from the same machine (DELL R720dx 12*4TB SATA) and hardware configuration.

Since the disk isn't saturated for ZoL I suspect that we're blocking longer than needed somewhere in the stack. We'll want to profile the workload to see where.

in CentOS 6.5 x64 yum installed zfs 0.6.3, the result is.

        fdatasync                         329.207 ops/sec    3038 usecs/op
        fsync                             328.739 ops/sec    3042 usecs/op
Non-Sync'ed 8kB writes:
        write                           76100.311 ops/sec      13 usecs/op

and disk utils output by iostat is between 20% and 30%

and in FreeBSD 10. iops below:

        fsync                            6676.001 ops/sec     150 usecs/op
        fsync_writethrough                            n/a
        open_sync                        6087.783 ops/sec     164 usecs/op
Non-Sync'ed 8kB writes:
        write                           67460.621 ops/sec      15 usecs/op

[behlendorf]

This is believe to have been resolved.