storage/engine: Do we need use_fsync?

[irfansharif]

Fallout from discussion at #16361.

RocksDB by default uses faster fdatasync() to sync files. If you want to use fsync(), you can set Options::use_fsync to true. You should set this to true on filesystems like ext3 that can lose files after a reboot.

Source: rocksdb/wiki

This is a flag set on rocksdb::Options, it should somewhere happen here but we don't seem to be doing so.

cc @petermattis.

[bdarnell]

Which filesystems are "like ext3" in this respect? How common is ext3 these days? ext4 was the default in RHEL6, which is the oldest platform we support. I would expect (but have not verified) that all platforms we care about default to filesystems where this is not an issue.

It would be nice if we could encapsulate this knowledge into rocksdb instead of doing our own inspection of the filesystem.

[petermattis]

This isn't new behavior in 1.1. Punting to 1.2.

[nvanbenschoten]

This is also discussed on the comment for rocksdb::Options.use_fsync.

I'm also kind of confused why this logic isn't encapsulated in RocksDB, especially because the criteria is pretty vague: "filesystems like ext3 that can lose files after a reboot". Is anyone aware of a way to detect systems that match this requirement?

I've audited a few popular uses of RocksDB (MyRocks, MongoRocks, and TIKV) and none of them have any logic to handle this. In fact, the only project that even uses the use_fsync option at all is MyRocks, which seems to link it to a command-line flag.

[bdarnell]

Wearing my programmer-archaeologist hat:

The line "filesystems like ext3 that can lose files after a reboot" was added in August 27, 2012 (facebook/rocksdb@fc20273)

An fdatasync bug in both ext3 and ext4 was reported and fixed in september. The fixes are torvalds/linux@156bddd for ext3 (first published in kernel version 3.6 AFAICT) and torvalds/linux@b71fc07 for ext4 (first published in kernel version 3.7).

So at the time that comment was written, ext4 was "like ext3" and could lose data if fdatasync were used instead of fsync. I think it's plausible that this is the bug it was referring to. This particular bug is long since fixed for both filesystems. It's always possible that there are other bugs but I don't see any particular reason to single out ext4 for this. I haven't seen any more recent reports of problems that are fixed by switching from fdatasync to fsync.

This is most interesting in the context of RHEL6, our oldest supported platform. They run with a custom kernel, branched from the mainline way back in 2.6.32 but with many fixes backported. I think the ext4 fix was not backported to the RHEL6 kernel until Dec 17 2015, in version 2.6.32-595.el6 (source)

[bdarnell]

Removing milestone because I don't think we need to do anything involving the use_fsync option. Leaving open until we get an upstream response on facebook/rocksdb#3410

[bdarnell]

Upstream doesn't know of any other reason to set this flag, so I think we can just close this issue without exposing a way to set use_fsync (unless and until a new bug here is discovered)