pull: Implement --enable-syncfs #49
Conversation
|
See https://bugzilla.gnome.org/show_bug.cgi?id=728065 The problem with this is that if the system crashes in the middle, there's no guarantee that the objects written before the syncfs() are valid. I actually hit that, see https://git.gnome.org/browse/ostree/commit/?id=2396608754060e9e7a68843b65e4d3b8f4a4b5f2 |
giuseppe
force-pushed
the
giuseppe/add-pull-enable-syncfs
branch
from
f86de76 to
b13e13b
Compare
|
I've pushed another version with the logic described here: https://bugzilla.gnome.org/show_bug.cgi?id=728065 All tests pass |
giuseppe
force-pushed
the
giuseppe/add-pull-enable-syncfs
branch
2 times, most recently
from
4631eb7 to
d062aaa
Compare
|
Hmm, didn't I have more comments on an older version of this commit? I can't find it offhand. But for example I thought I wrote up a concern about power cuts; this patch isn't addressing that yet is it? I wish github made it easier to find old comments... |
|
yes, I didn't implement that. I believed it was a comment for a later improvement. I am going to send another version that uses the boot uuid. |
Do not write directly to objects/ but maintain pulled files under tmp/ with a "tmpobject-$CHECKSUM.$OBJTYPE" name until they are syncfs'ed to disk. Move them under objects/ at ostree_repo_commit_transaction cleanup time. Before (test done on a local network): $ LANG=C sudo time ./ostree --repo=repo pull origin master 0 metadata, 3 content objects fetched; 83820 KiB; 4 delta parts fetched, transferred in 417 seconds 16.42user 6.73system 6:57.19elapsed 5%CPU (0avgtext+0avgdata 248428maxresident)k 24inputs+794472outputs (0major+233968minor)pagefaults 0swaps After: $ LANG=C sudo time ./ostree --repo=repo pull origin master 0 metadata, 3 content objects fetched; 83820 KiB; 4 delta parts fetched, transferred in 9 seconds 14.70user 2.87system 0:09.99elapsed 175%CPU (0avgtext+0avgdata 256168maxresident)k 0inputs+794472outputs (0major+164333minor)pagefaults 0swaps https://bugzilla.gnome.org/show_bug.cgi?id=728065 Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe
force-pushed
the
giuseppe/add-pull-enable-syncfs
branch
2 times, most recently
from
bc29e71 to
c64793d
Compare
This prevents to use files after a kernel crash or power failure and that can be not completely synced to disk. Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe
force-pushed
the
giuseppe/add-pull-enable-syncfs
branch
from
c64793d to
c88ea80
Compare
|
Squashed these fixups on top and pushed. 👍 |
Add manual bindings for MutableTree reading
Differently than fsync that is called after each file is modified,
syncfs is used only once at the end of the pull to sync the file
system which contains the repository.
Results of pull on an empty repository:
$ LANG=C sudo time ./ostree --repo=repo --disable-fsync pull master master
0 metadata, 0 content objects fetched; 83524 KiB; 6 delta parts fetched, transferred in 5 seconds
17.09user 2.41system 0:05.77elapsed 337%CPU (0avgtext+0avgdata 296132maxresident)k
0inputs+868584outputs (0major+102828minor)pagefaults 0swaps
$ LANG=C sudo time ./ostree --repo=repo pull master master
0 metadata, 0 content objects fetched; 83524 KiB; 6 delta parts fetched, transferred in 401 seconds
17.31user 5.73system 6:41.33elapsed 5%CPU (0avgtext+0avgdata 296864maxresident)k
0inputs+868584outputs (0major+132664minor)pagefaults 0swaps
$ LANG=C sudo time ./ostree --repo=repo --enable-syncfs pull master master
0 metadata, 0 content objects fetched; 83524 KiB; 6 delta parts fetched, transferred in 6 seconds
17.14user 2.57system 0:10.60elapsed 185%CPU (0avgtext+0avgdata 296212maxresident)k
0inputs+868584outputs (0major+86783minor)pagefaults 0swaps
Signed-off-by: Giuseppe Scrivano [email protected]