-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fuse-overlayfs: different performance tweaks #88
Conversation
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
64073d8
to
6f3daa5
Compare
this needs a fix in SELinux:
|
@rhatdan is it possible to enable it only for |
Was this actually blocking? Since this is an unconfined domain it should be allowed. But it looks like a bogus AVC? A process is attempting to change the attributes of something in /proc/self? The link file is fuse-overlayfs? Is this more about the setattr should be on the thing the link points at? |
the process inside the rootless container was blocked. I'm using |
If this is a container process trying to make this change then it is running with the wrong label. unconfined_t versus container_t. |
type=AVC msg=audit(1563197807.799:377): avc: denied { setattr } for pid=7950 comm="fuse-overlayfs" name="183" dev="proc" ino=160867 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=lnk_file permissive=0 IE is this supposed to be |
No problem allowing Problem with |
I see. fuse-overlayfs doesn't run in the container, only in its user+mount namespace created by podman |
a90a5d7
to
7fbe3df
Compare
@rhatdan could we get the With this PR fuse-overlayfs performs significantly better. I've added some new options, such as disabling xattrs. This helps with SELinux as we won't hit the FUSE file system with a getxattr for every operation. Also, disabling fsync seems a sane default for containers, at least for the rootfs where fuse-overlayfs is used. With this in place fuse-overlayfs performs better than native overlay in most cases. Using
with root and native overlay:
|
@wrabcak Could you update selinux-policy for Fedora and RHEL7/RHEL8 with |
Added to selinux-policy for F29, F30 and Rawhide: |
5465213
to
16beb8d
Compare
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
main.c
Outdated
|
||
if (asprintf (&whiteout_opq_path, "%s/" OPAQUE_WHITEOUT, path) < 0) | ||
return -1; | ||
sprintf (whiteout_opq_path, "%s/" OPAQUE_WHITEOUT, path); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should still use snprintf?
main.c
Outdated
ret = asprintf (&path, "%s/%s", parent->path, name); | ||
if (ret < 0) | ||
return ret; | ||
sprintf (path, "%s/%s", parent->path, name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
snprintf
main.c
Outdated
ret = asprintf (&whiteout_path, "%s/%s", parent->path, name); | ||
if (ret < 0) | ||
return ret; | ||
sprintf (whiteout_path, "%s/%s", parent->path, name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
snprintf
main.c
Outdated
ret = asprintf (&whiteout_wh_path, "%s/.wh.%s", parent->path, name); | ||
if (ret < 0) | ||
return ret; | ||
sprintf (whiteout_wh_path, "%s/.wh.%s", parent->path, name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
snprintf
main.c
Outdated
ret = asprintf (&whiteout_path, "%s/%s", parent->path, name); | ||
if (ret < 0) | ||
return ret; | ||
sprintf (whiteout_path, "%s/%s", parent->path, name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
snprintf
main.c
Outdated
ret = asprintf (&whiteout_path, ".wh.%s", name); | ||
if (ret < 0) | ||
return ret; | ||
sprintf (whiteout_path, ".wh.%s", name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
snprintf
main.c
Outdated
ret = asprintf (&whiteout_path, "%s/.wh.%s", parent->path, name); | ||
if (ret < 0) | ||
return ret; | ||
sprintf (whiteout_path, "%s/.wh.%s", parent->path, name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
snprintf
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And continue using it fall sprintf calls....
main.c
Outdated
ret = asprintf (&whiteout_path, ".wh.%s", dent->d_name); | ||
if (ret < 0) | ||
return NULL; | ||
sprintf (whiteout_path, ".wh.%s", dent->d_name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
snprintf
main.c
Outdated
ret = asprintf (&path, "%s/%s", pnode->path, name); | ||
if (ret < 0) | ||
return NULL; | ||
sprintf (path, "%s/%s", pnode->path, name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
snprintf
Would have been a lot easier to review, if you did a lot of these as separate reviews and documented how they helped improve performance. |
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
56ad386
to
59cd4c0
Compare
Happy to see work fuse-overlayfs' performance. This records the time taken to create a couple thousand folders (via
/usr/bin/fuse-overlayfs --version
/tmp/fuse-overlayfs-gh-88 --version
for storage_option in \
--storage-driver=vfs \
--storage-driver=overlay \
--storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs \
--storage-opt=overlay.mount_program=/tmp/fuse-overlayfs-gh-88 \
; do
printf \\n\\n
for count in 1000 2000 20000 40000 80000 ; do
podman --root="$(pwd)/containers" ${storage_option} run --rm -qit \
--workdir=/test --env=count="${count}" --env=TIME="${storage_option} ${count} %S %U %e %C" \
busybox sh -ec 'seq "${count}" | \time xargs mkdir && \time find . -delete ; printf \\n'
done
rm -rf ./containers
done | column -tLN storage,count,sys,user,wall,command -R storage,count,sys,user,wall
Regarding the new options: I have no idea (didn't think about/research it) whether setting |> podman --root="$(pwd)/containers" \
|> --storage-opt='"overlay.mountopt=threaded=0,fsync=0","overlay.mount_program=/tmp/fuse-overlayfs-gh-88"' \
|> run --rm -qit --workdir=/test --env=TIME='%S %U %e %C' \
|> busybox sh -ec 'seq 20000 | \time xargs mkdir && \time find . -delete'
0.90 0.07 7.13 xargs mkdir
1.66 0.25 13.43 find . -delete
|> podman --root="$(pwd)/containers" \
|> --storage-opt='"overlay.mountopt=threaded=0,fsync=1","overlay.mount_program=/tmp/fuse-overlayfs-gh-88"' \
|> run --rm -qit --workdir=/test --env=TIME='%S %U %e %C' \
|> busybox sh -ec 'seq 20000 | \time xargs mkdir && \time find . -delete'
0.94 0.04 7.18 xargs mkdir
1.62 0.35 13.52 find . -delete
|> podman --root="$(pwd)/containers" \
|> --storage-opt='"overlay.mountopt=threaded=1,fsync=0","overlay.mount_program=/tmp/fuse-overlayfs-gh-88"' \
|> run --rm -qit --workdir=/test --env=TIME='%S %U %e %C' \
|> busybox sh -ec 'seq 20000 | \time xargs mkdir && \time find . -delete'
1.19 0.03 10.17 xargs mkdir
1.61 0.55 17.83 find . -delete
|> podman --root="$(pwd)/containers" \
|> --storage-opt='"overlay.mountopt=threaded=1,fsync=1","overlay.mount_program=/tmp/fuse-overlayfs-gh-88"' \
|> run --rm -qit --workdir=/test --env=TIME='%S %U %e %C' \
|> busybox sh -ec 'seq 20000 | \time xargs mkdir && \time find . -delete'
1.10 0.11 10.30 xargs mkdir
1.76 0.51 18.03 find . -delete |
Re-ran the above with the latest changes from 59cd4c0 :
Some of those changes fixed the non-linear slowness -- massive improvement! |
Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
59cd4c0
to
9e20d96
Compare
LGTM |
@mbargull thanks a lot for your tests, I've opened a new PR that should improve the tests case you've reported above: #89 For your test case, the new options should not make any difference, in particular Could you re-run the test with #89? In general I am curious about the previous released version of fuse-overlayfs vs the version in #89. |
I'll do that once https://bodhi.fedoraproject.org/updates/FEDORA-2019-b156bd756a is finally moved to stable and #89 is also merged |
fuse-overlayfs: different performance tweaks continuation of #88
With pleasure! Thanks for your work on this. I'll report in gh-91, comparing release
Right, makes sense, the |
Sorry I have just opened a new PR. Could you just use that? The intermediate ones don't make much sense to be tested separately |
Ah, didn't see comment. Will do, gimme a couple of minutes. |
this PR has different performance improvements, most notably, there is some initial support for threading. Most of the code is still protected by a lock, so it runs on a single threaded, but operations like read/write/setattr/fsync can be dispatched on a different thread.
It is now possible to disable
fsync
that seems to slowapt
quite significantly. In general it is probably safer to run containers without access to fsync.The new options can be configured in Podman through
~/.config/containers/storage.conf
. I've got the best results with this combination:mountopt = "threaded=0,fsync=0"