lib/repo: Add a DEVINO_CANONICAL commit modifier flag #1357
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Dec 1, 2017
There's a lot more details in the libostree PR: ostreedev/ostree#1357 Basically loading the xattrs is slow; let's only do it if we need to, and "need to" is defined by "SELinux policy changed". On my test F27AH VM, the difference between a stat() + hash table lookup versus the full xattr load on my test case of rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm is absolutely dramatic; consistently on the order of 10s without this support, and <1s with (800ms).
|
rpm-ostree PR: coreos/rpm-ostree#1123 |
jlebon
reviewed
Dec 1, 2017
src/libostree/ostree-repo.h
Outdated
| @@ -644,6 +644,7 @@ typedef OstreeRepoCommitFilterResult (*OstreeRepoCommitFilter) (OstreeRepo *r | |||
| * @OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS: Canonicalize permissions for bare-user-only mode. | |||
| * @OSTREE_REPO_COMMIT_MODIFIER_FLAGS_ERROR_ON_UNLABELED: Emit an error if configured SELinux policy does not provide a label | |||
| * @OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CONSUME: Delete added files/directories after commit; Since: 2017.13 | |||
| * @OSTREE_REPO_COMMIT_MODIFIER_FLAGS_DEVINO_CANONICAL: If a devino cache hit is found, do not compare metadata/flags, directly consume object; Since: 2017.14 | |||
There was a problem hiding this comment.
Maybe let's say something like:
If a devino cache hit is found, skip modifier filters and use it
?
| child_info = NULL; | ||
| if (!ostree_repo_load_file (self, loose_checksum, NULL, &child_info, &source_xattrs, | ||
| cancellable, error)) | ||
| /* Go directly to checksum, do not pass Go, do not collect $200. |
I was seeing the `Writing OSTree commit...` phase of rpm-ostree being very slow lately. This turns out to be more fallout from ostreedev#1170 AKA commit: 8fe4536 Loading the xattrs is slow on my system (F27AW, XFS+LVM, NVMe). I haven't fully traced through why, but AIUI at least on XFS the xattrs are often stored outside of the inode so it's a little bit like doing an `open()+read()`. Plus there's the LSM overhead, etc. The thing is that for rpm-ostree's package layering use case, we basically always want to treat the on-disk state as canonical. (There's a subtle case here if one does overrides for something that contains policy but we'll fix that). Anyways, so we're in a state now where we do the slow but correct thing by default, which seems sane. But let's allow the app to opt-in to telling us "really trust devino". The difference between a `stat()` + hash table lookup versus the full xattr load on my test case of `rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm` is absolutely dramatic; consistently on the order of 10s without this support, and <1s with (800ms).
cgwalters
force-pushed
the
devino-canonical
branch
from
c44f01a to
8535a6e
Compare
|
Now with tests ✅ |
|
☀️ Test successful - status-atomicjenkins |
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Dec 4, 2017
There's a lot more details in the libostree PR: ostreedev/ostree#1357 Basically loading the xattrs is slow; let's only do it if we need to, and "need to" is defined by "SELinux policy changed". On my test F27AH VM, the difference between a stat() + hash table lookup versus the full xattr load on my test case of rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm is absolutely dramatic; consistently on the order of 10s without this support, and <1s with (800ms).
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Dec 4, 2017
There's a lot more details in the libostree PR: ostreedev/ostree#1357 Basically loading the xattrs is slow; let's only do it if we need to, and "need to" is defined by "SELinux policy changed". On my test F27AH VM, the difference between a stat() + hash table lookup versus the full xattr load on my test case of rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm is absolutely dramatic; consistently on the order of 10s without this support, and <1s with (800ms).
rh-atomic-bot
pushed a commit
to coreos/rpm-ostree
that referenced
this pull request
Dec 4, 2017
There's a lot more details in the libostree PR: ostreedev/ostree#1357 Basically loading the xattrs is slow; let's only do it if we need to, and "need to" is defined by "SELinux policy changed". On my test F27AH VM, the difference between a stat() + hash table lookup versus the full xattr load on my test case of rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm is absolutely dramatic; consistently on the order of 10s without this support, and <1s with (800ms). Closes: #1123 Approved by: jlebon
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was seeing the
Writing OSTree commit...phase of rpm-ostreebeing very slow lately. This turns out to be more fallout from
#1170
AKA commit: 8fe4536
Loading the xattrs is slow on my system (F27AW, XFS+LVM, NVMe). I haven't fully
traced through why, but AIUI at least on XFS the xattrs are often stored outside
of the inode so it's a little bit like doing an
open()+read(). Plus there'sthe LSM overhead, etc.
The thing is that for rpm-ostree's package layering use case, we
basically always want to treat the on-disk state as canonical. (There's
a subtle case here if one does overrides for something that contains
policy but we'll fix that).
Anyways, so we're in a state now where we do the slow but correct thing by
default, which seems sane. But let's allow the app to opt-in to telling us
"really trust devino". The difference between a
stat()+ hash table lookupversus the full xattr load on my test case of
rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpmis absolutely dramatic; consistently on theorder of 10s without this support, and <1s with (800ms).
No tests yet.