Initial fs-verity support #1959
Conversation
cgwalters
force-pushed
the
ostree-verity
branch
4 times, most recently
from
02aedbe to
21c36dd
Compare
|
I think Android is basically going to continue using dm-verity for the base OS, and use fs-verity just for apps which come as single .zip files. So they don't have the problem of verifying full filesystem trees. I was thinking about this some more, and a rather gross hack might be to store all of our directories/symlinks as effectively tarballs in the real root (that are fs-verity protected), then on boot we unpack them into a tmpfs, and use overlayfs with the tmpfs and the underlying fs-verity files. But clearly what we really want is an extension to fs-verity that lets us "seal" a directory and set of symlinks in it too. |
|
Continuing my half-baked ideation... Maybe we can use a dual-partition dm-verity to hold the directories/symlinks, and an overlayfs pointing to the underlying real rootfs with fs-verity files? But...what's bugging me here is that if an attacker gains |
|
Also, one thing we can and should do is use fs-verity for |
cgwalters
force-pushed
the
ostree-verity
branch
from
21c36dd to
957f05b
Compare
|
OK a number of tweaks, biggest is that we now also do fsverity for files in Also now: |
|
I think the remaining files to cover are the config file (circular, need to write it then verity it), and the origin file. |
|
Side note: I just confirmed that fs-verity is perfectly happy to run without any privileges (plain Unix user). This means that ostree's fsverity support would work just fine for flatpak user installs. One could also use fs-verity for random config files like We also should consider how this intersects with Hmm. Maybe now that overlayfs is pretty stable...we should go with adding a model where overlayfs is used for I'm excited about the potential of fs-verity! |
cgwalters
force-pushed
the
ostree-verity
branch
3 times, most recently
from
3a9132d to
2f516e7
Compare
cgwalters
force-pushed
the
ostree-verity
branch
from
2f516e7 to
c2301cb
Compare
|
Lifted the WIP from this - it definitely works and does something useful today: makes files immutable. I particularly want to do this for FCOS for The question is; is that worth merging as is? I'm uncertain. I think it's quite likely we'll need a "sign this file" callback. And...looking at that, we have the big decision whether to import the existing C code or to execute it as a subprocess (or, try to get a library made out of it upstream) (Or, git submodule it). And, none of the signing will really be worth much until we implement a full plan per above. And, actually need support for shipping the signatures out of band too. That's going to be hard to do elegantly while retaining backwards compatibility. Bigger picture, if one can assume verity, then we don't need the "basic OSTree sha256" anymore...which argues for thinking of this more like a fully separate "mode" for OSTree with its own repository format... |
cgwalters
force-pushed
the
ostree-verity
branch
from
c2301cb to
4aec484
Compare
|
I updated this to disable even the "opportunistic" use of fs-verity (previously it'd only happen if you happened to (That said also, we need to care for the case that |
|
OK, so confirm here, without the "measure" step, this is essentially a stronger version of |
Stronger in some ways, but also weaker in others; e.g. it does allow
I think a bottom line is this is useful as a starting point. |
|
/override f29-rpmostree |
|
@cgwalters: Overrode contexts on behalf of cgwalters: f29-rpmostree In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
cgwalters
force-pushed
the
ostree-verity
branch
2 times, most recently
from
7e10a8c to
8cefa5b
Compare
It's still experimental, update per review in ostreedev/ostree#1959
| if (fdatasync (tmp_dest.fd) < 0) | ||
| return glnx_throw_errno_prefix (error, "fdatasync"); | ||
|
|
||
| if (!_ostree_tmpf_fsverity_core (&tmp_dest, NULL, error)) |
There was a problem hiding this comment.
Shouldn't this be _ostree_tmpf_fsverity so it respects the settings?
There was a problem hiding this comment.
Yeah...the problem is that /boot may be a different filesystem type. The repo is trying to cache the supported state for the repo. Messy...
I tweaked this so we don't enable verity for /boot if we're not using it for / - but currently if it's required by / we downgrade to "opportunistic" for /boot.
Using fs-verity is natural for OSTree because it's file-based, as opposed to block based (like dm-verity). This only covers files - not symlinks or directories. And we clearly need to have integrity for the deployment directories at least. Also, what we likely need is an API that supports signing files as they're committed. So making this truly secure would need a lot more work. Nevertheless, I think it's time to start experimenting with it. Among other things, it does *finally* add an API that makes files immutable, which will help against some accidental damage. This is basic enablement work that is being driven by Fedora CoreOS; see also coreos/coreos-assembler#876
cgwalters
force-pushed
the
ostree-verity
branch
from
8cefa5b to
58fa579
Compare
|
To expand on this:
It may help to only separate files with nontrivial size (maybe > 1k?). Also with this it may be simplest to have one big "verity pack file" for the initial deployment, then create small "delta pack files" for updates. Then at some point we do a "repack"? Also related to signatures and such, in this model we'd need to verify the signature on the verity pack files as well as every "loose file", since we can't rely on overlayfs to do that for us. OR we try to go with the in-kernel verity signature support. |
It's still experimental, update per review in ostreedev/ostree#1959
| _OstreeFeatureSupport boot_verity = _OSTREE_FEATURE_NO; | ||
| if (repo->fs_verity_wanted != _OSTREE_FEATURE_NO) | ||
| boot_verity = _OSTREE_FEATURE_MAYBE; | ||
| if (!_ostree_tmpf_fsverity_core (&tmp_dest, boot_verity, NULL, error)) |
There was a problem hiding this comment.
Won't this cause one ioctl each time even if it's not supported? Hmm, how about we add an optional fsverity_wanted parameter to _ostree_tmpf_fsverity that overrides the one from OstreeRepo? Then we could use it here too and get the automatic no-op.
Ohh, but right the kernel might support it, but not the filesystem on /boot. Hmm, so probably instead we should have a boot_fsverity in the OstreeSysroot object? But ehhh, this is good as is too. The number of files we install in /boot pales in comparison to / so it's a small optimization we could do later.
There was a problem hiding this comment.
Yeah, an extra syscall or two for this doesn't matter much, but I try to keep the total small since I sometimes just resort to strace -f - we have a lot more objects than kernels.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cgwalters, jlebon The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
At the time we added fsverity code to ostree, fsverity was just a CLI tool; since then it has gained a C shared library which wraps all the signature bits and the enablement `ioctl()` conveniently. This makes it much easier for us to support signatures, so do so. Note that at this time, because ostree doesn't define a mechanism to transport fsverity signatures across repositories, this is mostly only useful for locally-generated signatures. The idea however is this is a starting point from which we can build more support, including signature transport, remote keys, etc. In order to simplify things, drop support for "opportunistic" use of fsverity. In practice we expect people using this to set it up fully, or not at all. The "read only files" aspect *is* useful, but complicated the code too much relative to its benefit. Also drop support for using fsverity for `/boot` for now; in practice most things there are read by the bootloader, which doesn't know about fsverity. Instead those should be covered by e.g. Secure Boot. This ensures that we only have one high level API `_ostree_tmpf_fsverity()` that is invoked from the core commit path. xref https://lwn.net/Articles/842002 xref ostreedev#1959 xref coreos/rpm-ostree#1883
At the time we added fsverity code to ostree, fsverity was just a CLI tool; since then it has gained a C shared library which wraps all the signature bits and the enablement `ioctl()` conveniently. This makes it much easier for us to support signatures, so do so. Note that at this time, because ostree doesn't define a mechanism to transport fsverity signatures across repositories, this is mostly only useful for locally-generated signatures. The idea however is this is a starting point from which we can build more support, including signature transport, remote keys, etc. In order to simplify things, drop support for "opportunistic" use of fsverity. In practice we expect people using this to set it up fully, or not at all. The "read only files" aspect *is* useful, but complicated the code too much relative to its benefit. Also drop support for using fsverity for `/boot` for now; in practice most things there are read by the bootloader, which doesn't know about fsverity. Instead those should be covered by e.g. Secure Boot. This ensures that we only have one high level API `_ostree_tmpf_fsverity()` that is invoked from the core commit path. xref https://lwn.net/Articles/842002 xref ostreedev#1959 xref coreos/rpm-ostree#1883
At the time we added fsverity code to ostree, fsverity was just a CLI tool; since then it has gained a C shared library which wraps all the signature bits and the enablement `ioctl()` conveniently. This makes it much easier for us to support signatures, so do so. Note that at this time, because ostree doesn't define a mechanism to transport fsverity signatures across repositories, this is mostly only useful for locally-generated signatures. The idea however is this is a starting point from which we can build more support, including signature transport, remote keys, etc. In order to simplify things, drop support for "opportunistic" use of fsverity. In practice we expect people using this to set it up fully, or not at all. The "read only files" aspect *is* useful, but complicated the code too much relative to its benefit. Also drop support for using fsverity for `/boot` for now; in practice most things there are read by the bootloader, which doesn't know about fsverity. Instead those should be covered by e.g. Secure Boot. This ensures that we only have one high level API `_ostree_tmpf_fsverity()` that is invoked from the core commit path. xref https://lwn.net/Articles/842002 xref ostreedev#1959 xref coreos/rpm-ostree#1883
At the time we added fsverity code to ostree, fsverity was just a CLI tool; since then it has gained a C shared library which wraps all the signature bits and the enablement `ioctl()` conveniently. This makes it much easier for us to support signatures, so do so. Note that at this time, because ostree doesn't define a mechanism to transport fsverity signatures across repositories, this is mostly only useful for locally-generated signatures. The idea however is this is a starting point from which we can build more support, including signature transport, remote keys, etc. In order to simplify things, drop support for "opportunistic" use of fsverity. In practice we expect people using this to set it up fully, or not at all. The "read only files" aspect *is* useful, but complicated the code too much relative to its benefit. Also drop support for using fsverity for `/boot` for now; in practice most things there are read by the bootloader, which doesn't know about fsverity. Instead those should be covered by e.g. Secure Boot. This ensures that we only have one high level API `_ostree_tmpf_fsverity()` that is invoked from the core commit path. xref https://lwn.net/Articles/842002 xref ostreedev#1959 xref coreos/rpm-ostree#1883
At the time we added fsverity code to ostree, fsverity was just a CLI tool; since then it has gained a C shared library which wraps all the signature bits and the enablement `ioctl()` conveniently. This makes it much easier for us to support signatures, so do so. Note that at this time, because ostree doesn't define a mechanism to transport fsverity signatures across repositories, this is mostly only useful for locally-generated signatures. The idea however is this is a starting point from which we can build more support, including signature transport, remote keys, etc. In order to simplify things, drop support for "opportunistic" use of fsverity. In practice we expect people using this to set it up fully, or not at all. The "read only files" aspect *is* useful, but complicated the code too much relative to its benefit. Also drop support for using fsverity for `/boot` for now; in practice most things there are read by the bootloader, which doesn't know about fsverity. Instead those should be covered by e.g. Secure Boot. This ensures that we only have one high level API `_ostree_tmpf_fsverity()` that is invoked from the core commit path. xref https://lwn.net/Articles/842002 xref ostreedev#1959 xref coreos/rpm-ostree#1883
At the time we added fsverity code to ostree, fsverity was just a CLI tool; since then it has gained a C shared library which wraps all the signature bits and the enablement `ioctl()` conveniently. This makes it much easier for us to support signatures, so do so. Note that at this time, because ostree doesn't define a mechanism to transport fsverity signatures across repositories, this is mostly only useful for locally-generated signatures. The idea however is this is a starting point from which we can build more support, including signature transport, remote keys, etc. In order to simplify things, drop support for "opportunistic" use of fsverity. In practice we expect people using this to set it up fully, or not at all. The "read only files" aspect *is* useful, but complicated the code too much relative to its benefit. Also drop support for using fsverity for `/boot` for now; in practice most things there are read by the bootloader, which doesn't know about fsverity. Instead those should be covered by e.g. Secure Boot. This ensures that we only have one high level API `_ostree_tmpf_fsverity()` that is invoked from the core commit path. xref https://lwn.net/Articles/842002 xref ostreedev#1959 xref coreos/rpm-ostree#1883
Using fs-verity is natural for OSTree because it's file-based,
as opposed to block based (like dm-verity). This only covers
files - not symlinks or directories. And we clearly need to
have integrity for the deployment directories at least.
More background on fs-verity:
So making this truly secure would need a lot more work. Nevertheless,
I think it's time to start experimenting with it. Among other things,
it does finally add an API that makes files immutable, which will
help against some accidental damage.