-
Notifications
You must be signed in to change notification settings - Fork 551
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add vTPM specification #920
Conversation
@stefanberger is this interface stable from the kernel POV? |
specs-go/config.go
Outdated
// VTPM is used to hold the configuration state of a VTPM | ||
type VTPM struct { | ||
// The directory where the TPM emulator writes the TPM state to | ||
Statepath string `json:"statepath"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
StatePath string json:"statePath"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed.
specs-go/config.go
Outdated
// The directory where the TPM emulator writes the TPM state to | ||
Statepath string `json:"statepath"` | ||
// Whether to create a certificate for the VTPM | ||
Createcerts bool `json:"createcerts"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CreateCerts bool json:"createCerts"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed.
specs-go/config.go
Outdated
// Whether to create a certificate for the VTPM | ||
Createcerts bool `json:"createcerts"` | ||
// Version of the TPM | ||
Vtpmversion string `json:"vtpmversion"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
VTPMVersion string json:"vtpmVersion"
config.md
Outdated
}, | ||
"vtpms": [ | ||
{ | ||
"Statepath": "/var/run/runc/ubuntu/tpm12_1", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these should be the same as the json tags on the go struct
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
config-linux.md
Outdated
```json | ||
"vtpms": [ | ||
{ | ||
"Statepath": "/var/run/runc/ubuntu/tpm12_1", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lowercase like the json tags
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
e8213b4
to
3f1ca50
Compare
@crosbymichael I updated the patches to reflect the requested changes. I added 3 fields to the VTPM struct in config.go. In runc I have a lot more fields. I suppose this is ok. |
@crosbymichael The kernel has a vTPM proxy driver that we would be using in runc. Its interface is stable. New ioctl's may be added, but that shouldn't be a problem. The interface of the TPM emulator implementation runc is using, swtpm, is also stable. |
@stefanberger why does runc have more fields than the spec? |
@crosbymichael It's keeping data such as the created major/minor numbers of the character device, file descriptor, etc. Here's where it resides then. https://github.com/stefanberger/runc/blob/vtpm/libcontainer/vtpm/vtpm.go#L21 |
@stefanberger ok, so those are implementation details, not input from the user |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Assorted minor copy-edits.
config-linux.md
Outdated
## <a name="configLinuxVTPMs" />vTPMs | ||
|
||
**`vtpms`** (array of objects, OPTIONAL) lists a number of emulated TPMs that | ||
will be made available to the container. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: this sentence is split over two lines, but it should be a single line.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed.
config-linux.md
Outdated
@@ -384,6 +384,31 @@ The following parameters can be specified to set up the controller: | |||
} | |||
``` | |||
|
|||
## <a name="configLinuxVTPMs" />vTPMs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed.
config-linux.md
Outdated
|
||
Each entry has the following structure: | ||
|
||
* **`statePath`** *(string, REQUIRED)* - full path to a directory where the vTPM is to write its persistent state into |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed.
config-linux.md
Outdated
* **`vtpmVersion`** *(string, OPTIONAL)* - The version of TPM to emulate; either 1.2 or 2; default is 1.2 | ||
* **`createCerts`** *(boolean, OPTIONAL)* - Whether to create certificates for the vTPM | ||
|
||
The `statePath` MUST be unique per container. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: suggested copy edit:
If the
vtpms
array contains duplicated entries with the samestatePath
, the runtime MUST generate an error.
to echo (ish) our existing language here.
specs-go/config.go
Outdated
// Whether to create a certificate for the VTPM | ||
CreateCerts bool `json:"createCerts"` | ||
// Version of the TPM | ||
VTPMversion string `json:"vtpmVersion"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This (and CreatCerts
) should be omitempty
because they are optional.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
specs-go/config.go
Outdated
@@ -161,6 +161,8 @@ type Linux struct { | |||
// IntelRdt contains Intel Resource Director Technology (RDT) information | |||
// for handling resource constraints (e.g., L3 cache) for the container | |||
IntelRdt *LinuxIntelRdt `json:"intelRdt,omitempty"` | |||
// VTPM configuration | |||
VTPMS []*VTPM `json:"vtpms"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
None of our other arrays are arrays of pointers. Can we use []VTPM
here?
And with aggressive namespacing (#567) we probably want []LinuxVTPM
or some such.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also added 'omitempty' now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The runtime-spec and runc differ significantly in this regard. In runc there are arrays with pointers, such as Mounts []*Mount
and Devices []*Device
. How do the runtime-spec and runc relate to each other in regards to these files ? If I don't make these changes in runc but leave it at *VTPM, I suppose this will be alright (not to create more churn...)?
config-linux.md
Outdated
|
||
* **`statePath`** *(string, REQUIRED)* - full path to a directory where the vTPM is to write its persistent state into | ||
* **`vtpmVersion`** *(string, OPTIONAL)* - The version of TPM to emulate; either 1.2 or 2; default is 1.2 | ||
* **`createCerts`** *(boolean, OPTIONAL)* - Whether to create certificates for the vTPM |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reformulated that a bit to "If true then create certificates for the vTPM, default to false."
config-linux.md
Outdated
|
||
Each entry has the following structure: | ||
|
||
* **`statePath`** *(string, REQUIRED)* - full path to a directory where the vTPM is to write its persistent state into |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From opencontainers/runc#1591, it looks like you're writing a swtpm
PID and log files to this directory. But this spec PR doesn't explain that, so folks consuming a generic runtime (which may or may not be runc) won't be able to rely on that. Do you have a use-case for the runtime-caller reaching into this directory to check those files? If so, I think you should specify the files here. If not, I think we can drop this property from runtime-spec, and generate it on the fly (or based on runtime-specific options, like runc's existing --root
option) instead of having it be configured here.
925a73d
to
3bd6071
Compare
specs-go/config.go
Outdated
// VTPM is used to hold the configuration state of a VTPM | ||
type LinuxVTPM struct { | ||
// The directory where the TPM emulator writes the TPM state to | ||
StatePath string `json:"statePath"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you really need a state path at all? Can this just be an implementation detail of the runtime and have it handle it internally?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it should be possible to remove it.
config-linux.md
Outdated
* **`vtpmVersion`** *(string, OPTIONAL)* - The version of TPM to emulate, either 1.2 or 2; default is 1.2. | ||
* **`createCerts`** *(boolean, OPTIONAL)* - If true then create certificates for the vTPM, defaults to false. | ||
|
||
The `statePath` MUST be unique per container. If the `vtpms` array contains duplicate entries with the same `statePath`, the runtime MUST generate an error. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These two sentences should be on two lines.
And I think the two sentences you have for the statePath
list entry above should be as well:
* **`statePath`** *(string, REQUIRED)* - a directory for persisting vTPM state.
This value MUST be an absolute path.
(with a four-space indent, #495, #846), although the current spec is not always consistent about list entries (e.g. here we have two sentences in one list-entry line).
Also, existing examples use a link for “generate an error” (e.g. see here).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your patience. Fixed.
4550cc0
to
04d0238
Compare
Is this needed at the runtime level because of interactions with the tpm and user namespaces? Sorry, I'm just trying to understand this more and hope I'm not asking stupid questions. |
@crosbymichael I am not sure what your latest comment ('Is this needed at the runtime level...')is referring to. |
@stefanberger why couldn't this be done outside of the runtime(runc) and just configured the spec to add the new device to the container? |
Looking over this PR and opencontainers/runc#1591, I have some slight concerns about making the On the other hand, with that increased flexibility, you'd have more moving parts. If we're ok claiming that the swtpm or other runtime-chosen emulator is a good enough for anyone using |
Is there a |
@wking So the kernel API does take a VTPMVersion as a parameter of an ioctl() but doesn't care about creation of certificates, which is a configuration parameter to the emulator. So in the case of vTPM we may not just have kernel parameters but also emulator parameters that need to be passed. Another possible parameter to the emulator would be whether the emulator is supposed to encrypt the files the vTPM writes out. On input this could be a boolean and runc creates a random key or one could pass an (AES) key directly. |
@wking There's no |
On Mon, Sep 11, 2017 at 09:20:57PM +0000, Stefan Berger wrote:
So in the case of vTPM we may not just have kernel parameters but
also emulator parameters that need to be passed.
What do we gain by making the runtime a middleman between the emulator
and the runtime-caller? It seems simpler to have the caller:
1. Setup whichever emulator they want however they want.
2. Put the resulting character device under the container's root.path
(or add it to linux.devices).
3. Call ‘create’ to launch the container.
Then there's no need to tell the runtime about vTPM at all, it's just
passing a device through to the container like all the other devices
it passes through.
|
As an example of this in another context, making the runtime a middleman for pseudoterminal creation allows you to create the pseudoterminal pair from a |
@wking @crosbymichael dumb question: if we take the runtime, which I suppose you are referring to is represented by this code base here, out, does that mean we wouldn't have anything vTPM related in this repository? Maybe I am not following the discussion correctly, but I suppose that at least the config.json would have to contain the |
Well, I could see you having:
But you can also accomplish the same thing by
But you wouldn't need that if the caller was setting up the emulator. |
@wking If we express a vTPM instance as shown above with a device on the level of the runtime-spec, then how do we represent it at the level of runc ? Can runc extend the JSON? |
Runc can store additional information in its libcontainer config. But why would it need to? If the caller is managing swtpm and the devices outside the runtime, there's nothing for the runtime to do. If you want a tighter binding, you could reroll opencontainers/runc#1591 into a third-party hook tool, with a setup command for a pre-start hook and a teardown command for a post-stop hook. But I don't see anything in a quick skim of the runc PR, except maybe checkpoint blocking, that cannot be handled by hooks or other external-to-the-runtime code. Am I missing something? Can you provide more details on the checkpoint breakage? |
@wking Who is the 'caller' in this case? ('If the caller is managing swtpm and the device...'). Do you want to support vTPM on the runc level or push to even higher levels? I assumed that checkpointing includes migration as well. With an attached vTPM its state would have to be migrated along with the state of the container, which isn't implemented. |
@wking FYI: I am also working on namespacing of IMA. There I am hooking up an IMA namespace with a virtual TPM instance and the vTPM receives the TPM commands (PCR Extends) from IMA that would normally go to the hardware TPM. If we wanted to support that on the runc level some day, then there should not only be support for an IMA namespace on that level but also to hook up a vTPM to it. |
The runtime-caller (e.g., see the steps I floated here).
I want to push it up to higher levels, unless we have a reason for embedding it in the runtime (like we already do for network setup).
That makes sense. But I don't see a reason why emulator checkpointing couldn't happen at higher levels too, so it's not a reason for putting this in the runtime vs. higher levels.
Depending on the details, this could end up being like network-namespace setup (which we punt to higher levels) or mount-namespace setup (which we handle in the runtime via |
@wking To resume this discussion. I integrated vTPM (with IMA namespacing) into Docker-CE 17.12. As part of that I found it to be necessary to support the following runtime spec for supporting a vTPM:
I extended RunC with vTPM. When running RunC with vTPM support in Docker-CE one surprise was that when running 'docker restart' the vTPM state path was deleted and thus all the persistent state of the vTPM got lost. This for example required the addition of the 'StatePathIsManaged' field in this patch here, which basically helps us restart a container on the docker level without erasing the state like we do when we delete the container on the Docker level or using Further, I find it necessary to integrate the vTPM into RunC also because of RunC spawning an IMA namespace (namespacing of IMA is not upstreamed) and to hook a vTPM to the IMA namespace I need to be able to transfer the vTPM device to the IMA namespace by calling an ioctl on the vTPM device's file descriptor. I doubt that this would be possible if vTPM was created in a hook like where we can setup networking using netns. |
That patch is making runc's removal of the vTPM state path conditional. But if all vTPM handling happens in higher layers, runc would never be deleting the vTPM state path, so I don't see an issue there.
I can't find code for this. Can you link to it?
I see no reason why this couldn't happen between |
house keeping: |
I think I had tried to answer some of the issues here.. I haven't looked at this in a while but my arguments for supporting vTPM on the runc level rather than higher layers doing it would be:
|
needs rebase |
e0768b8
to
1fe6983
Compare
The update I pushed today adds all those fields that are needed to support the vTPM features implemented in runc PR opencontainers/runc#1591 |
@crosbymichael / @wking are you able to recommence / continue your reviews please. As @vbatts points out, vTPM support would be immensely useful for measuring trust of containers. |
Add the vTPM specification to the documentation, config.go, and schema description. The following is an example of a vTPM description that is found under the path /linux/resources/vtpms: "vtpms": [ { "statePath": "/var/lib/runc/myvtpm1", "vtpmVersion": "2", "createCerts": false, "runAs": "tss", "pcrBanks": "sha1,sha512" } ] Signed-off-by: Stefan Berger <[email protected]>
This extension of LinuxDevice adds support for making a device available under a different path than it is seen on the host. The device path on the host may for example be /dev/foo and it may be made available under /dev/bar inside a container. Signed-off-by: Stefan Berger <[email protected]>
} | ||
|
||
// LinuxDevice represents the mknod information for a Linux special device file | ||
type LinuxDevice struct { | ||
// Path to the device. | ||
Path string `json:"path"` | ||
// Path of passed-through device on host | ||
Devpath string `json:"devpath"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this needed for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So that /dev/tpm10 on the host can appear as /dev/tpm0 inside the container.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to know the host path or the entity creating the config.json can get the host device major / minor? For e.g. docker/podman have this feature:
sudo podman run --rm -it --device=/dev/null:/dev/mynull fedora:32 sh
sh-5.0# ls -l /dev/
total 0
crw--w----. 1 root tty 136, 0 Aug 6 21:35 console
lrwxrwxrwx. 1 root root 11 Aug 6 21:35 core -> /proc/kcore
lrwxrwxrwx. 1 root root 13 Aug 6 21:35 fd -> /proc/self/fd
crw-rw-rw-. 1 root root 1, 7 Aug 6 21:35 full
drwxrwxrwt. 2 root root 40 Aug 6 21:35 mqueue
crw-rw-rw-. 1 root root 1, 3 Aug 6 21:35 mynull
crw-rw-rw-. 1 root root 1, 3 Aug 6 21:35 null
lrwxrwxrwx. 1 root root 8 Aug 6 21:35 ptmx -> pts/ptmx
drwxr-xr-x. 2 root root 0 Aug 6 21:35 pts
crw-rw-rw-. 1 root root 1, 8 Aug 6 21:35 random
drwxrwxrwt. 2 root root 40 Aug 6 21:35 shm
lrwxrwxrwx. 1 root root 15 Aug 6 21:35 stderr -> /proc/self/fd/2
lrwxrwxrwx. 1 root root 15 Aug 6 21:35 stdin -> /proc/self/fd/0
lrwxrwxrwx. 1 root root 15 Aug 6 21:35 stdout -> /proc/self/fd/1
crw-rw-rw-. 1 root root 5, 0 Aug 6 21:35 tty
crw-rw-rw-. 1 root root 1, 9 Aug 6 21:35 urandom
crw-rw-rw-. 1 root root 1, 5 Aug 6 21:35 zero
sh-5.0#
The config.json snippet for this:
"devices": [
{
"path": "/dev/mynull",
"type": "c",
"major": 1,
"minor": 3,
"fileMode": 8630,
"uid": 0,
"gid": 0
}
],
* **`statePathIsManaged`** *(string, OPTIONAL)* - Whether runc is allowed to delete the TPM's state path upon destroying the TPM, defaults to false. | ||
* **`vtpmVersion`** *(string, OPTIONAL)* - The version of TPM to emulate, either 1.2 or 2, defaults to 1.2. | ||
* **`createCerts`** *(boolean, OPTIONAL)* - If true then create certificates for the vTPM, defaults to false. | ||
* **`runAs`** *(string, OPTIONAL)* - Under which user to run the vTPM, e.g. 'tss'. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it make sense to run this as the container user or it is typically set to a separate tss
user?
@stefanberger are you still working on this PR? |
No, I am not working on it. |
Add the vTPM specification to the documentation, config.go, and
schema description. The following is an example of a vTPM description
that is found under the path /linux/resources/vtpms:
Signed-off-by: Stefan Berger [email protected]