Firecracker VMM support #1044

mcastelino · 2018-12-17T22:50:24Z

Initial Firecracker VMM support. This vendors in the firecracker go sdk. It also brings in the initial code to integrate firecracker with Kata.

virtcontainers/fc.go

bergwolf · 2018-12-18T01:50:17Z

Thanks for revendoring @mcastelino !

sboeuf · 2018-12-18T18:50:56Z

/test

In order to let the user choose firecracker hypervisor instead of QEMU (from the configuration.toml), let's add it to the list of supported hypervisors. Fixes kata-containers#1042 Depends-on: github.com/kata-containers#1044 Signed-off-by: Eric Ernst <[email protected]>

Gopkg.toml

virtcontainers/fc.go

virtcontainers/hypervisor.go

In order to let the user choose firecracker hypervisor instead of QEMU (from the configuration.toml), let's add it to the list of supported hypervisors. Fixes kata-containers#1042 Depends-on: github.com/kata-containers#1044 Signed-off-by: Eric Ernst <[email protected]>

Vendor in all firecracker dependencies. This allows virtcontainers to pull call the firecracker REST API. Signed-off-by: Manohar Castelino <[email protected]>

virtcontainers/resource_storage.go

mcastelino · 2018-12-20T03:28:41Z

@bergwolf @sboeuf @jodh-intel @egernst all the code review comments should have been addressed.

When this PR merges will have support for docker+kata+firecracker.

The merge order needs to be
#1045
#1044
#1043

This also requires the latest agent kata-containers/agent@8e48125 in the kata image which includes

It also needs the kernel with virtio-mmio support which was included in
kata-containers/packaging#272

The configuration toml needs to have the following set

[hypervisor.firecracker]
path = "/usr/bin/firecracker"
kernel_params = ""
block_device_driver = "virtio-mmio"
disable_block_device_use = false
use_vsock = true
...
[runtime]
internetworking_model="tcfilter"

bergwolf

Mostly looks good. Just have some minor comments.

virtcontainers/fc.go

bergwolf · 2018-12-20T08:11:26Z

virtcontainers/fc.go

+		//The drive placeholder has to exist prior to Update
+		return nil, fc.fcUpdateBlockDrive(*devInfo.(*config.BlockDrive))
+	default:
+		fc.Logger().WithFields(logrus.Fields{"devInfo": devInfo,


This means endpoint hotplug is not supported. Do we need to return failure here otherwise kata-runtime network add and kata-netmon might be confusing?

Makes sense. Added this. Will test and see if results in any top level failures.

bergwolf · 2018-12-20T08:14:07Z

virtcontainers/fc.go

+}
+
+func (fc *firecracker) disconnect() {
+	fc.state.set(notReady)


Why is fc state changed here? I thought the state should reflect the real state of the FC guest. And the disconnect() call only means to close the current runtime <-> hypervisor control channel connection, which is fc.fcClient.

This is just a logical disconnect. We may still need the client connection for cleanup. We can change this later.

jodh-intel

Hi @mcastelino - this is looking really good - just a few nits + @bergwolf's questions (which I agree with) to deal with I think.

virtcontainers/fc.go

jodh-intel · 2018-12-20T08:57:43Z

virtcontainers/fc.go

+// firecracker guest VM.
+// We attach a pool of placeholder drives before the guest has started, and then
+// patch the replace placeholder drives with drives with actual contents.
+const fcDiskPoolSize = 8


Nit: Could be moved up to the const's above.

is this a limitation in firecrackers? why 8 ?

done. fixed

virtcontainers/fc.go

jodh-intel · 2018-12-20T09:15:52Z

virtcontainers/fc.go

+
+	// Use the global block index as an index into the pool of the devices
+	// created for firecracker.
+	driveID := "drive-" + strconv.Itoa(drive.Index)


Nit: A small amount of duplication with createDiskPool(). We could create something like the following:

func (fc *firecracker) driveParams(index int) { driveID := fmt.Sprintf("drive-" + strconv.Itoa(index) driveParams := ops.NewPutGuestDriveByIDParams() driveParams.SetDriveID(driveID) : }

... but it looks like we'd need to handle the models.Drive{} vs. models.PartialDrive{} logic for that.

Do you mind if I close this later. All other review comments have been merged.

virtcontainers/fc.go

In order to let the user choose firecracker hypervisor instead of QEMU (from the configuration.toml), let's add it to the list of supported hypervisors. Fixes kata-containers#1042 Depends-on: github.com/kata-containers#1044 Signed-off-by: Eric Ernst <[email protected]>

mcastelino · 2018-12-20T19:11:56Z

@sboeuf @egernst Most review comments addressed.

A test branch with all PRs incorporated is here https://github.com/mcastelino/runtime-1/tree/topic/fc-final-hopefully is anyone wants to try it out. With this we have full docker support with proper support for termination.

Initial Support for the firecracker VMM Note: - 9p is unsupported by firecracker - Enable pseudo hotplug block device hotplug capability Initially, this will be a pseudo capability for Firecracker hypervisor, but we will utilize a pool of block devices and block device rescan as a temporary workaround. Fixes: kata-containers#1064 Signed-off-by: Eric Ernst <[email protected]> Signed-off-by: Samuel Ortiz <[email protected]> Signed-off-by: Archana Shinde <[email protected]> Signed-off-by: Sebastien Boeuf <[email protected]> Signed-off-by: Manohar Castelino <[email protected]>

Add firecracker as a supported hypervisor. This connects the newly defined firecracker implementation as a supported hypervisor. Move operation definition to the common hypervisor code. Signed-off-by: Manohar Castelino <[email protected]>

Unlike QEMU firecracker cannot accept a fd as part of the REST API. Close the vsock vhostfd close to the point where we launch the VM. Note: This is still racy. Signed-off-by: Manohar Castelino <[email protected]>

Use the firecracker rescan logic to update the pre-attached drive. This allows us to emulate hotplug. Initially the drive backing stores are set to empty files on the host. Once the actual block based device or file is available swap the backing store. The rescan needs to be issued iff the VM is running. Signed-off-by: Manohar Castelino <[email protected]>

Because firecracker currently does not support a proper stop from the caller, and because we don't want the agent to initiate a reboot to shutdown the VM, the simplest and most efficient solution at the moement is to signal the VM process with SIGTERM first, followed by a SIGKILL if the process is still around. Signed-off-by: Sebastien Boeuf <[email protected]>

sboeuf · 2018-12-20T20:21:33Z

/test

sboeuf · 2018-12-20T20:22:46Z

/test

sboeuf · 2018-12-20T23:01:53Z

All green except metrics that is pending... Let's merge it as this introduces only new code related to firecracker and should not impact metrics anyway.

In order to let the user choose firecracker hypervisor instead of QEMU (from the configuration.toml), let's add it to the list of supported hypervisors. Fixes kata-containers#1042 Depends-on: github.com/kata-containers#1044 Signed-off-by: Eric Ernst <[email protected]>

mcastelino force-pushed the topic/fc-vendor branch 2 times, most recently from 42d5faa to a4f0086 Compare December 17, 2018 23:26

This was referenced Dec 17, 2018

Add support for firecracker in addition to QEMU as a supported hypervisor in Kata. #1031

Closed

Adding: Firecracker Client API Support #1026

Closed

bergwolf reviewed Dec 18, 2018

View reviewed changes

virtcontainers/fc.go Outdated Show resolved Hide resolved

virtcontainers/fc.go Outdated Show resolved Hide resolved

virtcontainers/fc.go Outdated Show resolved Hide resolved

mcastelino force-pushed the topic/fc-vendor branch 2 times, most recently from 11c961d to 41b3546 Compare December 18, 2018 18:47

mcastelino changed the title ~~WIP: Firecracker VMM support~~ Firecracker VMM support Dec 18, 2018

sboeuf mentioned this pull request Dec 18, 2018

config: Add firecracker as a supported hypervisor configuration #1043

Merged

jodh-intel reviewed Dec 19, 2018

View reviewed changes

firecracker: vendor in all firecracker dependencies

c34b045

Vendor in all firecracker dependencies. This allows virtcontainers to pull call the firecracker REST API. Signed-off-by: Manohar Castelino <[email protected]>

mcastelino force-pushed the topic/fc-vendor branch from 41b3546 to eedd0e5 Compare December 20, 2018 00:24

egernst reviewed Dec 20, 2018

View reviewed changes

virtcontainers/resource_storage.go Outdated Show resolved Hide resolved

mcastelino force-pushed the topic/fc-vendor branch 4 times, most recently from 5c9e7a1 to a290b0c Compare December 20, 2018 01:02

egernst approved these changes Dec 20, 2018

View reviewed changes

bergwolf reviewed Dec 20, 2018

View reviewed changes

jodh-intel reviewed Dec 20, 2018

View reviewed changes

mcastelino force-pushed the topic/fc-vendor branch from 0a2aec1 to 91a0518 Compare December 20, 2018 19:06

mcastelino force-pushed the topic/fc-vendor branch from 91a0518 to ca4c191 Compare December 20, 2018 19:23

mcastelino and others added 5 commits December 20, 2018 11:54

firecracker: Close the vsock vhostfd

22ebc09

Unlike QEMU firecracker cannot accept a fd as part of the REST API. Close the vsock vhostfd close to the point where we launch the VM. Note: This is still racy. Signed-off-by: Manohar Castelino <[email protected]>

mcastelino force-pushed the topic/fc-vendor branch from ca4c191 to b4c3a2f Compare December 20, 2018 19:56

sboeuf approved these changes Dec 20, 2018

View reviewed changes

sboeuf merged commit 8764fc1 into kata-containers:master Dec 20, 2018

jodh-intel mentioned this pull request Jan 29, 2019

ppc64le: Fix hotplug issue #1186

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Firecracker VMM support #1044

Firecracker VMM support #1044

mcastelino commented Dec 17, 2018

bergwolf commented Dec 18, 2018

sboeuf commented Dec 18, 2018

mcastelino commented Dec 20, 2018

bergwolf left a comment

bergwolf Dec 20, 2018

mcastelino Dec 20, 2018

bergwolf Dec 20, 2018

mcastelino Dec 20, 2018

jodh-intel left a comment

jodh-intel Dec 20, 2018

devimc Dec 20, 2018

mcastelino Dec 20, 2018

jodh-intel Dec 20, 2018

mcastelino Dec 20, 2018

mcastelino commented Dec 20, 2018

sboeuf commented Dec 20, 2018

sboeuf commented Dec 20, 2018

sboeuf commented Dec 20, 2018

Firecracker VMM support #1044

Firecracker VMM support #1044

Conversation

mcastelino commented Dec 17, 2018

bergwolf commented Dec 18, 2018

sboeuf commented Dec 18, 2018

mcastelino commented Dec 20, 2018

bergwolf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcastelino commented Dec 20, 2018

sboeuf commented Dec 20, 2018

sboeuf commented Dec 20, 2018

sboeuf commented Dec 20, 2018