Switch to PVH

[rootkovska]

For Qubes 4 we want to move away from using PV as the default method of virtualization in favor of using hw-aided (i.e. SLAT-enforced) virtualization, which currently Xen offers as PVH. The main reason for this is security. We believe SLAT should be less buggy than PV memory virtualization, as e.g. XSA 148 has shown a few months ago. Today most platforms should support SLAT, which was not the case 6 years ago when we originally chose PV over HVM. HVM without SLAT requires clunky Shadow Page Tables virtualization, arguably even more complex and error-prone than PV virtualization.

This ticket serves as a central place to track progress of this task, which should include:

• Enabling (PV)GRUB (not PyGRUB!) support in our template-builders
• Adjustments to the Linux GUI agent (get rid of u2mfn module, report gust mfns), and the GUI daemon (we decided we still want to use xc_map_foreign_page for the time being)
• Modify default Xen VM config generated by core (potentially also: apply patch to libvirt to support PVH?)
• Installer/firstboot: detect if the CPU has SLAT support (which also implies VT-x), and refuse to install otherwise (possibly recommend Qubes 3.x stable branch instead).

[marmarek]

Possible problems: PCI passthrough (as currently broken on HVM - #1659)

[Jeeppler]

Does SLAT and PVH improve the guest VM performance (boot time/runtime)?

[marmarek]

Additionally:

• have qubes-hcl-report (also in older releases) check for SLAT

[v6ak]

On a non-VT-d computer, does PVH support PCI passthrough?

(If not, I think it would be safe to use PV fallback, as VMs with PCI passthrough are already somehow privileged on non-VT-d computers.)

[marmarek]

On Wed, Aug 03, 2016 at 11:10:50AM -0700, Vít Šesták wrote:

On a non-VT-d computer, does PVH support PCI passthrough?

(If not, I think it would be safe to use PV fallback, as VMs with PCI passthrough are already somehow privileged on non-VT-d computers.)

The idea is to not support such hardware in Qubes 4.x, and for older
hardware recommend (besides upgrading the hardware...) Qubes 3.x.
Currently, because of large spectrum of supported hardware, statement "I
use Qubes OS" may mean very different security level. This is
specifically bad for less technical users, who can't reason about
implications of not having particular feature.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Some major problem: it looks like populate-on-demand doesn't work - domain crashes when started with memory < maxmem. This is a blocker for dynamic memory management (qmemman).
Later lowering memory assignment does work.

[marmarek]

Does SLAT and PVH improve the guest VM performance (boot time/runtime)?

Theoretically yes, but haven't tested it yet.

[v6ak]

Thanks for the info. I understand the reason, especially with the limited resources for development. The QSB-24 is, however, confusing. It just mentions SLAT/EPT will be needed, but not VT-d, which might sound like 99 % of Qubes users have CPU compatible with the future Qubes 4.0. When counting Qubes 4.0, it does not sound like this, as non-VT-d-compatible machines seem to be more frequent.

Not supporting VT-d in 4.0 is controversial, but it seems to be somehow justified. Moreover, it does not sound like a practical issue for me if it happens in 4.0. My main point is the communication: Please make this future incompatibility more visible. Without thinking about implementation of PVH, I would not have any idea that such issue might arise. All Qubes users should have a real chance to realize the adjusted hardware requirements soon enough. There is arguably some time before the last non-VT-d-requiring release (3.2?) gets unsupported (maybe an year), but one buying a new laptop considering Qubes compatibility should know (or have real chance to know) that fact today.

[marmarek]

You're right. We'll update system requirements page soon. But we need to work out some details.

[andrewdavidwong]

The idea is to not support such hardware in Qubes 4.x, and for older hardware recommend (besides upgrading the hardware...) Qubes 3.x.

Does this mean changing our support period for prior versions? Currently it's "six months after each subsequent major or minor release." If so, this is another thing we should make sure to communicate clearly.

[rootkovska]

On Wed, Aug 03, 2016 at 01:55:21PM -0700, Marek Marczykowski-Górecki wrote:

Some major problem: it looks like populate-on-demand doesn't work - domain
crashes when started with memory < maxmem. This is a blocker for dynamic
memory management (qmemman). Later lowering memory assignment does work.

Shall we create a new ticket for tracking of this? Obvious problem with "VM must
start with maxmem" is that it might be hard to launch more then a few VMs.

j.

[v6ak]

That's great. I am in favour of communicating this via blog/mailinglist/Twitter/etc.

[marmarek]

Based on current state of PVH in Xen, for Qubes 4.0 we'll go with standard HVM, then later switch to PVHv2 when ready.

[Jeeppler]

@marmarek do you really want to switch to pure HVM? I think for Linux it would make sense to use PVHVM: A HVM which uses hardware virtualization extensions and PV driver for networking and disk I/O. (source: https://wiki.xen.org/wiki/Xen_Project_Software_Overview)
I mean the HVMlite implementation would add a way to directly boot into the kernel (source: http://fossies.org/linux/xen/docs/misc/hvmlite.markdown) which PVHVM does not offer.

[marmarek]

Yes, of course I meant HVM with PV drivers.

[v6ak]

So, will stubdomains be needed? IIUC, PVHVM does not need this, while HVM+PV does.

And when there is a full HVM (e.g. a Windows VM) which needs a stubdomain, what domain type will it use? If it will run in a PV (like today's PV do), then the issue with PV security is solved only partially, especially when considering QEMU as insecure.

[marmarek]

See linked discussion on xen-devel - in short PVH isn't usable yet.

[marmarek]

If it will run in a PV (like today's PV do), then the issue with PV security is solved only partially, especially when considering QEMU as insecure.

Yes, that's unfortunately right.

[marmarek]

PVHv2 status update (after talking in person to Xen people at FOSDEM): there is still slight disagreement on details of Linux support for PVHv2 (AFAIR about CPU hotplug). Should be resolved and implemented this year, but probably will take more than 1-2 months. This is all about PVHv2 without PCI passthrough, which is another story. Which means there wont be PVHv2 Linux VMs in Qubes 4.0.

[Jeeppler]

Phoronix reported that Xen developer submitted patches for PVHv2 (formerly known as HVMLite) to the 4.11 kernel: Xen Changes For Linux 4.11: Lands PVHv2 Guest Support

Here is the Linux Kernel-Archive pull request message: [GIT PULL] xen: features and fixes for 4.11-rc0

[marmarek]

Good to know, thanks!

[marmarek]

Adjustments to the Linux GUI agent (get rid of u2mfn module, report gust mfns), and the GUI daemon (we decided we still want to use xc_map_foreign_page for the time being)

Actually, it turned out this step isn't needed. But it is still nice to have.

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-dom0-4.0.16-1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[Jeeppler]

@marmarek what is the final decision? Are you switching entirely to HVM or is PVH still an option?

[marmarek]

Where possible (no PCI devices, Linux >= 4.11), we're switching to PVH.

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-dom0-4.0.21-1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[iamahuman]

PVH + PCI passthrough: http://markmail.org/message/xabm6msg6amgjzrd