Port Qubes to ppc64 [3 bitcoin bounty]

[Rspigler]

QubesOS is the most secure operating system available, by far. However, it unfortunately only runs on the x86 instruction set, which runs on unauditable and insecure firmware. The Power Architecture is a much more secure ISA. Products like the Talos II (edit: and now much more affordable Blackbird) with the Power9 CPU are fully open, with auditable schematics, firmware, and software - and being able to run QubesOS on such devices would be a huge win for the infosec community.

There are various ways to achieve this compatibility, so I thought that this issue could be a way to track them/discuss.

1 - Xen could have a ppc64 port (Raptor Computing Systems has offered free hardware to incentivize)
2 - Using the seL4 microkernel (#3894), which is already looking into supporting the Power Architecture
3 - Qubes' Hypervisor Abstraction Layer (HAL), which utlizes libvirt to support multiple hypervisors, yet currently only supports Xen, could be expanded to support KVM, to run on ppc64.

March 26, 2022: We are now all in agreement for Xen+Power (option 1).

Funds available as of May 7th, 2022:
I (Robert Spigler) have 0.35 bitcoin & Blackbird Bundle
@Leo-LB has pledged 0.8 btc (need to confirm)
Total 1.15 btc

@madscientist159 Has offered to do the Xen port for 2 btc (just Xen port; no Qubes integration yet)

Power Foundation has made a statement of support (https://twitter.com/OpenPOWERorg/status/1504112361975730186?s=20), but this needs to be clarified.

We will be moving from Github -> Gitlab for development.
(https://gitlab.com/groups/xen-project/-/epics/6)

We have made a Mailing List and Matrix Room:
[email protected]; https://lists.riseup.net/www/info/qubes_port
https://matrix.to/#/#qubes-port:matrix.org

We have now adopted this milestone approach for this Port: (done here)

1. Phase 1: 0.65BTC. Build tooling, minimal boot to serial console of a Xen kernel on a single core (no SMP, missing drivers, core locked at 100% power).

2. (Proposed) Phase 1.5: 0.65BTC (Pricing subject to change due to economic fluctuations):
   SMP, some driver integration (possible power state management?) required to get a usable system in preparation for Phase 2

I (Robert) donated 0.65 bitcoin out of my remaining 1 bitcoin bounty to fulfill the Phase 1 requirement. See here

@Rudd-O donated the entirety of his bounty (0.5 bitcoin) towards Phase 1.5. He no longer has any remaining pledge, and Phase 1.5 has 0.15 btc left to fulfill. See here

We are still waiting for @Leo-LB to re-confirm his pledge.

Last updated May 7th, 2022

---

Details/History of Funding Below:

Please see the below chronological updates to funding:

• 1
• 2
• 3

In summary, we have a 3 bitcoin bounty, and an additional 0.5 bitcoin remaining for matching funds (deadline passed with 0.5 matching funds filled out of 1 bitcoin matching funds offered - see here). The match offer expired on July 28th 2021.

Details of the bounty are below:

@Leo-LB paid @shawnanastasio 0.2 btc out of his 1 bitcoin bounty here: #4318 (comment)

@Rspigler (me) paid Shawn 0.5 bitcoin out of his 1.5 bitcoin bounty here. I have also offered hardware (Blackbird mainboard and one 4 core Power9 CPU) for a developer who will use it towards this project. See post here.

@Rudd-O pledged 0.5 bitcoin here (has paid 0).

I (Robert) have a remaining 0.5 matching bitcoin offer that expires on July 28th 2021.

Last updated: July 31st, 2022

[madscientist159]

My vote would be seL4. Not only are they responsive to the threat posed by closed, highly privileged firmware, but the kernel is already being used in very high security installations.

[Rspigler]

I would truly love seL4 as well and hope that is the end goal; however, I believe that that course of action would take the longest of all 3. I would argue that porting Qubes for ppc64 (by way of 1 or 3) is of higher priority than working on substituting the seL4 microkernel for Xen, since that offers the quickest route to Qubes running a truly open platform. The security benefit of seL4 could then come after.

I could be wrong though.

[madscientist159]

Of the two, I suspect 3 is the fastest and probably a decent stopgap solution. 1 would be duplicating effort that could go into the seL4 port IMO.

[jaesharp]

If needed, I can provide a significant quantity of build and test infrastructure for this effort.

[andrewdavidwong]

There are various ways to achieve this compatibility, so I thought that this issue could be a way to track them/discuss.

The qubes-devel mailing list is the appropriate place for this sort of discussion. By contrast, qubes-issues is reserved for actionable issues as described in our issue reporting guidelines. If this issue is to remain open, it needs to be action-oriented rather than just a place for tracking and discussion. I'll change the title to reflect this.

[awokd]

I humbly propose this thread: https://groups.google.com/d/msg/qubes-devel/IFLyyCUbLmQ/_HtgOJK4AQAJ as a good place to continue the discussion!

[Peter-Easton]

I have had some difficulty getting to Google Groups, but I'd like to put my vote in as well for a port to the Talos II, whichever the professionals decide it will take the form of. I'm not well-versed enough in hypervisor design to be able to make any meaningful comment on which one would be best, but on the PPC64el, Debian runs KVM quite well with Libvirt for running other Debian and CentOS guests and is happily usable as a desktop with a minimum of work. I'd be very happy to beta-test a port to the PPC64, if you guys need testers.

Not only that but the Talos Lite is now here, which is about the price of high-end business grade or one of those high-end name-branded gaming laptops, and offers far better upgradeability for the future than a gaming laptop even when we don't look at the owner control/openness aspect of it. For many people interested in and serious about security, the price equivalent of a good quality laptop isn't a high one to pay, especially if the user isn't a firmware hacker that has the skills and know how to create their own "trusted stick" like what Joanna Rutkowska was suggesting for a stateless, verified boot x86 laptop.

[tlaurion]

The OTF funding form is here.

Time to tag developers, fund grant filling people (@mfc!) that might help so the discussion can follow on google groups and the information for the grant gets filled for the next OTF round!

[mfc]

hi all, i don't have any capacity to help out right now, but realistically i don't think OTF would fund such effort.

their target group is non-tech-savvy users in at-risk environments. these users have the current computers they own, they don't have the money/luxury of buying new hardware. Qubes' existing hardware requirements are already a very high barrier for these users and their target audience.

so if anything they would be more interested in funding Qubes development onto phones than Qubes on ppc64.

not stopping you all from submitting a proposal to OTF of course. maybe this is more something for a crowdfunding approach, for example.

[Rspigler]

Understandable. Going ahead with both might be possible. If a crowdfunding approach were to be done, what would be the estimated target goals for the KVM approach, and the seL4 approach? (seL4 could possibly be set as a stretch goal).

[marmarek]

As for KVM:

Well, the first non-trivial task would be research what really needs to be done. There are two main parts here:

1. add KVM support in general
2. solve ppc64 specific problems (including verification of devices isolation etc)

The first one is much more predictable, I think it basically is:

• implementing vchan using KVM-specific mechanism
• implement "driver domains" (running a disk/net/usb backend in separate VM, instead of the host) - this is major feature missing in KVM compared to Xen; there are possible workarounds sacrificing some security
• add support for KVM to GUI agent/daemon - one hypervisor specific part there is accessing actual window composition buffers, using shared memory mechanism provided by particular hypervisor
• adjust startup scripts, libvirt configuration etc
• test everything and fix bugs

I think we should assume 50-100kEUR goal for this part alone. Note it may be also beneficial for other architectures, not only ppc64.

The second part may take several months of work, but also may turn out to be trivial. Require someone knowing ppc64 architecture (which I don't know).

For seL4, I don't even try to provide any estimate, without doing some research on the current state of it and what features crucial for Qubes are missing. See FAQ for some hints.

[Rspigler]

Thank you for your detailed response and all your work on Qubes.

What would your estimate be for approach #1? (ppc64 port of Xen?)

[marmarek]

What would your estimate be for approach #1? (ppc64 port of Xen?)

Take a look at this xen-devel thread

[tlaurion]

GSoC proposal, anyone?

[marmarek]

In theory deadline for setting up GSoC idea page was Feb 6, but in practice period for students to choose projects haven't started yet. Feel free to open PR against gsoc.md. It's ok to have a proposal with very basic description, although this ticket already contain content that could be re-used.

[tlaurion]

Added here.

[Rspigler]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I am placing 1 bitcoin as a bounty for completing this task in porting Qubes.

As multiple developers will most likely be involved, I will divide the bounty between based on work done as I see fit.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEwsYOJ56G8Q1Wl3glNc4P5sIUGCMFAlxjj+AACgkQNc4P5sIU
GCNDWw//YeGQtVH2xc6+qNRezvcBG3M4Ql0DaPFndSPqQ3tBaHsaBoAH7BT6o3Ih
EU/pTCBWDAItC13yot4FejYp2eLRY4VtHPd3wHB+e5WVCZdKbW7Z1F2SWsPJutDo
NSKC+Rb8CEZO0sxSe9CYP7plc6a6GK+WF+ZjgiSN0ISeBf/VMr1V7aXJgZ+cIjSp
eaJCzQ0XSah0oO2brJXUTvPOHUKsFHv3vJLcZag+N5T3HZ5P6Zt6jYorCGRtRBzK
aWVaqGb8MVR9CQUOcLJ8c0ubyRHwwpTWMgr0QQ0QUzwRTBKMyt8jAfPVshI1l/T5
xzr067/sX1cUIWXGp5kpvK32aX9Gb+Sh7e+BqYour4hGiSpq2Vzz2UAakLutM2s4
hkFfnZA1HiZsqS0bxR9/iNi1+dvEBc2U4KUd6ou++p4osve+PpHqWXJGd5Acewgf
fcdhNDPSSif5ON56gZRzvEMhtbE4MXfIdEI8NQGihZfdyO+R+muKxDRDPq9HFK3s
hH1RcAYx6jC1T7E4UENZMzw2n9xKIcBnbkURmx6byzRhZACcX/I6YMciZks5Xxdb
yIItqpTqjW7IdabmDxEYUlG7g+9G8kzZMNFNiqbbxmvsd83O4OUuPOcYj9XAIRtw
w5QmHMVKbPMoKPN9s15i3KXGTnpIQ80E0nGWai8NykgzzzMOH3c=
=Tehn
-----END PGP SIGNATURE-----

[shawnanastasio]

As for KVM:

Well, the first non-trivial task would be research what really needs to be done. There are two main parts here:

1. add KVM support in general

2. solve ppc64 specific problems (including verification of devices isolation etc)

The first one is much more predictable, I think it basically is:

* implementing vchan using KVM-specific mechanism

I have been working on a reimplementation of vchan using KVM with the ivshmem shared memory backend, available here (comments welcome!).

I originally started it to assist in porting individual Qubes projects (mainly qubes-gui-daemon), but it would be great if it was useful for a port of Qubes as a whole.

[marmarek]

I have been working on a reimplementation of vchan using KVM with the ivshmem shared memory backend, available here (comments welcome!).

Here you are: shawnanastasio/libkvmchan#1

[Rspigler]

In the Qubes Architecture spec (Version 0.3), in the rational for choosing Xen over KVM, the authors mention how KVM relies on the Linux kernel to provide isolation. They also argue that Xen is much easier to audit due to a smaller code base, and that the I/O emulator, networking code, and drivers can be moved out of Dom0, further minimizing TCB.

This document was written 2010. These concerns still present?

[marmarek]

Mostly yes. In the meantime, some extra precautions were developed to isolate device emulator (I think the most relevant is seccomp filtering), so it is slightly better now.
But the arguments of a) Linux vs Xen code size, and b) backend drivers on the host system still stands.
The "b" point is especially bad, as it makes it hard to have host system isolated from the outside world (networking, usb etc).

On the other hand, a lot of development have happened on KVM since 2010, some of it very relevant for Qubes. For example much better support for GPU passthrough. And (the main point of this discussion) support for other architectures.

[marmarek]

Very interesting development: KVM support for Xen guests, including PV drivers. Given that Xen PV drivers supports backends in another guest, there is a chance this mechanism will support that too. Especially those patches emulates grant tables and event channels, which are designed with non-host backend in mind.

[madscientist159]

@Rspigler Perfect, sounds like we're in agreement then. I'll consider this locked in and get the work scheduled for the Phase 1 portion, which is the minimal bootable Xen kernel on a POWER9 host. I'm also assuming that if the remaining funding (another 1BTC) does arrive that I'll just be told to keep going on the Phase 2 work.

On the send side, I'd imagine the BTC payment comes after the work is done, but not 100% sure what that looks like (i.e. who would determine that, who makes the final payout, etc.)? In any case there is time to get that all figured out, currently looking at work start in May or so.

Etherpad would be great!

[marcungeschikts]

@marcungeschikts: Maybe we should have a meeting soon

Hi @Rspigler,
I am available whenever you have time.

[Rspigler]

Unfortunately I have been very sick, so did not get the application in on time (we only had a couple days anyway; it would have been difficult to coordinate).

Fortunately we can apply for June 1st submission. I will also look for other orgs that give funding. If anyone has other ideas for funding, please let me know.

@madscientist159 We still just have my 1 bitcoin. Hopefully at some point @Leo-LB and @Rudd-O respond, and we get the other 1.3.

I think it would be fair to give you some payment before your work. We've also known each other for a while so I am not too worried to be honest. I am thinking something like 0.35 BTC now and 0.65 BTC after. Perhaps the 0.65 can be split up further if we set up some specific tasklists. Or, we could convert the bounty to USD right now, and do it at 35% and 65%. Just let me know your thoughts / what you want.

@marcungeschikts I don't think we need another meeting until @olivierlambert, @no-112, and/or @liilac respond about how they wish to contribute; or @madscientist159 gets a considerable amount of work done. Of course if you think differently please let me know.

[madscientist159]

@Rspigler Sorry to hear that. If there's anything I can do to assist with the June 1 submission let me know...

Appreciate the offer on the prepayment, let's do that when I have a firm start date on my calendar. Still looking at May timeframe but need to clear a couple of other projects first. I'd prefer to receive BTC as it may weather inflation a bit better than USD, but if it doesn't the risk is all on my side. 😉

[Rspigler]

Sounds good, thanks!

[olivierlambert]

I already answered on my contribution :) (doing project mgmt via Marc, but also acting as a liaison between Qubes, Xen Project and OpenPOWER). I also gave latest progress on the Matrix channel.

[Rudd-O]

@madscientist159 recommitting!

Where is a Matrix chat I can join so we can chat without me having to keep Github open all day?

[olivierlambert]

Just here: #4318 (comment)

[Rspigler]

@madscientist159 Will do, thanks! I will post the submission draft here once I get it going. Sounds good re: payment.

@olivierlambert Perfect, sorry to tag you over and over. Have you heard from Power Foundation re: clarification on their statement of support? Anyone else from the Xen project who would be interested in helping?

@Rudd-O thank you for confirming the funding! Looking forward to seeing you in the Matrix chat!

I have updated the OP

I am considering starting a website for the project so we can start a social media/marketing push. Not my area of expertise at all (I don't have any social media), but I think it's something that could help a lot.

[olivierlambert]

Xen Project will help to setup the hardware in their colo with the direct access and out of the band too. They will also try to help on giving directions on where to start exactly. This will require discussion on Xen devel but instead of posting thing twice, I'd like to focus on the Matrix channel where I already explained the next steps :)

[madscientist159]

@Rspigler We've been discussing the recent downturn on the Matrix channel, it's looking like there will be a delay while we wait for BTC/USD to recover from its unexpected drop. Thoughts wecome on the channel...

[Rspigler]

Signed Statement of Work (SoW) attached.

Verify first for @madscientist159 sig, then my sig
Xen_SoW_Signed.txt
.

Important updates:

This Port will be done in Phases:

Phase 1: 0.65BTC. Build tooling, minimal boot to serial console of a Xen kernel on a single core (no SMP, missing drivers, core locked at 100% power).

(Proposed) Phase 1.5: 0.65BTC (Pricing subject to change due to economic fluctuations):
SMP, some driver integration (possible power state management?) required to get a usable system in preparation for Phase 2

[Rspigler]

I paid 0.65 bitcoin out of my 1 bitcoin bounty, which completed Phase 1 payment.

I have 0.35 bitcoin left to pay

[Rspigler]

Signed Statement of Work by @Rudd-O
Signed Payment by Rudd
Rudd SoW.txt
Rudd Payment.txt

Rudd has paid his complete bounty (0.5 bitcoin). This is towards phase 1.5, as my 0.65 payment went completely towards Phase 1

Rudd-O no longer has any remaining pledge, and Phase 1.5 has 0.15 btc left to fulfill.

[Rspigler]

@madscientist159 Please confirm both payments.

I have updated OP

[madscientist159]

@Rspigler Confirming receipt of full payment for Phase 1.0 and the 0.5BTC payment toward Phase 1.5. Tentative thinking between parties here is that the differential for Phase 1.5 will be made up in USD in the $5.5k range at Phase 1.5 work start.

We have a development repository here, and I've pushed the very first commit to set up the build system for POWER:
https://gitlab.raptorengineering.com/raptor-engineering-public/xen/

Development might be a bit start/stop to begin with as I am currently wrapping up some other projects, but it'll gain momentum. Currently targeting Q3 2022 for Phase 1.0 completion / Phase 1.5 start.

[tlaurion]

@madscientist159 I see no commit in the branch. Any timeline, updates?

[madscientist159]

@tlaurion I had a sync meeting with @Rspigler not too long ago, the upshot is that it's in progress but a couple of other projects currently have priority due to BTC plummeting in value (basically I'm having to do this in spare time at the moment).

As soon as possible I would like to get an update out to the public repositories, but I don't have a timeframe just yet,

[francuss]

So, what is happening with this project? A lot of activity, a bounty and then... months of silence.
Please confirm this is still alive!

[madscientist159]

@francuss It's still in process, just got pushed back a few months with the fall in BTC value and related impacts. Still driving this project forward with a projected restart date in January...

[crazyrobo]

Any updates?

[madscientist159]

@crazyrobo Raptor now has full time resources on this and patches are starting to flow upstream:

https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg00439.html

[proudmuslim-dev]

Glad to see it!

[jonathancross]

FYI: You can follow @shawnanastasio's progress with Xen / PPC here (ordered by date).

[lestitaniseur]

any update ?