Improve template distribution mechanism

[andrewdavidwong]

Create a tool to manage installed templates. It should support:

• listing available templates, including new versions of installed templates (but clearly mark which are installed and which are not)
• download new templates, or updated version of installed template (again, with clear distinction of those modes, to not accidentally override already installed template)
• install downloaded template - either as a new template, or override existing one
• should also cover qubes-dom0-update combined --action=upgrade --action=reinstall command #4518 aka override template X with its latest version (regardless if it's the same version as currently installed or not)
• all downloaded packages should have signature checked before analyzing their content
• listing templates should also have machine-readable output format (may be separate option), so it would be easier to build new GUI around it
• the tool should not assume running in dom0 - it should be possible to run it also from MgmtVM with appropriate Admin API access (it's true about all the qvm-* tools, so as long the new tool use only that, it should be fine)
• (optional) select storage pool to use for installing template into

Template packages should not be recorded in rpm database of dom0 for multiple reasons:

• rpm (and yum/dnf) would try to update those packages automatically, which in most cases leads to data loss
• rpm independently keep track of template files, which makes removing, renaming and other such actions complicated (this is why right now qvm-remove refuse to remove rpm-installed templates)

Additionally, processing rpm package should be safe for dom0 in sense that no template-provided scripts are executed in dom0. It should be safe to install arbitrary template package regardless of its origin, without risking dom0 compromise. Template package should only influence template itself (its root img and possibly selected settings).

The tool should use "updatevm" to access network repository, where needed.

There is already qvm-template-postprocess tool that given directory with root.img and default menu entries list (whitelisted-appmenus.list) import it into qubes (create appropriate TemplateVM, import root.img into appropriate pool, setup menu entries etc). The tool may be adjusted as part of this task, but the main work is about providing input to that tool.

Based on discussion on IRC and below, it should be ok to stick with yum/dnf repository as distribution method, but create separate tool to extract template rpm packages. That tool should ignore any rpm scripts inside and unexpected files, to satisfy "safe to install arbitrary template package" requirement.
Parts of qubes-dom0-update tool may be useful, but that's not requirement.

---

Background:
On 2016-12-23 02:49, Marek Marczykowski-Górecki wrote:

On Thu, Dec 22, 2016 at 11:56:13PM -0800, Andrew David Wong wrote:

On 2016-12-22 07:33, J. Eppler wrote:

1. How can I search for qubes-templates in dom0 rpm repository? The qubes-dom0-update tool excludes all templates from a search.

$ sudo qubes-dom0-update --action=search qubes-template

This will pass through the terminal output from the UpdateVM, so even
though templates are excluded in the dom0 dnf command, you'll still get
the search results (in red text by default).

Remember that you won't see community templates unless you enable their repo:

$ sudo qubes-dom0-update --enablerepo=qubes-templates-community --action=search qubes-template

Reading red text is not a really good solution from an usability perspective. What are the reasons to exclude templates from a search in the first place?

Actually, after further testing, I've found that this doesn't work. Even with --enablerepo=qubes-templates-community, I'm only getting the Fedora and Arch templates in search results (not Whonix or Debian). Without --enablerepo=qubes-templates-community, I only get the Fedora templates in the search results. So, this method doesn't work. Moreover, the results are inconsistent, since Debian is in the official templates-itl repo, whereas Arch is in the templates-community repo.

Marek, any idea what's going on here?

Already installed templates are excluded from every qubes-dom0-update
operation. Mostly to not accidentally upgrade such package - which would
override all the changes made inside (including installed packages,
applied updates etc). Currently the only exception is "reinstall"
action, with one specific template given as an argument. There is a plan
to extend this to some other actions:
#2527

But in general, we need some better mechanism to distribute templates,
this is one of the reasons...

[Jeeppler]

Guix is a functional transactional package manager and it can be installed besides any existing package manger on Linux. Guix is self contained and focuses on reproducible builds. I think Guix could be used, slightly modified (chroot in disposable vm), to manage template packages.

[marmarek]

Some better description - requirements:

template package format

• a container for root.img
• compressed
• integrity protected (signature)
• possibly a minimal metadata (description, version, preferred virt_mode, some other settings?)

repository format

• a repository for packages above
• searchable/listable
• understands different versions of the same package (optional)
• integrity protected (signature)

tools

• builder-side:
  • generate template package and repository offline, then upload to mirrors as static files
• client-side:
  • download repository metadata (info about templates) without downloading actual packages
  • download selected template
  • verify signature
  • extract root.img and possibly other metadata, then create actual TemplateVM using Admin API

The above is just a brain dump, nothing set in stone. We prefer an existing format with tools meeting as much as possible the above points. While tooling could be adjusted, IMO we shouldn't really change the format itself.

One possibility to consider is something based on OVF. Pros include being well adopted standard, but on the cons side it's quite complex (if going this way, we'd definitely need to use DispVM for actual package extraction).

Another option is to stick with rpm+yum/dnf as package and repository format, but do not really install those packages (in terms of rpm database in dom0), but only extract root.img from there.

[marmarek]

Another option is to stick with rpm+yum/dnf as package and repository format, but do not really install those packages (in terms of rpm database in dom0), but only extract root.img from there.

After some discussion, this looks to be a good idea.

[adrelanos]

Maybe it should be possible to be able to update (some of?) the metadata but without requiring to upgrade to the actual template image?