Design next major version of qubes-builder

[marmarek]

This is a ticket for track things we would like to see in the next major version of qubes-builder. Actual implementation should be tracked in separate issues, linked to this one.

• isolate component builds from the actual qubes-builder environment - preferably using DisposableVMs (currently we have it for template builds only)
• isolate iso build (same as the above)
• simplify configuration and usage - reduce entry barrier
• more rely on existing and maintained tools (this is especially about "legacy" builds which currently prepare chroot for builds more or less using custom scripts) - consider docker/podman? (side note: "legacy" builds are mostly useful for development builds, as the build environment setup is much faster - it doesn't require installing build deps each time)
• clearer separate source (and dependencies) download from the actual build - allow more reliable logging of build inputs and environment, but also allow extra sanitization of inputs (things like checking their reproducibility proofs or plugging in rpmcanon)
• allow semi-trusted maintainers of only specific components, without giving practical unlimited control over other components as a side effect (see examples-configs: set maintainers components qubes-builder#123 (comment) for more details)
• improve updates handling - painful limitations of the current qubes-builder:
  • no support for un-pushing an update (whether current-testing or current repo)
  • cannot move not the most recent update from current-testing to current

[andrewdavidwong]

Now that we have projects, I've turned this into one, per our guideline that every issue must be about a single, actionable thing:

https://github.com/QubesOS/qubes-issues/projects/11

[fepitre]

@andrewdavidwong @marmarek where are we supposed to discuss to the general ideas for the new design?

[andrewdavidwong]

@andrewdavidwong @marmarek where are we supposed to discuss to the general ideas for the new design?

The issue tracker is not the place for discussing general ideas. That should take place on the mailing lists and/or forum before an issue is created. Once the discussion yields a specific actionable proposal, create an issue for it. That way, comments on the issue can be more usefully focused on specific implementation details.

qubes-devel exists precisely for discussing general ideas about topics like this one.

[marmarek]

I don't want to use this issue for discussing the design, but as a place for coordination and reference for this design process. Things like referring to "requirement number 3" in the discussion. And also a place to put the final design when done (at which point the task is complete, the issue can be closed, and actual implementation can be tracked separately).

I don't thing github project is a good fit for this, as those points are not really separate actionable tasks, but rather a set of a requirements for a single task. We could use some separate document for that (wiki? cryptpad? gist?) but I thought using an issue would help keep things organized in one place. @andrewdavidwong do you have some alternative proposal?

[andrewdavidwong]

I don't want to use this issue for discussing the design, but as a place for coordination and reference for this design process. Things like referring to "requirement number 3" in the discussion. And also a place to put the final design when done (at which point the task is complete, the issue can be closed, and actual implementation can be tracked separately).

I don't thing github project is a good fit for this, as those points are not really separate actionable tasks, but rather a set of a requirements for a single task. We could use some separate document for that (wiki? cryptpad? gist?) but I thought using an issue would help keep things organized in one place. @andrewdavidwong do you have some alternative proposal?

Ah, now that I understand what you have in mind, I think it's fine to use an issue for this. There are a few different things that can sometimes seem similar:

1. A "bucket" for holding several issues. Sometimes people try to create a "meta-issue" for this, but it's better just to use a "project."
2. A discussion area or collaborative scratchpad. People also try to use issues for this, but a mailing list, forum, or actual shared document are better suited.
3. A to-do item for creating an intangible deliverable, such as a design. An issue is appropriate for this, because it is actionable, focused on one thing, and has clearly-defined end conditions. This may not be obvious at first, because usually the deliverable is code, not something intangible like a design, but a design is just as legitimate of a deliverable. It sounds like this is what you have in mind.

The way the issue description is phrased made me think you only wanted (1), but it sounds like you actually want (3), which is fine by me.

[ydirson]

Maybe we could add a items for documentation, both for using it as a user just willing to modify/build existing packages (which the current version has some of, though incomplete), and for using it as an integrator of new packages (which AFAIK is not documented for the current version).

[marmarek]

builderv2 already exists, and most of the above points are covered.