Qubes Integration Proof of Concept #437
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
CI is awfully broken. I'll can look into that shortly. |
deeplow
commented
Jun 7, 2023
deeplow
force-pushed
the
qubes-integration-poc
branch
13 times, most recently
from
ff8b956 to
36de9e7
Compare
apyrgio
requested changes
Jun 7, 2023
apyrgio
reviewed
Jun 7, 2023
deeplow
force-pushed
the
qubes-integration-poc
branch
2 times, most recently
from
ab3ae26 to
e714f8d
Compare
apyrgio
reviewed
Jun 12, 2023
apyrgio
force-pushed
the
qubes-integration-poc
branch
2 times, most recently
from
7a77de5 to
ac4a81b
Compare
4 tasks
deeplow
force-pushed
the
qubes-integration-poc
branch
from
468ffe3 to
6ebd959
Compare
deeplow
added a commit
that referenced
this pull request
Jun 14, 2023
Following a suggestion from @apyrgio [1] to not pollute /usr/local/bin. [1]: #437 (comment)
deeplow
added a commit
that referenced
this pull request
Jun 15, 2023
Also removes exit codes in the qubes wrapper. Following a suggestion from #437 (comment)
apyrgio
force-pushed
the
qubes-integration-poc
branch
2 times, most recently
from
4c362a5 to
51baa34
Compare
apyrgio
force-pushed
the
qubes-integration-poc
branch
3 times, most recently
from
75003e2 to
71b8f67
Compare
deeplow
commented
Jun 17, 2023
apyrgio
force-pushed
the
qubes-integration-poc
branch
2 times, most recently
from
9a75540 to
29612f4
Compare
The files in `container/` no longer make sense to have that name since
the "document to pixels" part will run in Qubes OS in its own virtual
machine.
To adapt to this, this PR does the following:
- Moves all the files in `container` to `dangerzone/conversion`
- Splits the old `container/dangerzone.py` into its two components
`dangerzone/conversion/{doc_to_pixels,pixels_to_pdf}.py` with a
`common.py` file for shared functions
- Moves the Dockerfile to the project root and adapts it to the new
container code location
- Updates the CircleCI config to properly cache Docker images.
- Updates our install scripts to properly build Docker images.
- Adds the new conversion module to the container image, so that it can
be imported as a package.
- Adapts the container isolation provider to use the new way of calling
the code.
NOTE: We have made zero changes to the conversion code in this commit,
except for necessary imports in order to factor out some common parts.
Any changes necessary for Qubes integration follow in the subsequent
commits.
For using in containers, creating a /dangerzone directory is fine but it is more standard to do this in /tmp.
Add a way to check if the code runs (or should run) on Qubes. Refs #451
It seems that there are at least two Python libraries with libmagic support: * PyPI: python-magic (https://pypi.org/project/python-magic/) On Fedora it's `python3-magic` * PyPI: filemagic (https://pypi.org/project/filemagic/) On Fedora it's `python3-file-magic` The first package corresponds to the `py3-magic` package on Alpine Linux, and it's the one we install in the container. The second package uses a different API, and it's the only one we can use on Qubes. To make matters worse, we: * Can't install the first package on Fedora, because it installs the second under the hood: https://bugzilla.redhat.com/show_bug.cgi?id=1899279 * Can't install the second package on Alpine Linux (untested), due to Musl being used instead of libC: https://stackoverflow.com/a/53936722 Ultimately, we need to support both, by trying the first API, and on failure using the other API.
The "document to pixels" code assumes that the client has called it with some mount points in which it can write files. This is true for the container isolation provider, but not for Qubes, who can communicate with the client only via stdin/stdout. Add a Qubes wrapper for this code that reads the suspicious document from stdin and writes the pages to stdout. The on-wire format is the same as the one that TrustedPDF uses.
Add two RPC calls that can run on disposable VMs:
* dz.Convert: This call simply imports the dangerzone package and runs
the Qubes wrapper for the "document to pixels" code. This call is
similar to the way we run the conversion part in a container.
* dz.ConvertDev: This call is for development purposes, and does the
following:
- First it receives the `dangerzone.conversion` module as Python
zipfile. This way, we can quickly iterate on changes on the
server-side part of Qubes, without altering the templates.
- Second, it calls the Qubes wrapper for the "document to pixels"
code, as dz.Convert does.
Add an isolation provider for Qubes, that performs the document
conversion as follows:
Document to pixels phase
------------------------
1. Starts a disposable qube by calling either the dz.Convert or the
dz.ConvertDev RPC call, depending on the execution context.
2. Sends the file to disposable qube through its stdin.
* If we call the conversion from the development environment, also
pass the conversion module as a Python zipfile, before the
suspicious document.
3. Reads the number of pages, their dimensions, and the page data.
Pixels to PDF phase
-------------------
1. Writes the page data under /tmp/dangerzone, so that the
`pixels_to_pdf` module can read them.
2. Pass OCR parameters as envvars.
3. Call the `pixels_to_pdf` main function, as if it was running within a
container. Wait until the PDF gets created.
4. Move the resulting PDF to the proper directory.
Fixes #414
Autodetect in the CLI/GUI if we should run the conversion in disposable qubes.
Allow creating an RPM package that is to be installed specifically on Qubes. This package has the following extra properties from our regular RPM packages: 1. Make `python3-magic`, `libreoffice` and `tesseract` requirements for installing Dangerzone, since the conversion takes place in a disposable qube that needs these packages. 2. Ignore the container.tar.gz file, if it exists. 3. Add our RPC calls under `/etc/qubes-rpc`
apyrgio
pushed a commit
that referenced
this pull request
Jun 21, 2023
Following a suggestion from @apyrgio [1] to not pollute /usr/local/bin. [1]: #437 (comment)
apyrgio
pushed a commit
that referenced
this pull request
Jun 21, 2023
Also removes exit codes in the qubes wrapper. Following a suggestion from #437 (comment)
apyrgio
force-pushed
the
qubes-integration-poc
branch
from
a4390ea to
abcc7b4
Compare
apyrgio
force-pushed
the
qubes-integration-poc
branch
from
abcc7b4 to
c4cc1a9
Compare
Add instructions aimed at developers who want to try out Qubes integration. Fixes #411
apyrgio
force-pushed
the
qubes-integration-poc
branch
from
c4cc1a9 to
20b24a6
Compare
apyrgio
approved these changes
Jun 21, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial work to port Dangerzone to Qubes OS, making use of Qubes VMs as an isolation provider. See
BUILD.mdfor instructions on how to setup and run. Fixes #411 and fixes #414.(some) implementation details:
document_to_pixels) takes place in disposable qubeMore details can be found on the wiki page
Primary code changes
dangerzone/conversion/and spitingdangerzone.pyinto the two filesThis was needed given that the second stage of the conversion original container code is ran on the primary qube (the one which runs the dangerzone GUI)
dangerzone/conversion/doc_to_pixels_qubes_wrapper.py)At the moment, the Qubes conversion is a simple wrapper around the
doc_to_pixelscode. This was because in Qubes pages are streamed one-by-one instead of all files sent at once after the conversion is done. The container-code however is still not ready to stream pages. So our wrapper converts the pages like before and then streams the results back to the client.Limitations
It does not support simultaneous file conversion as the file being converted is hard-coded as
/tmp/input-file.conversion progress was out of scope, but part of it is already working (for the first container stage). However, due to the "wrapper approach" it will jump from 0-50% in fractions of a second since when it starts sending pages, all the conversion has already happened.