Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow unsafe characters if invoked as qubes.UnsafeFileCopy #497

Merged
merged 10 commits into from
Jun 18, 2024

Conversation

DemiMarie
Copy link
Contributor

@DemiMarie DemiMarie commented May 9, 2024

This uses QREXEC_SERVICE_FULL_NAME to detect what the service was invoked as. Non-empty arguments are reserved for future use.

This requires QubesOS/qubes-linux-utils#113.

Part of QubesOS/qubes-issues#8332

qubes-rpc/qfile-unpacker.c Outdated Show resolved Hide resolved
qubes-rpc/qfile-unpacker.c Outdated Show resolved Hide resolved
@qubesos-bot
Copy link

qubesos-bot commented May 17, 2024

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024061618-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024052808-4.3&flavor=update

  • system_tests_guivm_vnc_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'clipboard-copy-notifi...
  • system_tests_basic_vm_qrexec_gui_xfs

Failed tests

12 failures
  • system_tests_pvgrub_salt_storage

    • [unstable] TC_41_HVMGrub_debian-12-xfce: test_000_standalone_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • [unstable] TC_41_HVMGrub_debian-12-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_fedora-40-xfce: test_000_standalone_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_fedora-40-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_splitgpg

  • system_tests_extra

    • [unstable] TC_00_QVCTest_fedora-40-xfce: test_020_webcam (failure + cleanup)
      AssertionError: 'qubes-video-companion webcam' exited early (0): b'...

    • [unstable] TC_00_QVCTest_whonix-workstation-17: test_020_webcam (failure + cleanup)
      self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0

  • system_tests_guivm_vnc_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'clipboard-copy-notifi...
  • system_tests_basic_vm_qrexec_gui_btrfs

    • [unstable] TC_20_AudioVM_Pulse_fedora-40-xfce-pool: test_225_audio_rec_unmuted_hvm (failure)
      AssertionError: too short audio, expected 10s, got 7.30310657596371...
  • system_tests_basic_vm_qrexec_gui_ext4

  • system_tests_basic_vm_qrexec_gui_xfs

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/101100#dependencies

39 fixed
  • system_tests_basic_vm_qrexec_gui

  • system_tests_pvgrub_salt_storage

    • StorageFile: test_001_non_volatile (error)
      subprocess.CalledProcessError: Command '/usr/lib/qubes/destroy-snap...
  • system_tests_extra

    • TC_00_QVCTest_debian-12-xfce: test_020_webcam (failure)
      AssertionError: 'qubes-video-companion webcam' exited early (0): b'...

    • TC_00_QVCTest_whonix-workstation-17: test_010_screenshare (failure)
      self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0

  • system_tests_usbproxy

  • system_tests_network_updates

    • TC_00_Dom0Upgrade_whonix-gateway-17: test_006_update_flag_clear (failure)
      Error: Failed to download metadata for repo 'test': Cannot download...

    • TC_10_QvmTemplate_debian-12-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_10_QvmTemplate_fedora-40-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_10_QvmTemplate_whonix-gateway-17: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_11_QvmTemplateMgmtVM_debian-12-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_11_QvmTemplateMgmtVM_fedora-40-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

    • TC_11_QvmTemplateMgmtVM_whonix-gateway-17: test_010_template_install (failure)
      AssertionError: qvm-template failed: Downloading 'qubes-template-de...

  • system_tests_basic_vm_qrexec_gui_zfs

  • system_tests_basic_vm_qrexec_gui_btrfs

  • system_tests_basic_vm_qrexec_gui_ext4

  • system_tests_basic_vm_qrexec_gui_xfs

  • system_tests_basic_vm_qrexec_gui@hw1

  • system_tests_guivm_gui_interactive

    • update_guivm: Failed (test died)
      # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...

Unstable tests

  • system_tests_suspend

    suspend/ (2/4 times with errors)
    suspend/Failed (2/4 times with errors)
    • job 102431 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    • job 102443 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    suspend/wait_serial (2/4 times with errors)
    • job 102431 # wait_serial expected: qr/2E8vz-\d+-/...
    • job 102443 # wait_serial expected: qr/2E8vz-\d+-/...
  • system_tests_basic_vm_qrexec_gui

    TC_20_AudioVM_Pulse_whonix-workstation-17/test_220_audio_play_pulseaudio (2/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101758 AssertionError: too short audio, expected 10s, got 8.73532879818594...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_222_audio_rec_unmuted_pulseaudio (2/2 times with errors)
    • job 101109 AssertionError: only silence detected, no useful audio data
    • job 101758 AssertionError: too short audio, expected 10s, got 9.36446712018140...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_223_audio_play_hvm (1/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_252_audio_playback_audiovm_switch_hvm (1/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_pvgrub_salt_storage

    TC_41_HVMGrub_debian-12-xfce/test_000_standalone_vm (1/2 times with errors)
    • job 101773 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    StorageFile/test_001_non_volatile (1/2 times with errors)
    • job 101124 subprocess.CalledProcessError: Command '/usr/lib/qubes/destroy-snap...
    TC_41_HVMGrub_debian-12-xfce/test_010_template_based_vm (1/2 times with errors)
    • job 101773 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_extra

    TC_00_QVCTest_whonix-workstation-17/test_010_screenshare (1/2 times with errors)
    • job 101116 self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0
    TC_00_QVCTest_debian-12-xfce/test_020_webcam (1/2 times with errors)
    • job 101116 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    TC_00_QVCTest_fedora-40-xfce/test_020_webcam (1/2 times with errors)
    • job 101765 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    TC_00_QVCTest_whonix-workstation-17/test_020_webcam (1/2 times with errors)
    • job 101116 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
  • system_tests_usbproxy

    TC_20_USBProxy_core3_whonix-gateway-17/test_070_attach_not_installed_front (1/2 times with errors)
    • job 101101 qubesusbproxy.core3ext.QubesUSBException: Device attach failed: 202...
  • system_tests_network_updates

    TC_00_Dom0Upgrade_whonix-gateway-17/test_006_update_flag_clear (1/2 times with errors)
    • job 101123 Error: Failed to download metadata for repo 'test': Cannot download...
    TC_10_QvmTemplate_debian-12-xfce/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_10_QvmTemplate_fedora-40-xfce/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_10_QvmTemplate_whonix-gateway-17/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_11_QvmTemplateMgmtVM_debian-12-xfce/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_11_QvmTemplateMgmtVM_fedora-40-xfce/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_11_QvmTemplateMgmtVM_whonix-gateway-17/test_010_template_install (1/2 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
  • system_tests_dispvm

    TC_20_DispVM_fedora-40-xfce/test_100_open_in_dispvm (1/2 times with errors)
    • job 101764 AssertionError: './open-file test.txt' failed with ./open-file test...
  • system_tests_basic_vm_qrexec_gui_btrfs

    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (1/2 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (1/2 times with errors)
    • job 101110 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (1/2 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_fedora-40-xfce-pool/test_225_audio_rec_unmuted_hvm (1/2 times with errors)
    • job 101782 AssertionError: too short audio, expected 10s, got 6.28875283446712...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (2/2 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101782 AssertionError: pacat for test-inst-vm1 (xid 90) running(False) in ...
  • system_tests_basic_vm_qrexec_gui_ext4

    TC_00_Basic/test_141_libvirt_objects_reconnect (1/2 times with errors)
    • job 101111 AssertionError: libvirt event impl drain timeout
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (1/2 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (1/2 times with errors)
    • job 101111 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (1/2 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (1/2 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_basic_vm_qrexec_gui_xfs

    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (2/2 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101781 AssertionError: too short audio, expected 10s, got 8.33947845804988...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (2/2 times with errors)
    • job 101112 AssertionError: only silence detected, no useful audio data
    • job 101781 AssertionError: too short audio, expected 10s, got 8.17784580498866...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (2/2 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101781 AssertionError: too short audio, expected 10s, got 7.71551020408163...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (1/2 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_basic_vm_qrexec_gui@hw1

    TC_20_AudioVM_Pulse_whonix-workstation-17/test_220_audio_play_pulseaudio (2/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101758 AssertionError: too short audio, expected 10s, got 8.73532879818594...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_222_audio_rec_unmuted_pulseaudio (2/2 times with errors)
    • job 101109 AssertionError: only silence detected, no useful audio data
    • job 101758 AssertionError: too short audio, expected 10s, got 9.36446712018140...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_223_audio_play_hvm (1/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_252_audio_playback_audiovm_switch_hvm (1/2 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_suspend@hw1

    suspend/ (2/4 times with errors)
    suspend/Failed (2/4 times with errors)
    • job 102431 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    • job 102443 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    suspend/wait_serial (2/4 times with errors)
    • job 102431 # wait_serial expected: qr/2E8vz-\d+-/...
    • job 102443 # wait_serial expected: qr/2E8vz-\d+-/...

Copy link
Member

@marmarek marmarek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is missing actually introducing qubes.UnsafeFilecopy service. And also changing qvm-copy to choose service based on files to be copied (see QubesOS/qubes-issues#8332 (comment))

@DemiMarie DemiMarie marked this pull request as draft May 20, 2024 21:14
@DemiMarie DemiMarie requested a review from marmarek May 27, 2024 18:47
@DemiMarie DemiMarie marked this pull request as ready for review May 27, 2024 18:47
DemiMarie added 7 commits May 27, 2024 15:00
Fixes a compiler warning.
Previously various invalid arguments were silently ignored or misparsed.
According to the manual page, setfsuid() does not provide any indication
of success or failure.  The only way to check if it succeeded is to call
it again with -1 as the UID and check the return value.
There is no good reason not to use these flags.
This requires QubesOS/qubes-linux-utils#113.  It also adds a new
argument parser based on getopt_long(), which is used instead of the old
hand-rolled code unless there are at least two arguments and the first
one starts with an ASCII digit.

Part of QubesOS/qubes-issues#8332
@marmarek
Copy link
Member

This fails to build on Debian:

12:57:49,692 [executor:docker:0e16ac29c7ff] output: cc -g -pie -Wl,-z,relro '-Wl,-rpath,$ORIGIN/../../$LIB'  vm-file-editor.o  -lqubes-rpc-filecopy -lqubes-pure -o vm-file-editor
12:57:49,697 [executor:docker:0e16ac29c7ff] output: /usr/bin/ld: cannot find -lqubes-pure: No such file or directory

Likely a missing Build-Depends:

@marmarek
Copy link
Member

In fact it fails to build on Fedora too, due to a different reason:

18:50:55,505 [executor:docker:5697546c802e] output: DEBUG:     Installed (but unpackaged) file(s) found:
18:50:55,505 [executor:docker:5697546c802e] output: DEBUG:    /usr/lib/debug/usr/lib/qubes/qubes-fs-tree-check-4.3.0-1.2.fc40.x86_64.debug
18:50:55,506 [executor:docker:5697546c802e] output: DEBUG:    /usr/lib/qubes/qubes-fs-tree-check

@DemiMarie
Copy link
Contributor Author

Build failed because QubesOS/qubes-linux-utils#113 is not part of the latest released package.

@marmarek
Copy link
Member

Build failed because QubesOS/qubes-linux-utils#113 is not part of the latest released package.

The logs I pasted were from a pipeline that included that PR too.

But yes, this PR alone is expected to fail.

@DemiMarie
Copy link
Contributor Author

Build failed because QubesOS/qubes-linux-utils#113 is not part of the latest released package.

The logs I pasted were from a pipeline that included that PR too.

But yes, this PR alone is expected to fail.

I checked and this version builds with QubesOS/qubes-linux-utils#113.

@marmarek
Copy link
Member

qvm-copy tests fail (see updated openqa report above), example failure:

# test_100_qrexec_filecopy
# failure: 

# timestamp 2024-05-29T03:57:36.815652
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/qubes/tests/integ/vm_qrexec_gui.py", line 612, in test_100_qrexec_filecopy
    self.loop.run_until_complete(
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
subprocess.CalledProcessError: Command 'qvm-copy-to-vm test-inst-vm2 /tmp/passwd' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/qubes/tests/integ/vm_qrexec_gui.py", line 617, in test_100_qrexec_filecopy
    self.fail('qvm-copy-to-vm failed: {}'.format(e.stderr))
AssertionError: qvm-copy-to-vm failed: b'qubes-fs-tree-check: Argument 5 is an absolute path\n'

@DemiMarie
Copy link
Contributor Author

This is due to an outdated check from when I misunderstood how qfile-agent selects the filenames to pass to the remote side. I thought it used the full path, but it only uses the basename.

@marmarek
Copy link
Member

marmarek commented Jun 1, 2024

Manual test passed.
But it may need adjustment for QubesOS/qubes-issues#8332 (comment)

@DemiMarie
Copy link
Contributor Author

I will update the code to use the service argument.

This is done by means of the --allow-unsafe-characters flag to
qfile-unpacker, and is only done if allow-unsafe-characters is passed as
a service argument.  The service argument is always visible in the
qrexec prompt, so users will know if this flag will be used and can
choose to reject the call.
This adds a filesystem tree checker that determins if a directory tree
can be copied with qubes.Filecopy or if qubes.UnsafeFileCopy is
required.  It also produces the total size of the tree, which
qfile-agent uses for progress reporting.  The checker can also be used
for other purposes, such as checking if there are nasty characters or
symbolic links in a filesystem tree unpacked via a command such as
'tar'.
This makes testing much, *much* easier.  The binaries use an rpath to
find the libraries they depend on, and qfile-copy uses $0 to find the
binaries it uses.  /usr/bin/qfile-unpacker needs a different rpath
than the binaries under /usr/lib/qubes, so they must now be built
separately (from the same object files).
@marmarek marmarek merged commit 6c1a2bd into QubesOS:main Jun 18, 2024
1 of 2 checks passed
@DemiMarie DemiMarie deleted the less-safe-unpack branch June 18, 2024 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants