-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Podman Image SCP rootful to rootless transfer #11958
Conversation
92b7343
to
74f1326
Compare
overall, nice work as usual @cdoern TYVM! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not think this can work correctly. This is dangerous and relies on many assumptions which might not be true.
I was unable to make this work. I added debug output and this is how it fails:
INFO[0000] /usr/bin/podman filtering at log level debug
DEBU[0000] Called load.PersistentPreRunE(/usr/bin/podman --log-level=debug --cgroup-manager=cgroupfs load --input /var/tmp/podman214679695)
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/paul/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /home/paul/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/paul/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
Error: mkdir /run/containers/storage: permission denied
Error: exit status 125
The reason why this fails is because I did not run a rootless podman command before and this command created the db for the first time with corrupt values since it will try to use the rootful paths.
If I create the podman db before it still fails with
INFO[0000] /usr/bin/podman filtering at log level debug
DEBU[0000] Called load.PersistentPreRunE(/usr/bin/podman --log-level=debug --cgroup-manager=cgroupfs load --input /var/tmp/podman027624033)
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/paul/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/containers/storage" with "/run/user/1000/containers" from database
DEBU[0000] Overriding graph root "/var/lib/containers/storage" with "/home/paul/.local/share/containers/storage" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/paul/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/paul/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/paul/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
INFO[0000] metacopy option not supported on this kernelnodev,metacopy=on
DEBU[0000] overlay test mount indicated that metacopy is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /home/paul/.config/cni/net.d/87-podman.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 13
DEBU[0000] Loading image from "/var/tmp/podman027624033"
DEBU[0000] -> Attempting to load "/var/tmp/podman027624033" as an OCI directory
DEBU[0000] parsed reference into "[overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]localhost/var/tmp/podman027624033:latest"
DEBU[0000] Copying source image /var/tmp/podman027624033: to destination image [overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]localhost/var/tmp/podman027624033:latest
DEBU[0000] Error loading /var/tmp/podman027624033: initializing source oci:/var/tmp/podman027624033:: open /var/tmp/podman027624033/index.json: not a directory
DEBU[0000] -> Attempting to load "/var/tmp/podman027624033" as an OCI archive
DEBU[0000] Error deleting temporary directory: <nil>
DEBU[0000] parsed reference into "[overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]registry.fedoraproject.org/fedora:latest"
DEBU[0000] Copying source image /var/tmp/podman027624033: to destination image [overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]registry.fedoraproject.org/fedora:latest
DEBU[0000] Using blob info cache at /home/paul/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] IsRunningImageAllowed for image oci-archive:/var/tmp/podman027624033
DEBU[0000] Using default policy section
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Detected compression format gzip
DEBU[0000] Using original blob without modification
Copying blob 944c4b241113 done
Copying blob 944c4b241113 done
DEBU[0003] error deleting tmp dir: <nil>
DEBU[0003] Error loading /var/tmp/podman027624033: writing blob: adding layer with blob "sha256:944c4b241113c7047b3aa00c286aee989dc5b808ec0cae27d79d8dba87501ece": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:5 for /usr/bin/write): Check /etc/subuid and /etc/subgid: lchown /usr/bin/write: invalid argument
DEBU[0003] -> Attempting to load "/var/tmp/podman027624033" as a Docker dir
DEBU[0003] parsed reference into "[overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]localhost/var/tmp/podman027624033:latest"
DEBU[0003] Copying source image /var/tmp/podman027624033 to destination image [overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]localhost/var/tmp/podman027624033:latest
DEBU[0003] Using blob info cache at /home/paul/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0003] Error loading /var/tmp/podman027624033: determining manifest MIME type for dir:/var/tmp/podman027624033: open /var/tmp/podman027624033/manifest.json: not a directory
DEBU[0003] -> Attempting to load "/var/tmp/podman027624033" as a Docker archive
DEBU[0003] No compression detected
DEBU[0003] Error loading /var/tmp/podman027624033: loading tar component manifest.json: file does not exist
DEBU[0003] Error loading /var/tmp/podman027624033: payload does not match any of the supported image formats (oci, oci-archive, dir, docker-archive)
Error: payload does not match any of the supported image formats (oci, oci-archive, dir, docker-archive)
Error: exit status 125
I generally think it is impossible to get this right, especially when the correctly working alternative is sudo podman image save fedora | podman image load
@Luap99 I think all of your concerns and breakages stem from the environmental variables. I feel like I am probably missing some modifications to these. Generally, this works and is more fluid than This is the main process we wanted to add to |
Why aren't you using the localhost syntax? None of these fancy flags. It should be the same syntax, we just need to look for the key word localhost and attempt to shortcut it. |
@rhatdan I can convert to that syntax with the inclusion of a |
@Luap99 I think I began to address some of your concerns with environmental variables, I changed it so that we actually retrieve the users environmental vars and check for their run directory, using that for the other variables as necessary. @rhatdan I switched the syntax to Honestly I would like to remove the necessity of calling this command using sudo but I do not think it is possible to start a rootful process (the save from root) out of a rootless one. |
63d3e18
to
ec12a7c
Compare
@Luap99 would adding some sort of edit: after some investigation and messing around I think I can make a function (internal for now) that initializes the rootless podman db, we can investigate whether or not we want to expose this to the user. |
147b077
to
cd9c792
Compare
@rhatdan @mheon PTAL at what I did here, Paul made a good point that if the rootless DB is not initialized, this will fail. Made a new setup function that creates a basic rootless DB and I also think this could be made into a |
efcfcd5
to
4a462d9
Compare
wait, machinectl is causing some issues with downloading the image now. Need to investigate after class today. |
Added functionality for users to transfer images from root storage to rootless storage without using sshd. This is done through rootful podman by running `sudo podman image scp root@localhost::image user@localhost:: the user is needed in order to find and use their uid/gid to exec a new process. added necessary tests, and functions for this implementation. Created new image function Transfer so that the underlying code is majorly removed from CLI Signed-off-by: cdoern <[email protected]>
nevermind @Luap99 the issue was with my laptop, tested it on my other machine and works perfectly with machinectl. I think this is good now after solving that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cdoern, Luap99 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/lgtm |
@cdoern Not sure what I am doing wrong, but this is definitely broken.
|
@rhatdan you need to specify a user. I will submit a patch to not allow for just one argument with root. the syntax is also this is transferring from root storage so the default sytax (if we choose to add) would be |
User should be assumed to be me, if I don't specify. podman image scp myimage root@localhost:: Should be ok syntax. |
ok @rhatdan I will make a quick PR to patch the syntax up. Sticking with the current image scp syntax |
@rhatdan the issue with assuming to be "me" is that this command (root to rootless) needs to be run using sudo. Is there a way to get default user or should i just assume 1000:1000. |
@cdoern You can use |
Again, I do not want to run as sudo, I want magic to happen. If I am running with sudo, then fine. Looking at SUDO_USER would be fine. My question why can't podman image save $XDG_RUNTIME_DIR:/myimage; sudo podman load $XDG_RUNTIME_DIR:/myimage sudo podman image scp root@localhost::foobar . Would do |
This does not work because you cannot run sudo inside the podman user namespace. |
see comment on new PR it seems to be impossible to invoke sudo inside of a rootless podman user NS |
Sure, correct the command to not enter the user namespace until it has to. |
Seems to me that we should be able to break into a couple of different threads, one that enters the user namespace to do its thing and one to do the sudo command and exec itself. |
correct which command? the original |
a thread cannot enter a user namespace. The entire process must do it (when it is single threaded). |
This all be done with two execs, One into the user namespace and one with sudo? |
Added functionality for users to transfer images from root storage to rootless storage without using sshd. This is
done through rootful podman by running `sudo podman image scp root@localhost::image user@localhost:: the user is needed
in order to find and use their uid/gid to exec a new process.
added necessary tests, and functions for this implementation. Created new image function
Transfer
so thatthe underlying code is majorly removed from CLI.
Signed-off-by: cdoern [email protected]