Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Podman Image SCP rootful to rootless transfer #11958

Merged
merged 1 commit into from
Nov 8, 2021

Conversation

cdoern
Copy link
Contributor

@cdoern cdoern commented Oct 13, 2021

Added functionality for users to transfer images from root storage to rootless storage without using sshd. This is
done through rootful podman by running `sudo podman image scp root@localhost::image user@localhost:: the user is needed
in order to find and use their uid/gid to exec a new process.

added necessary tests, and functions for this implementation. Created new image function Transfer so that
the underlying code is majorly removed from CLI.

Signed-off-by: cdoern [email protected]

@cdoern cdoern force-pushed the scp branch 3 times, most recently from 92b7343 to 74f1326 Compare October 13, 2021 23:33
@TomSweeneyRedHat
Copy link
Member

overall, nice work as usual @cdoern TYVM!

Copy link
Member

@Luap99 Luap99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think this can work correctly. This is dangerous and relies on many assumptions which might not be true.

I was unable to make this work. I added debug output and this is how it fails:

INFO[0000] /usr/bin/podman filtering at log level debug 
DEBU[0000] Called load.PersistentPreRunE(/usr/bin/podman --log-level=debug --cgroup-manager=cgroupfs load --input /var/tmp/podman214679695) 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/paul/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /run/containers/storage       
DEBU[0000] Using static dir /home/paul/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/paul/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
Error: mkdir /run/containers/storage: permission denied
Error: exit status 125

The reason why this fails is because I did not run a rootless podman command before and this command created the db for the first time with corrupt values since it will try to use the rootful paths.

If I create the podman db before it still fails with

INFO[0000] /usr/bin/podman filtering at log level debug 
DEBU[0000] Called load.PersistentPreRunE(/usr/bin/podman --log-level=debug --cgroup-manager=cgroupfs load --input /var/tmp/podman027624033) 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/paul/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Overriding run root "/run/containers/storage" with "/run/user/1000/containers" from database 
DEBU[0000] Overriding graph root "/var/lib/containers/storage" with "/home/paul/.local/share/containers/storage" from database 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/paul/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/paul/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/paul/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
INFO[0000] metacopy option not supported on this kernelnodev,metacopy=on 
DEBU[0000] overlay test mount indicated that metacopy is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/paul/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 13             
DEBU[0000] Loading image from "/var/tmp/podman027624033" 
DEBU[0000] -> Attempting to load "/var/tmp/podman027624033" as an OCI directory 
DEBU[0000] parsed reference into "[overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]localhost/var/tmp/podman027624033:latest" 
DEBU[0000] Copying source image /var/tmp/podman027624033: to destination image [overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]localhost/var/tmp/podman027624033:latest 
DEBU[0000] Error loading /var/tmp/podman027624033: initializing source oci:/var/tmp/podman027624033:: open /var/tmp/podman027624033/index.json: not a directory 
DEBU[0000] -> Attempting to load "/var/tmp/podman027624033" as an OCI archive 
DEBU[0000] Error deleting temporary directory: <nil>    
DEBU[0000] parsed reference into "[overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]registry.fedoraproject.org/fedora:latest" 
DEBU[0000] Copying source image /var/tmp/podman027624033: to destination image [overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]registry.fedoraproject.org/fedora:latest 
DEBU[0000] Using blob info cache at /home/paul/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image oci-archive:/var/tmp/podman027624033 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Detected compression format gzip             
DEBU[0000] Using original blob without modification     
Copying blob 944c4b241113 done  
Copying blob 944c4b241113 done  
DEBU[0003] error deleting tmp dir: <nil>                
DEBU[0003] Error loading /var/tmp/podman027624033: writing blob: adding layer with blob "sha256:944c4b241113c7047b3aa00c286aee989dc5b808ec0cae27d79d8dba87501ece": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:5 for /usr/bin/write): Check /etc/subuid and /etc/subgid: lchown /usr/bin/write: invalid argument 
DEBU[0003] -> Attempting to load "/var/tmp/podman027624033" as a Docker dir 
DEBU[0003] parsed reference into "[overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]localhost/var/tmp/podman027624033:latest" 
DEBU[0003] Copying source image /var/tmp/podman027624033 to destination image [overlay@/home/paul/.local/share/containers/storage+/run/user/1000/containers:overlay.mountopt=nodev,metacopy=on]localhost/var/tmp/podman027624033:latest 
DEBU[0003] Using blob info cache at /home/paul/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0003] Error loading /var/tmp/podman027624033: determining manifest MIME type for dir:/var/tmp/podman027624033: open /var/tmp/podman027624033/manifest.json: not a directory 
DEBU[0003] -> Attempting to load "/var/tmp/podman027624033" as a Docker archive 
DEBU[0003] No compression detected                      
DEBU[0003] Error loading /var/tmp/podman027624033: loading tar component manifest.json: file does not exist 
DEBU[0003] Error loading /var/tmp/podman027624033: payload does not match any of the supported image formats (oci, oci-archive, dir, docker-archive) 
Error: payload does not match any of the supported image formats (oci, oci-archive, dir, docker-archive)
Error: exit status 125

I generally think it is impossible to get this right, especially when the correctly working alternative is sudo podman image save fedora | podman image load

pkg/domain/infra/abi/images.go Outdated Show resolved Hide resolved
pkg/domain/infra/abi/images.go Outdated Show resolved Hide resolved
pkg/domain/infra/abi/images.go Outdated Show resolved Hide resolved
pkg/util/utils.go Outdated Show resolved Hide resolved
pkg/domain/infra/abi/images.go Outdated Show resolved Hide resolved
@cdoern
Copy link
Contributor Author

cdoern commented Oct 14, 2021

@Luap99 I think all of your concerns and breakages stem from the environmental variables. I feel like I am probably missing some modifications to these. Generally, this works and is more fluid than sudo podman image pull/save | podman image load which often fails due to formatting errors. If we can get this to the point where it uses all of the correct files and paths so Chowning isn't necessary, should be better off. Howver, I do chown the files (for the ones that aren't eventually removed like the temp tarball) back to their original uid and gid @mheon @rhatdan can you PTAL so we can all get a sense of what is trying to be accomplished here?

This is the main process we wanted to add to image scp and I think it is a valuable one. but @Luap99 if the goal of the flags is to load into rootless storage why would the user never have used rootless podman before? I feel like that is a generally safe assumption to make most of the time, and if not it is a relatively quick fix that I can add to the man pages.

@rhatdan
Copy link
Member

rhatdan commented Oct 14, 2021

Why aren't you using the localhost syntax?
podman image scp root@localhost:alpine .
podman image scp alpine root@localhost

None of these fancy flags.

It should be the same syntax, we just need to look for the key word localhost and attempt to shortcut it.

@cdoern
Copy link
Contributor Author

cdoern commented Oct 14, 2021

@rhatdan I can convert to that syntax with the inclusion of a user@localhost to send the image to since we have to execute the process as sudo podman image scp user@localhost I was having trouble starting the process rootless and sending the image to root. I will continue to modify it, I'm going to convert this to a draft and see what I can change.

@cdoern cdoern added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 14, 2021
@cdoern
Copy link
Contributor Author

cdoern commented Oct 14, 2021

@Luap99 I think I began to address some of your concerns with environmental variables, I changed it so that we actually retrieve the users environmental vars and check for their run directory, using that for the other variables as necessary. @rhatdan I switched the syntax to sudo podman image scp root@localhost::image user@localhost:: since we need to exec the original podman command through sudo.

Honestly I would like to remove the necessity of calling this command using sudo but I do not think it is possible to start a rootful process (the save from root) out of a rootless one.

@cdoern cdoern changed the title Podman Image SCP --rootless and --user flag Podman Image SCP rootful to rootless transfer Oct 14, 2021
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 14, 2021
@cdoern cdoern force-pushed the scp branch 2 times, most recently from 63d3e18 to ec12a7c Compare October 15, 2021 00:11
@cdoern
Copy link
Contributor Author

cdoern commented Oct 15, 2021

@Luap99 would adding some sort of podman setup command to set up the rootless DB make sense here? I feel like having something we can run internally at times to make sure everything is set up would be a good fail safe.

edit: after some investigation and messing around I think I can make a function (internal for now) that initializes the rootless podman db, we can investigate whether or not we want to expose this to the user.

@cdoern cdoern marked this pull request as draft October 19, 2021 03:17
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 19, 2021
@cdoern cdoern force-pushed the scp branch 3 times, most recently from 147b077 to cd9c792 Compare October 21, 2021 01:16
@cdoern
Copy link
Contributor Author

cdoern commented Oct 21, 2021

@rhatdan @mheon PTAL at what I did here, Paul made a good point that if the rootless DB is not initialized, this will fail. Made a new setup function that creates a basic rootless DB and I also think this could be made into a podman system setup command to close the loop on other commands like reset, migrate, etc. I might've jumped to a few conclusions in terms of file names so let me know if there are any glaring issues.

@cdoern cdoern force-pushed the scp branch 3 times, most recently from efcfcd5 to 4a462d9 Compare October 21, 2021 03:19
@cdoern
Copy link
Contributor Author

cdoern commented Nov 5, 2021

wait, machinectl is causing some issues with downloading the image now. Need to investigate after class today.

Added functionality for users to transfer images from root storage to rootless storage without using sshd. This is
done through rootful podman by running `sudo podman image scp root@localhost::image user@localhost:: the user is needed
in order to find and use their uid/gid to exec a new process.

added necessary tests, and functions for this implementation. Created new image function Transfer so that
the underlying code is majorly removed from CLI

Signed-off-by: cdoern <[email protected]>
@cdoern
Copy link
Contributor Author

cdoern commented Nov 5, 2021

nevermind @Luap99 the issue was with my laptop, tested it on my other machine and works perfectly with machinectl. I think this is good now after solving that.

Copy link
Member

@Luap99 Luap99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 8, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cdoern, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 8, 2021
@Luap99
Copy link
Member

Luap99 commented Nov 8, 2021

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2021
@openshift-merge-robot openshift-merge-robot merged commit c9ba1fb into containers:main Nov 8, 2021
@rhatdan
Copy link
Member

rhatdan commented Nov 8, 2021

@cdoern Not sure what I am doing wrong, but this is definitely broken.

 $ ./bin/podman image scp myimage root@localhost:
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
github.com/containers/podman/v3/cmd/podman/images.parseArgs(0xc0003bd9a0, 0x2, 0x2, 0xc0000e4000, 0x1f471b0, 0xc000010718, 0x0)
	/home/dwalsh/podman/cmd/podman/images/scp.go:291 +0x1465
github.com/containers/podman/v3/cmd/podman/images.scp(0x2983040, 0xc0003bd9a0, 0x2, 0x2, 0x0, 0x0)
	/home/dwalsh/podman/cmd/podman/images/scp.go:91 +0x236
github.com/spf13/cobra.(*Command).execute(0x2983040, 0xc000138030, 0x2, 0x2, 0x2983040, 0xc000138030)
	/home/dwalsh/podman/vendor/github.com/spf13/cobra/command.go:856 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0x2992cc0, 0xc000130030, 0x19ef400, 0x2a55f10)
	/home/dwalsh/podman/vendor/github.com/spf13/cobra/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/home/dwalsh/podman/vendor/github.com/spf13/cobra/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/dwalsh/podman/vendor/github.com/spf13/cobra/command.go:895
main.Execute()
	/home/dwalsh/podman/cmd/podman/root.go:91 +0xe7
main.main()
	/home/dwalsh/podman/cmd/podman/main.go:39 +0x92

@cdoern
Copy link
Contributor Author

cdoern commented Nov 8, 2021

@cdoern Not sure what I am doing wrong, but this is definitely broken.

 $ ./bin/podman image scp myimage root@localhost:
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
github.com/containers/podman/v3/cmd/podman/images.parseArgs(0xc0003bd9a0, 0x2, 0x2, 0xc0000e4000, 0x1f471b0, 0xc000010718, 0x0)
	/home/dwalsh/podman/cmd/podman/images/scp.go:291 +0x1465
github.com/containers/podman/v3/cmd/podman/images.scp(0x2983040, 0xc0003bd9a0, 0x2, 0x2, 0x0, 0x0)
	/home/dwalsh/podman/cmd/podman/images/scp.go:91 +0x236
github.com/spf13/cobra.(*Command).execute(0x2983040, 0xc000138030, 0x2, 0x2, 0x2983040, 0xc000138030)
	/home/dwalsh/podman/vendor/github.com/spf13/cobra/command.go:856 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0x2992cc0, 0xc000130030, 0x19ef400, 0x2a55f10)
	/home/dwalsh/podman/vendor/github.com/spf13/cobra/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/home/dwalsh/podman/vendor/github.com/spf13/cobra/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/dwalsh/podman/vendor/github.com/spf13/cobra/command.go:895
main.Execute()
	/home/dwalsh/podman/cmd/podman/root.go:91 +0xe7
main.main()
	/home/dwalsh/podman/cmd/podman/main.go:39 +0x92

@rhatdan you need to specify a user. I will submit a patch to not allow for just one argument with root. the syntax is sudo podman image scp root@localhost::image user@localhost:: this is necessary to chown the image file to a specific user. I guess i could set it up so if no user is specified it defaults to 1000:1000

also this is transferring from root storage so the default sytax (if we choose to add) would be sudo podman image scp root@localhost::image image root@localhost:: signifies tranferring to root

@rhatdan
Copy link
Member

rhatdan commented Nov 8, 2021

User should be assumed to be me, if I don't specify.

podman image scp myimage root@localhost::
podman image scp root@localhost::foo bar

Should be ok syntax.

@cdoern
Copy link
Contributor Author

cdoern commented Nov 8, 2021

ok @rhatdan I will make a quick PR to patch the syntax up. Sticking with the current image scp syntax podman image scp root@localhost::image is the only supported format since I am still working on transferring from root.

@cdoern
Copy link
Contributor Author

cdoern commented Nov 8, 2021

@rhatdan the issue with assuming to be "me" is that this command (root to rootless) needs to be run using sudo. Is there a way to get default user or should i just assume 1000:1000.

@Luap99
Copy link
Member

Luap99 commented Nov 8, 2021

@cdoern You can use SUDO_USER and SUDO_UID env vars when the command is run with sudo. Otherwise I would just error and say that you need to specify the user.

@cdoern
Copy link
Contributor Author

cdoern commented Nov 8, 2021

@cdoern You can use SUDO_USER and SUDO_UID env vars when the command is run with sudo. Otherwise I would just error and say that you need to specify the user.

@Luap99 yep, I just submitted a PR using SUDO_USER

@rhatdan
Copy link
Member

rhatdan commented Nov 8, 2021

Again, I do not want to run as sudo, I want magic to happen. If I am running with sudo, then fine. Looking at SUDO_USER would be fine.

My question why can't
podman image scp myimage root@localhost::
Work? I would figure this would just do

podman image save $XDG_RUNTIME_DIR:/myimage; sudo podman load $XDG_RUNTIME_DIR:/myimage
Under the covers.

sudo podman image scp root@localhost::foobar .

Would do
sudo podman image save $XDG_RUNTIME_DIR:/foobar; podman load $XDG_RUNTIME_DIR:/foobar

@Luap99
Copy link
Member

Luap99 commented Nov 8, 2021

Again, I do not want to run as sudo, I want magic to happen. If I am running with sudo, then fine. Looking at SUDO_USER would be fine.

My question why can't podman image scp myimage root@localhost:: Work? I would figure this would just do

podman image save $XDG_RUNTIME_DIR:/myimage; sudo podman load $XDG_RUNTIME_DIR:/myimage Under the covers.

sudo podman image scp root@localhost::foobar .

Would do sudo podman image save $XDG_RUNTIME_DIR:/foobar; podman load $XDG_RUNTIME_DIR:/foobar

This does not work because you cannot run sudo inside the podman user namespace.

@cdoern
Copy link
Contributor Author

cdoern commented Nov 8, 2021

see comment on new PR it seems to be impossible to invoke sudo inside of a rootless podman user NS

@rhatdan
Copy link
Member

rhatdan commented Nov 8, 2021

Sure, correct the command to not enter the user namespace until it has to.
@giuseppe Isn't that possible.

@rhatdan
Copy link
Member

rhatdan commented Nov 8, 2021

Seems to me that we should be able to break into a couple of different threads, one that enters the user namespace to do its thing and one to do the sudo command and exec itself.

@cdoern
Copy link
Contributor Author

cdoern commented Nov 9, 2021

correct which command? the original podman image scp or the exec'd load? If this is to be done rootless, the save command must also be an exec.Command reaching into root storage.

@giuseppe
Copy link
Member

giuseppe commented Nov 9, 2021

Seems to me that we should be able to break into a couple of different threads, one that enters the user namespace to do its thing and one to do the sudo command and exec itself.

a thread cannot enter a user namespace. The entire process must do it (when it is single threaded).

@rhatdan
Copy link
Member

rhatdan commented Nov 9, 2021

This all be done with two execs, One into the user namespace and one with sudo?
podman image scp alpine root@localhost
---> exec podman image save alpine XDG_RUNTIME_DIR/alpine
---> exec sudo podman image load XDG_RUNTIME_DIR/alpine

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants