Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Volume Plugins] Specifying options resets Volume Driver to local. #9650

Closed
thephoenixofthevoid opened this issue Mar 7, 2021 · 9 comments
Closed
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue

Comments

@thephoenixofthevoid
Copy link
Contributor

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

If using volume plugin with specified options, Driver (--driver) argument is completely ignored.

Steps to reproduce the issue:

  1. Forked docker-volume-sshfs and changed paths to socket (it has been hardcoded).

  2. Added a record in containers.conf

  3. Started a plugin. Then tried to create a volume without options and specifying them.

Describe the results you received:

For podman volume create --driver testplugin as well as using API endpoint /libpod/volumes/create with { Driver: "testplugin" } got:

$ ./podman-volume-ssh 
INFO[0000] listening on /tmp/podman-plugins-test.sock   
2021/03/07 14:59:13 Entering go-plugins-helpers getPath
ERRO[0054] volume 784484bf997f98d0b04a07525eb12560da1740f6139cc0e7463132427496937a not found 
2021/03/07 14:59:13 Entering go-plugins-helpers createPath
ERRO[0054] 'sshcmd' option required         

API Endpoint response:

{
  cause: "'sshcmd' option required",
  message: `error creating volume "9874028d86d0e6404b0dcda74cd46da441dcc0b2a6650bb0f8fef56da8bc0eb4" in plugin testplugin: error on /VolumeDriver.Create on volume 9874028d86d0e6404b0dcda74cd46da441dcc0b2a6650bb0f8fef56da8bc0eb4 in volume plugin testplugin: 'sshcmd' option required`,
  response: 500
}

Which is correct actually. But.

For podman volume create --driver testplugin --opt sshcmd=<whatever> as well as using API endpoint /libpod/volumes/create with { Driver: "testplugin", Options: { sshcmd: <whatever> } } got:

<NOTHING REACHES PLUGIN ENDPOINT>

and the response:

{
  cause: 'invalid argument',
  message: 'error running volume create option: unrecognized volume option "sshcmd" is not supported with local driver: invalid argument',
  response: 500
}

Describe the results you expected:

Expected to create a volume that corresponds to a remote sshfs.

Additional information you deem important (e.g. issue happens only occasionally):

This might be related:

response, err := registry.ContainerEngine().VolumeCreate(context.Background(), createOpts)

Output of podman version:

Version:      3.0.0
API Version:  3.0.0
Go Version:   go1.15.7
Built:        Fri Feb 12 02:12:57 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-1.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 777074ecdb5e883b9bec233f3630c5e7fa37d521'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: thefallen
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.15-200.fc33.x86_64
  linkmode: dynamic
  memFree: 142139392
  memTotal: 8190509056
  ociRuntime:
    name: crun
    package: crun-0.17-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 9418559488
  swapTotal: 12445540352
  uptime: 10h 41m 15.58s (Approximately 0.42 days)
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/feanor/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/feanor/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 26
  runRoot: /run/user/1000
  volumePath: /home/feanor/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1613085177
  BuiltTime: Fri Feb 12 02:12:57 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.0-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 7, 2021
@thephoenixofthevoid
Copy link
Contributor Author

thephoenixofthevoid commented Mar 7, 2021

I checked the state of createOpts right before the

response, err := registry.ContainerEngine().VolumeCreate(context.Background(), createOpts)

and it is correct:

./podman volume create --driver testplugin --opt sshcmd=1

Gives

{
	 "Name": "",
	 "Driver": "testplugin",
	 "Label": {},
	 "Options": {
	  "sshcmd": "1"
	 }
}

So need to investigate registry.ContainerEngine().VolumeCreate method.


Then... In function (runtime_volume_linux.go)

func (r *Runtime) newVolume(ctx context.Context, options ...VolumeCreateOption) (_ *Volume, deferredErr error) {
    print("ENTERED--------------")
	volume := newVolume(r)
	for _, option := range options {
		if err := option(volume); err != nil {
			return nil, errors.Wrapf(err, "error running volume create option")
		}
	}
	
	print("SURVIVED--------------")

Entered gets printed in both cases, but SURVIVED only when options are blank.


And finally, I got here:

func WithVolumeOptions(options map[string]string) VolumeCreateOption {

This function gets called for every driver, but actually local-specific one. This must be the reason of the bug.


By doing so:

func WithVolumeOptions(options map[string]string) VolumeCreateOption {
	return func(volume *Volume) error {
		if volume.valid {
			return define.ErrVolumeFinalized
		}

		volume.config.Options = make(map[string]string)
		for key, value := range options {
			// switch key {
			//case "type", "device", "o", "UID", "GID":
			//	volume.config.Options[key] = value
			//default:
			//	return errors.Wrapf(define.ErrInvalidArg, "unrecognized volume option %q is not supported with local driver", key)
			//}

			volume.config.Options[key] = value
		}

		return nil
	}
}

Bug is fixed. Create a PR?


Deleting this checks here seems to be ok, because we have:

for key := range volume.config.Options {

the same checks here.

Considering

plugin, err := r.getVolumePlugin(volume.config.Driver)

better solution would be refactoring local into the kind-of-plugin with the same interface. As such this interface could be extended with checkCreateVolumeOptions method, checkNeedCreate, actuallyCreateVolume and so on. This will greatly reduce complexity of current implementation.

@mheon
Copy link
Member

mheon commented Mar 7, 2021

Can you verify that podman volume create (the actual CLI command) has/does not have this issue? I'm strongly suspecting that it's an issue with the REST endpoint for volume creation, not the underlying logic.

@thephoenixofthevoid
Copy link
Contributor Author

thephoenixofthevoid commented Mar 7, 2021

It does have an issue. Actually I first encountered it in cli, but tracking down is simpler using REST API. One of the important thing here is that option must have different option key than ones that local driver supports.

So, you won't have that exact bug with plugin that only supports "type", "device", "o", "UID" and/or "GID". But the following issue will arise instead: "cannot mount in rootless mode" . I think I figured out this one too: I will update this message with details ASAP.


Here is the patch that allowed me to use rootlessly the volume driver plugin:

diff --git a/libpod/options.go b/libpod/options.go
index 6344e1acc..9e09a2164 100644
--- a/libpod/options.go
+++ b/libpod/options.go
@@ -1580,12 +1580,12 @@ func WithVolumeOptions(options map[string]string) VolumeCreateOption {
 
 		volume.config.Options = make(map[string]string)
 		for key, value := range options {
-			switch key {
-			case "type", "device", "o", "UID", "GID":
-				volume.config.Options[key] = value
-			default:
-				return errors.Wrapf(define.ErrInvalidArg, "unrecognized volume option %q is not supported with local driver", key)
-			}
+			//switch key {
+			//case "type", "device", "o", "UID", "GID":
+			//	volume.config.Options[key] = value
+			//default:
+			//	return errors.Wrapf(define.ErrInvalidArg, "unrecognized volume option %q is not supported with local driver", key)
+			//}
 
 			volume.config.Options[key] = value
 		}
diff --git a/libpod/volume_internal_linux.go b/libpod/volume_internal_linux.go
index 67ac41874..bd6c5650b 100644
--- a/libpod/volume_internal_linux.go
+++ b/libpod/volume_internal_linux.go
@@ -33,7 +33,7 @@ func (v *Volume) mount() error {
 	}
 
 	// We cannot mount volumes as rootless.
-	if rootless.IsRootless() {
+	if !v.UsesVolumeDriver() && rootless.IsRootless() {
 		return errors.Wrapf(define.ErrRootless, "cannot mount volumes without root privileges")
 	}
 
@@ -147,8 +147,9 @@ func (v *Volume) unmount(force bool) error {
 			v.state.MountCount = 0
 			return v.save()
 		}
-
-		return errors.Wrapf(define.ErrRootless, "cannot mount or unmount volumes without root privileges")
+		if !v.UsesVolumeDriver() {
+			return errors.Wrapf(define.ErrRootless, "cannot mount or unmount volumes without root privileges")
+		}
 	}
 
 	if !force {

I need some more research into this to confirm that It resolves all issues and doesn't break anything.

The second fix (cannot mount without root error) has very solid foundation behind it: plugins are the way of delegation the hard work to something external with well defined API. Being a black box from podman point of view, this service might or might not require root to function. We don't know this information and should not apply a local-specific heuristics to try to detect that.

@thephoenixofthevoid
Copy link
Contributor Author

Here is latest code of my ongoing rework of famous docker plugin for podman:

https://gist.github.com/thephoenixofthevoid/2ced5a66e94777ad60d6a0b3b54d66c5

It depended on like 3-6 packages like docker-volume-sdk and so on. I examined all of them and inlined only relevant part of these (both relevant for such application and relevant for podman). The goal is to factor out common framework to build arbitrary plugins rapidly. The structure is already clearly visible.

Example:

  1. Host path volume plugin
  2. Remountable volume plugin (like you can attach different USB hard drive and change drives as you want, the same idea applied to volumes)
  3. "Routing Volume": each containers gets a single path in it to use as Unix Domain Endpoint, backed by a HTTP server. A single container acts like a reverse proxy and routes requests based on directory structure of that volume and paths of endpoints. (Poor man's service mesh, it looks like) =)

@mheon
Copy link
Member

mheon commented Mar 7, 2021

If you're willing to submit it as a PR, I would accept the above patch if it was changed to still validate options if the driver was set to local or "".

@thephoenixofthevoid
Copy link
Contributor Author

thephoenixofthevoid commented Mar 7, 2021

It happens here:

if volume.config.Driver == define.VolumeDriverLocal {
logrus.Debugf("Validating options for local driver")
// Validate options
for key := range volume.config.Options {
switch key {
case "device", "o", "type", "UID", "GID":
// Do nothing, valid keys
default:
return nil, errors.Wrapf(define.ErrInvalidArg, "invalid mount option %s for driver 'local'", key)
}
}
}

And there is the check you are talking about right now.

It looks like there are 2 checks of the same thing before normalization of arguments and right after that.

@mheon
Copy link
Member

mheon commented Mar 8, 2021

Alright, this looks fine, then.

@github-actions
Copy link

github-actions bot commented Apr 8, 2021

A friendly reminder that this issue had no activity for 30 days.

@mheon
Copy link
Member

mheon commented Apr 8, 2021

I'm going to go ahead and close, since #9808 merged with the fix.

@mheon mheon closed this as completed Apr 8, 2021
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue
Projects
None yet
Development

No branches or pull requests

3 participants