podman run --storage-opt size for container root filesystem quota is not honored in xfs based quota enabled backend

[vikas-goel]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman run command does not honor --storage-opt size=10G option.
The storage graph driver is set to overlay/overlay2 and the storage backend being xfs file-system with prjquota option enabled. When a container is started with the said option, there is no effect of it. The container can continue to fill the root file-system that may create denial of service once the hosts file-system is full.

In similar Docker situation, the command is honored and the container cannot write more than the size specified in the docker run --storage-opt size=<size> command line argument.

I have tried size = 10G attribute under [storage.options.overlay2] and [storage.options.thinpool] sections in /etc/containers/storage.conf file but in vain. When I set size = 10G under [storage.options] section (undocumented), then the containers cannot write more than the specified size. The podman run --storage-opt size command line argument is still ignored. As the size attributed in /etc/containers/storage.conf file is global, I am unable to set the size per container.

Steps to reproduce the issue:

1. Create a xfs file-system based backed storage for /var/lib/containers with project quota on
   /dev/sdb1 on /var/lib/containers type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)

2. Start a container with podman run --storage-opt size=10G option

3. Login to the container and run dd if=/dev/zero of=/fill.txt.

Describe the results you received:
The dd command continues to write beyond 10G and fills the backend storage on the host

Describe the results you expected:
The dd command should abort after around writing 10G of data

[root@flex-vm-02 ~]# podman exec -it tme-mas-01 bash
bash-4.2# dd if=/dev/zero of=/fill.txt
dd: writing to '/fill.txt': No space left on device
19139409+0 records in
19139408+0 records out
9799376896 bytes (9.8 GB) copied, 58.0655 s, 169 MB/s
bash-4.2#

Additional information you deem important (e.g. issue happens only occasionally):
The issue happens consistently

Output of podman version:

Version:      3.0.2-dev
API Version:  3.0.0
Go Version:   go1.15.7
Built:        Tue Mar  2 07:10:06 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-1.module+el8.4.0+10198+36d1d0e3.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 0a5175681bdd52b99f1f0f442cbba8f8c126a1c9'
  cpus: 8
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: flex-vm-02.dc2.ros2100.veritas.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-293.el8.x86_64
  linkmode: dynamic
  memFree: 14152466432
  memTotal: 33511845888
  ociRuntime:
    name: runc
    package: runc-1.0.0-70.rc92.module+el8.4.0+10198+36d1d0e3.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 16921104384
  swapTotal: 16924012544
  uptime: 62h 19m 4.72s (Approximately 2.58 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 2
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1614697806
  BuiltTime: Tue Mar  2 07:10:06 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.0.2-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.1-3.module+el8.4.0+10198+36d1d0e3.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):
Red Hat Enterprise Linux 8.4 Beta
VMware virtual machine

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[vikas-goel]

Hello @rhatdan ,

Would #9911 fix this issue?

The latest podman RPM is not available for Centos Stream 8. So, I am unable to verify it.

[rhatdan]

If the issue is just handling of the command line option, then probably yes. If the xfs quota handling is broken, then no.

If you setup storage-opt size in storage.conf, does it work?

[vikas-goel]

When I set ‘size’ in ‘[storage.options]’ section (undocumented), then it works.

[rhatdan]

Then yes #9911 should fix this.

[vikas-goel]

/reopen

[openshift-ci]

@vikas-goel: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vikas-goel]

Verified on the main branch build. There storage-opt size=5G is not honored.

[rhatdan]

Is it failing or not? IE You are just able to build an image larger then 5G

[vikas-goel]

Is it failing or not? IE You are just able to build an image larger then 5G

With podman run --storage-opt size=5G command, the dd command inside the container kept increasing the file size until I manually cancelled the operation. At that point, the file size was already 70GB.

The podman inspect <container> did not have any indication of the storage opt.

[rhatdan]

Try:

podman --storage-opt size=5G run ...

I tried both and only in the option before the run did the command work.

[vikas-goel]

That works when running manually using podman command. However, it doesn't work with docker-compose.

[rhatdan]

Ok that is a separate issue.

[rhatdan]

Open another Issue. Lets fix this issue first.

[rhatdan]

@vikas-goel If you move the --storage-opt before the run, does your test work?

[vikas-goel]

Yes, it works when moved before.

[rhatdan]

Ok, I am working on remote now.

[vikas-goel]

Thank you @rhatdan for making the change. I built the podman binary from main branch and verified that it works when the storage-opt size is passed after podman run.

The docker-compose still doesn't work. Do I need to replace some other binary/library or does that require a separate fix?

[rhatdan]

No I need to continue working on it. It is a complex fix. Will need to modify containers/storage as well.

[vikas-goel]

I've created #11016 to track the Docker Compose issue

[Chaz6]

Has there been any progress on this?

[rhatdan]

This should be available now.