podman run not displaying output from commands

[TomSweeneyRedHat]

/kind bug

Description
When doing a podman run command that has a command at the end of it, the output from the container is not displayed on the calling terminal. These commands were run as root.

Steps to reproduce the issue:

1. image=$(podman pull fedora)

2. podman run $image ls -alF /etc

Describe the results you received:

A slight pause and then a return to the command line prompt.

Describe the results you expected:

A listing of the container's /etc directory.
Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

# podman version
Version:       0.8.3
Go Version:    go1.10.3
OS/Arch:       linux/amd64

Output of podman info:

# podman info
host:
  Conmon:
    package: podman-0.8.3-1.git9d09a4d.fc28.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: c72f69ce3f3fd7e46dced78dd99b67ab5441441e-dirty'
  MemFree: 1039663104
  MemTotal: 2089201664
  OCIRuntime:
    package: runc-1.0.0-46.dev.gitb4e2ecb.fc28.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 2147479552
  SwapTotal: 2147479552
  arch: amd64
  cpus: 1
  hostname: localhost.localdomain
  kernel: 4.17.14-202.fc28.x86_64
  os: linux
  uptime: 37m 46.07s
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

Run as root on a VM running a fully updated Fedora 28 machine as of 8/20/18

[TomSweeneyRedHat]

From journalctl:

Aug 20 14:12:42 localhost.localdomain audit: NETFILTER_CFG table=filter family=2 entries=82
Aug 20 14:12:42 localhost.localdomain systemd[1]: Started libpod-conmon-f505ba4fcb8c9bc6b065708ab0a0e9cef65bf39992a9f20ccb0157fb69439d5e.scope.
Aug 20 14:12:42 localhost.localdomain systemd[1]: libcontainer-3629-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Aug 20 14:12:42 localhost.localdomain systemd[1]: libcontainer-3629-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Aug 20 14:12:42 localhost.localdomain systemd[1]: Created slice libcontainer_3629_systemd_test_default.slice.
Aug 20 14:12:42 localhost.localdomain systemd[1]: Removed slice libcontainer_3629_systemd_test_default.slice.
Aug 20 14:12:42 localhost.localdomain systemd[1]: Started libcontainer container f505ba4fcb8c9bc6b065708ab0a0e9cef65bf39992a9f20ccb0157fb69439d5e.
Aug 20 14:12:42 localhost.localdomain kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Aug 20 14:12:42 localhost.localdomain audit[3640]: AVC avc:  denied  { read write } for  pid=3640 comm="ls" path="/dev/null" dev="tmpfs" ino=68254 scontext=system_u:system_r:container_t:s0:c341,c746 tcontext=system_u:object_r:container_file_t:s0:c341,c746 tclass=chr_file permissive=0
Aug 20 14:12:42 localhost.localdomain audit[3640]: AVC avc:  denied  { map } for  pid=3640 comm="ls" path="/usr/bin/coreutils" dev="dm-0" ino=16963562 scontext=system_u:system_r:container_t:s0:c341,c746 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
Aug 20 14:12:42 localhost.localdomain audit[3640]: ANOM_ABEND auid=0 uid=0 gid=0 ses=1 subj=system_u:system_r:container_t:s0:c341,c746 pid=3640 comm="ls" exe="/usr/bin/coreutils" sig=11 res=1
Aug 20 14:12:42 localhost.localdomain systemd[1]: libpod-f505ba4fcb8c9bc6b065708ab0a0e9cef65bf39992a9f20ccb0157fb69439d5e.scope: Consumed 26ms CPU time

[mheon]

Can't reproduce locally (fully-updated F28, Podman built from master), but those AVCs look suspicious. @rhatdan PTAL

[TomSweeneyRedHat]

with setenforce 0, works fine. Wasn't reproducible on F27.

[TomSweeneyRedHat]

----
time->Mon Aug 20 14:14:28 2018
type=AVC msg=audit(1534788868.697:444): avc:  denied  { read write } for  pid=3783 comm="ls" path="/dev/null" dev="tmpfs" ino=70067 scontext=system_u:system_r:container_t:s0:c861,c927 tcontext=system_u:object_r:container_file_t:s0:c861,c927 tclass=chr_file permissive=1
----
time->Mon Aug 20 14:14:28 2018
type=AVC msg=audit(1534788868.697:445): avc:  denied  { map } for  pid=3783 comm="ls" path="/usr/bin/coreutils" dev="dm-0" ino=16963562 scontext=system_u:system_r:container_t:s0:c861,c927 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1
----
time->Mon Aug 20 14:14:28 2018
type=AVC msg=audit(1534788868.697:446): avc:  denied  { read execute } for  pid=3783 comm="ls" path="/usr/bin/coreutils" dev="dm-0" ino=16963562 scontext=system_u:system_r:container_t:s0:c861,c927 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1
----
time->Mon Aug 20 14:14:28 2018
type=AVC msg=audit(1534788868.697:447): avc:  denied  { open } for  pid=3783 comm="ls" path="/etc/ld.so.cache" dev="dm-0" ino=16878420 scontext=system_u:system_r:container_t:s0:c861,c927 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

[rhatdan]

This is the container-selinux package did not install properly issue,

yum -y update container-selinux
or
yum -y reinstall container-selinux
restorcon -R -v /var/lib/containers

And you should be good to go.

[TomSweeneyRedHat]

FWIW,

dnf is telling me I'm up to date when I do dnf -y update container-selinux

# rpm -q container-selinux
container-selinux-2.69-1.git452b90d.fc28.noarch

After doing the dnf -y reinstall:

# rpm -q container-selinux
container-selinux-2.69-1.git452b90d.fc28.noarch

Which looks to be the same, I did the restorecon -R -v /var/lib/containers after the update command and still had issues. However after doing the reinstall command then the restorecon a second time, it worked.

After doing all that, then the baseline tests worked too.

Is there anything we can do kit wise to keep this from happening?

[rhatdan]

Did this happen on a fresh system?

[TomSweeneyRedHat]

It happened on two clones and then to triple check I installed F28 from ico, installed docker, did a 'dnf -y update' then dnf -y install podman*.rpm. It happened there as well. That's what has me the most concerned is it seems to be happening on a newly installed machine.

[baude]

so whats the result here? no longer a bug?

[rhatdan]

It is fixed in an update container-selinux package.