CI SystemTests: FAIL: SELinux role should always be system_r

[cevich]

Issue Description

In CI, with updated Rawhide, and Fedora images (c20230706t200047z-f38f37d13 at time of opening issue), the entire matrix fails on three tests with the same/similar error:

Failed tests (3):

• 419 podman selinux: container with label=disable
• 420 podman selinux: privileged container
• 425 podman selinux: pid=host

Sample Error:

# [13:11:39.570157592] # /var/tmp/go/src/github.com/containers/podman/bin/podman run --rm --privileged --userns=host quay.io/libpod/testimage:20221018 cat -v /proc/self/attr/current
# [13:11:40.003076552] unconfined_u:unconfined_r:spc_t:s0^@
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #|     FAIL: SELinux role should always be system_r
# #| expected: '.*_u:system_r:.*' (using expr)
# #|   actual: 'unconfined_u:unconfined_r:spc_t:s0^@'
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Steps to reproduce the issue

Steps to reproduce the issue

1. Allow PR CI to run w/ updated VM images

Describe the results you received

Most/all the Fedora & Rawhide matrix fails on these three tests.

Describe the results you expected

CI should pass

podman info output

$SCRIPT_BASE/logcollector.sh podman
+  ./bin/podman system info
------------------------------------------------------------
host:
  arch: amd64
  buildahVersion: 1.32.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 55.37
    systemPercent: 16.93
    userPercent: 27.7
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: cloud
    version: "39"
  eventLogger: journald
  freeLocks: 2045
  hostname: cirrus-task-6298176643137536
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.5.0-0.rc0.20230705gitd528014517f2.10.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2950213632
  memTotal: 4091301888
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230627.g289301b-1.fc39.x86_64
    version: |
      pasta 0^20230627.g289301b-1.fc39.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-14.fc39.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 4090490880
  swapTotal: 4090490880
  uptime: 0h 33m 28.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: mirror.gcr.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
    PullFromMirror: ""
  docker.io/library:
    Blocked: false
    Insecure: false
    Location: quay.io/libpod
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io/library
    PullFromMirror: ""
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 213588619264
  graphRootUsed: 2787540992
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.0-dev
  Built: 1690305995
  BuiltTime: Tue Jul 25 12:26:35 2023
  GitCommit: f737e5bbe936c407760b8a93777eda757eb8bae8
  GoVersion: go1.20.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0-dev
------------------------------------------------------------

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

These three tests have been failing for a few months, ever since beginning testing with updated images beyond c20230426t140447z-f38f37d12.

Additional information

Please see #18612 for history.

[rhatdan]

This is the new SELinux policy, which I opened a PR a long time to fix, but never moved forward.

#18439

If you can just grab that patch it should fix the problems.

[cevich]

Ahh yeah, I suspected so. I opened this issue as an "anchor" to skip these tests for now. But if that PR is closer/ready now, I'll happily cherry-pick it into #18612 Thanks!

[rhatdan]

If the PR does not work, skip the tests and I will fix them after we move forward. The issue I had was I could never get the new VMs to work for other reasons.

[cevich]

Ahh okay, so maybe my PR has already addressed some/all of those reasons, let's see...

[edsantiago]

I truly have no idea what's going on with this issue, but FWIW tests are still failing on f38 and rawhide and RHEL gating tests. Super-trivial to reproduce, just run the selinux tests on any of them.

container-selinux-2.218.0-1.fc39.noarch
container-selinux-2.219.0-1.fc38.noarch

[cevich]

...in the image update PR, the results are much better with Dan's commit.

@edsantiago Dan's commit seems to be the key. My original intent here was to point here from skip() added to the tests. That seems to not be necessary now.

@rhatdan can we merge #18439 w/o the (broken) image update (in that PR), or does the fix cause tests to break on the old CI VM images? If the later, I'm happy to include it in #18612 which seems like it may be ready to merge soon-ish? I hesitate, because my PR has been a (typical) roller-coaster of problems.

[rhatdan]

Yes it will fail with the old image, take it and merge away. I will be very happy when this gets updated.

[cevich]

Yes it will fail with the old image

Ahh okay great. I built new images yesterday and tried again and it's still passing with your commit. I'll leave it cherry-picked in while we work out the last few remaining (unrelated) hiccups.

[AdamWill]

I don't know the whole story here, but the SELinux tests still failed for https://bodhi.fedoraproject.org/updates/FEDORA-2023-1734b09012 four days ago. Is the updated image not yet actually pushed to wherever those tests use?

[edsantiago]

@AdamWill this has been a nightmare for months. I have a strong mind to just disable those tests. And in fact I will do so today, but unfortunately it'll take some time for that to propagate to an actual build.

I am sorry for the hassle.

[AdamWill]

np. I'm waiving the failures for the 4.6.1 updates, since the same failures were waived for 4.6.0.

[cevich]

It case it helps, there's some unfortunate history here: The 4.6 (and 4.6.1?) branch was cut before the CI VM image update PR could get past all it's merge-hurdles. It might be possible to implement the new (F37/38) images from main, into the 4.6 (and 4.6.1?) branches. Or it might be a nightmare.