Errors when re-launching container after updates to /dev

[GDYendell]

This happened after a reboot

❯ run-dev-c7.sh
restarting stopped dev-c7 container 2.0.3 ...
Error: unable to start container "2f5d49718c55683cf3bc029fd5c6add0e8f8286951c4fdefe2fc7d82695ab315": /usr/bin/crun: error stat'ing file `/dev/snd/hwC0D0`: No such file or directory: OCI runtime attempted to invoke a command that was not found
Error: can only create exec sessions on running containers: container state improper

A similar thing happens with the epics 7 dev container (https://github.com/epics-containers/ioc-adaravis):

❯ cdev-launch 
+ podman start ioc-adaravis
Error: unable to start container "8c202faf0eb9b6513e310deeb9361be9b2f53ae03b8e6976b8eeaf23cb572687": runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting "/dev/loop0" to rootfs at "/dev/loop0" caused: stat /dev/loop0: no such file or directory: OCI runtime attempted to invoke a command that was not found
+ podman exec -it ioc-adaravis bash
Error: can only create exec sessions on running containers: container state improper

Both are fixed by podman rm or equivalent, e.g.

❯ run-dev-c7.sh -d
2f5d49718c55683cf3bc029fd5c6add0e8f8286951c4fdefe2fc7d82695ab315
creating new dev-c7 container 2.0.3 ...
300f67b0691c4926650a9131a3da2901c2d3ffc4320b38fe58dce9b47217ee86
...
⬢ [dev-c7:2.0.3] ❯

[ronaldomercado]

I can confirm that I get this everyday when I power up my PC.

[gilesknap]

So you power cycle!? I'll have to try that to see if it affects me.

[ronaldomercado]

I copy below the error from this morning. I find that it's a different 'dev' component every time.

$ run-dev-c7.sh
restarting stopped dev-c7 container 2.0.4 ...
Error: unable to start container "3739061bf0722355adedd57d1ca591df6799bf26de050c9358c5dd9a2cbaa9bd": /usr/bin/crun: error stat'ing file /dev/vcsa63: No such file or directory: OCI runtime attempted to invoke a command that was not found
Error: can only create exec sessions on running containers: container state improper

It's "cured" okay using the "-d" switch (delete).

$ run-dev-c7.sh -d
3739061bf0722355adedd57d1ca591df6799bf26de050c9358c5dd9a2cbaa9bd
creating new dev-c7 container 2.0.4 ...
68b6c35f1441956e1c0697a5bddeed3f288c4df1e35fbc8e0c8d0f7a39493b2d
[C7] 2022-Aug-18 07:34[rjq35657@pc0122 ~]$

[aawdls]

I got this too just now. No reboot for me though

[gilesknap]

Are you sure you did not reboot? This message happens when we start the container and it only stops on reboot (maybe also on logout)

[aawdls]

I logged out, but did not reboot.
Message was

Error: unable to start container "29d99244831fb02aaf5c38a492ce0e1ddbe1bd3d56d2e59224abe4555f1422cb": /usr/bin/crun: error stat'ing file `/dev/hidraw4`: No such file or directory: OCI runtime attempted to invoke a command that was not found
Error: can only create exec sessions on running containers: container state improper

[gilesknap]

OK this is an issue with podman that is fixed in v4.2
containers/podman#13899
https://github.com/containers/podman/releases/tag/v4.2.0

We await its release into RHEL8 ...

...

...

[gilesknap]

OK I found a workaround to this.

The following changes to the podman start command will fix the issue.

remove:

    --privileged 

make sure this option is included:

    --security-opt=label=type:container_runtime_t

Dropping --privileged means it does not map all the /dev files and avoids the podman bug.
The security option means that we still have access to the same filesystem mounts that we were getting through --privileged.

I have upated these with this change:

• k8s-peics-utils dev-functions.sh
• vscode-python-workspace devcontainer.json
• dev-c7 devcontainer.json and run-dev-c7.sh