Error: open /var/lib/containers/storage/overlay-containers/.../userdata/.../exit/...: no such file or directory

[pbrezina]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I'm running Ansible with podman connection plugin inside GitHub Action to provision and build container images. I get randomly this error:

Failed to set execute bit on remote files (rc: 255, err: Error: open /var/lib/containers/storage/overlay-containers/533d4811e95ebd5c35560110dc65388896cda5a195cfc1af40d27589f43cd488/userdata/98371086d912c4e42684ff33d30271131ed2fc583fa5a44f7db4c10b464c7493/exit/533d4811e95ebd5c35560110dc65388896cda5a195cfc1af40d27589f43cd488: no such file or directory

It happens when Ansible tries to call chmod a+x on a file that it wants to execute, it goes through the code down to podman exec and I believe that podman exec produces this error mentioned in err field.

It is only reproducible in GitHub-hosted runner and only sporadically. The runner uses podman 3.4.2 version.

Steps to reproduce the issue:

1. Run this workflow inside GitHub-hosted runner.

Describe the results you received:

Error.

Describe the results you expected:

No error.

Additional information you deem important (e.g. issue happens only occasionally):

It happens only occasionally.

Output of podman version:

podman version 3.4.2

Output of podman info --debug:

N/A

Package info (e.g. output of rpm -q podman or apt list podman):

N/A

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No (3.4.2 is latest available version on Ubuntu 20.04)

Additional environment details (AWS, VirtualBox, physical, etc.):

GitHub-hosted ubuntu-latest runner.

[giuseppe]

@mheon could #13600 fix this issue as well?

[mheon]

Potentially? Depends if this is the same race. @pbrezina Do you have a more complete reproducer?

[mheon]

(By that, I mean one that ideally doesn't require a Gitlab install)

[pbrezina]

Unfortunately I don't have any other reproducer. The description in the PR sounds like it may fix it, besides "If an exec session is created that exits almost instantly, but generates a large amount of output (e.g. prints thousands of lines)". Ansible only runs chmod in this case and that will not generate thousands of lines.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

This is potentially fixed and noone has reported it again. Reopen if the fix actually did not work.