system tests: safer container-stop signaling

[edsantiago]

Having a container spin-wait on a /stop file, then exit, is
unsafe: 'podman exec $ctr touch /stop' can get sucked into
container cleanup before the exec terminates, resulting in
the podman-exec failing and hence the test failing.

Most existing instances of this pattern are unnecessary.
Replace those with just 'podman rm -f'.

When necessary, use a variety of safer alternatives.

Re-Closes: #10825 (already closed; this addresses remaining cases)

Signed-off-by: Ed Santiago [email protected]

None

[vrothberg]

LGTM
I am curious. Do some of the fixed tests show up in your flake data, @edsantiago ?

@Luap99 PTAL

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [edsantiago,vrothberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Luap99]

/lgtm

[edsantiago]

Do some of the fixed tests show up in your flake data

This one is too infrequent: most logs have cycled away, and my reports from 2021-22 are worthless: they do not include excerpts nor test names. (I have since improved my reporting habits).

Since I still have text logs archived on my laptop I could conceivably run a multi-line grep -B10 "rc=137.*xpected" ... |grep "podman.*exec.*touch" and cross-reference that to tests... but I'm not feeling strongly motivated to do so. I believe the pattern described above ("spin-wait on file, run exec to touch it") is dangerous, and I have fixed all such instances, and I don't have to worry about this kind of flake any more. Do you see a reason to pursue this further? (Serious question. If you do, please let me know, and I'll invest that effort).

[vrothberg]

Thanks, @edsantiago! Please don't waste time pursuing this further. I was mostly wondering if there are some ripple effects of this PR that would allow closing a number of flakes at once.