auto update: return restart error

[vrothberg]

Return the error when restarting the unit failed during an update. The task is correctly marked to have failed but we really need to return the error to the user.

@edsantiago 🤞

[NO NEW TESTS NEEDED] - The flakes in #17607 will reveal errors.

Does this PR introduce a user-facing change?

Return errors from systemd when restarting a unit failed during an auto update. 

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vrothberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Luap99]

LGTM

[edsantiago]

Well, here's the error message [source]:`

Error: restarting unit container-c_image_OBvhQIJzmV.service during update: expected "done" but received "failed"

[vrothberg]

This is really poking in the dark. I added some more debug logs to the test in hope to get something useful.

Really no idea what's going on on Debian. It's always the same pattern that restarting the systemd units fails during auto update.

Why is it only flaking on Debian? @edsantiago have you seen these tests fail on Fedora as well but maybe at such a low frequency that it fell under the (flake) radar?

[edsantiago]

There are only two Fedora flakes in auto-update, and both are the "1 vs 2" flake:

#|     FAIL: Number of images updated from registry.
#| expected: '2'
#|   actual: '1'

• fedora-37 : sys podman fedora-37 root host
  • PR test/system: fix wait_for_port() to wait for bind #17707
    • 03-14 10:15 in [sys] podman auto-update with multiple services
• fedora-37-aarch64 : sys podman fedora-37-aarch64 root host
  • PR system service --log-level=trace: support hijack #17781
    • 03-14 10:35 in [sys] podman auto-update with multiple services

[edsantiago]

Still happening, still debian. Is there anything in that log that can help you?

[vrothberg]

Still happening, still debian. Is there anything in that log that can help you?

I'd need to clone myself at the moment to find time to look into it, sorry :( Will take a look as soon as I find time.

[vrothberg]

@edsantiago are you cool to merge or shall I remove the echos in the tests?

[edsantiago]

/lgtm

[vrothberg]

Thanks! My current "feeling" is that after this change we will only see podman auto-update failures as we now return the error and exit non-zero. All other flavors of failures in the tests should disappear.

I will look into the dbus/systemd dependency we use and check whether a bug may linger there. Other than that, we may try to restart the service another time if the first attempt has failed. But that would be my last resort as it feels like giving up.