auto-update: simple rollback

[vrothberg]

Add support for simple rollbacks during podman auto-update. Rollbacks
are enabled by default. If a systemd unit cannot be restarted after an
update, the previous image will be retagged and the unit will be
restarted a second time.

Add system tests for rollbacks. Also fix a bug in the restart sequence;
we have to use the channel to actually know whether the restart was
successful or not.

NOTE: To make rollbacks really useful, users must run their containers
with --sdnotify=container such that the containers send the ready
message over the (mounted) socket. This way, restarting the systemd
units during auto update will block until the message has been received
(or a timeout kicked in).

Signed-off-by: Valentin Rothberg [email protected]

[vrothberg]

@edsantiago, I'd love your eyes on the system tests.

[vrothberg]

FYI: @fatherlinux @mrguitar @nullr0ute

@containers/podman-maintainers PTAL

[vrothberg]

Restarted. Had to make the CI image quay.io/libpod/autoupdatebroken public. It worked locally since I was logged into quay.

[Luap99]

What happens if the application in the container crashes (exits with non zero exit code)? Will the rollback be performed? Or does it only work when podman run -d ... fails.

[vrothberg]

What happens if the application in the container crashes (exits with non zero exit code)? Will the rollback be performed? Or does it only work when podman run -d ... fails.

Good call. I tested only with an empty image. Let me add a test when the command inside the container fails. I guess we need some massaging for that.

/hold

[vrothberg]

What happens if the application in the container crashes (exits with non zero exit code)? Will the rollback be performed? Or does it only work when podman run -d ... fails.

OK. It only fails when podman run -d ... fails. Which makes sense since we are only looking at conmon which has started successfully.

I think we can merge as it. The remainder may require healthchecks.

[vrothberg]

Moving to WIP. I had a chat with @giuseppe to discuss a solution. We gravitated toward letting the container send sdnotify messages such that a restart will wait until the container send message. To do that, we need to massage the generate-systemd code a bit to not always enforce --sdnotify=conmon.

Will tackle that tomorrow.

[edsantiago]

@vrothberg can you make sure the autoupdatebroken image is multiarch-safe please?

[vrothberg]

@vrothberg can you make sure the autoupdatebroken image is multiarch-safe please?

It's now a manifest list with one image for amd64 👍

[vrothberg]

not ok 188 podman auto-update - label io.containers.autoupdate=local with rollback

@edsantiago, do I need to the "socat dance" here as in the sdnotify tests?

[edsantiago]

It's now a manifest list with one image for amd64

We need the actual images (or, the multiarch-testing group does). See https://github.com/containers/podman/blob/cbad5616961520831b1f169f03da2a9f81203f71/test/system/build-testimage for an example of how to produce those images.

[edsantiago]

@vrothberg I'm sorry; I've looked at this one off-and-on today, but been unable to understand what's going wrong. My intuition is telling me "please do not use alpine", because that is docker.io/alpine, which is rate-limited, which is going to randomly screw up CI runs. I don't think that's the cause of the problem, but it's all I can recommend for now.

[vrothberg]

It's now a manifest list with one image for amd64

We need the actual images (or, the multiarch-testing group does).

Can you elaborate on "actual images"?

See https://github.com/containers/podman/blob/cbad5616961520831b1f169f03da2a9f81203f71/test/system/build-testimage for an example of how to produce those images.

I'll add images for arm64, ppc64le and s390x to the manifest list.

[vrothberg]

I'll add images for arm64, ppc64le and s390x to the manifest list.

Done ✔️

$ skopeo inspect --raw docker://quay.io/libpod/autoupdatebroken:latest|jq .
{                                                                                                        
  "schemaVersion": 2,                                                                                    
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",                              
  "manifests": [                                                                                         
    {                                                                                                    
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",                               
      "size": 422,                                                                                       
      "digest": "sha256:40bc10ecedfbea7594e452757b3bd2b2440317e1c296bdf91c4545f539995c67",               
      "platform": {                                                                                      
        "architecture": "amd64",                                                                         
        "os": "linux"                                                                                    
      }                                                                                                  
    },                                                                                                   
    {                                                                                                    
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",                               
      "size": 422,                                                                                       
      "digest": "sha256:a1cdef8749843f33eb5af1ec86dac4728cfa3d940119d37dbda6e9692d2d0592",               
      "platform": {                                                                                      
        "architecture": "arm64",                                                                         
        "os": "linux"                                                                                    
      }                                                                                                  
    },                                                                                                   
    {                                                                                                    
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",                               
      "size": 422,                                                                                       
      "digest": "sha256:ef3e97abba2c003766a660c5ffc957091346509f9e3836585ff2ea84324fb193",               
      "platform": {                                                                                      
        "architecture": "ppc64le",                                                                       
        "os": "linux"                                                                                    
      }                                                                                                  
    },                                                                                                   
    {                                                                                                    
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",                               
      "size": 422,                                                                                       
      "digest": "sha256:bb228dfe14265aeb8f458fad72fbad1b13e42badce0f5eee95c3896e833ab66c",               
      "platform": {                                                                                      
        "architecture": "s390x",                                                                         
        "os": "linux"                                                                                    
      }                                                                                                  
    }                                                                                                    
  ]                                                                                                      
}                                                                                                        

[vrothberg]

I don't think that's the cause of the problem, but it's all I can recommend for now.

Thanks a lot for checking, @edsantiago! I restarted one of the two failed jobs earlier and it passed. It smells like bug/flake that I don't want to introduce with this PR. I repushed with the updates tests to see how it's going now.

[vrothberg]

First run green. Let's see what the second one does.

[vrothberg]

The empirical law: once is never, twice is always :)

PTAL

[rhatdan]

LGTM

[fatherlinux]

This sounds like a great feature. I assume it just tags the penultimate image layer as latest and restarts? It obviously can't go modify the image on the registry right?

[vrothberg]

This sounds like a great feature. I assume it just tags the penultimate image layer as latest and restarts?

That's right. If we have an image quay.io/foo:latest and pull down a new image, the previous one will be untagged. If the restart failed, we'll rollback by retagging the old image with quay.io/foo:latest and doing another restart.

It obviously can't go modify the image on the registry right?

In some cases it could. Are you thinking about some kind of self-healing mechanism to prevent other nodes from pulling a potentially broken image?

[vrothberg]

@containers/podman-maintainers PTAL

[giuseppe]

LGTM

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: giuseppe, vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [giuseppe,vrothberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vrothberg]

ready to merge :)

[rhatdan]

/lgtm

[rhatdan]

/hold cancel