bots: Adjust for Fedora Atomic now being version 28

[martinpitt]

• Publish f28/cockpit container: https://pagure.io/releng/issue/7483
• image-refresh fedora-atomic
• adjust ostree tests for new rollback behaviour

[cockpituous]

image-refresh in progress on cockpit-tasks-mrr3b.
Log: http://fedorapeople.org/groups/cockpit/logs/image-refresh-9090-20180502-151717/

[cockpituous]

Task failed: http://fedorapeople.org/groups/cockpit/logs/image-refresh-9090-20180502-151717/

[martinpitt]

Image build fails with

Trying to pull repository registry.fedoraproject.org/f28/cockpit ... 
manifest for registry.fedoraproject.org/f28/cockpit:latest not found

Indeed there is no f28 container on https://registry.fedoraproject.org/, and https://src.fedoraproject.org/container/cockpit/branch/f28 looks outdated. The f27 branch got updated regularly. Needs investigation and supplying this container.

[martinpitt]

I updated the master and f28 Fedora container branches and did a build:

repositories:
  candidate-registry.fedoraproject.org/f28/cockpit:f28-container-candidate-92075-20180502205553
  candidate-registry.fedoraproject.org/f28/cockpit:0-1.f28container
  candidate-registry.fedoraproject.org/f28/cockpit:0
  candidate-registry.fedoraproject.org/f28/cockpit:latest

However, I don't know how to publish that to actually become a released Fedora Registry container. @stefwalter , do you know?

[stefwalter]

No don't know ... maybe @petervo does. Or @dustymabe

[martinpitt]

#fedora-releng says to file a releng ticket, so I did: https://pagure.io/releng/issue/7483

[martinpitt]

f28 container was published, rebuilding image

[cockpituous]

image-refresh in progress on cockpit-tasks-f85z1.
Log: http://fedorapeople.org/groups/cockpit/logs/image-refresh-9090-20180503-205555/

[cockpituous]

image-refresh fedora-atomic done: https://github.com/cockpituous/cockpit/commits/image-refresh-fedora-atomic-20180503-212525

[martinpitt]

I figured out the fedora-atomic.install failure and pushed a fix. This now formally affects the other atomics too, so triggering tests for them.

[martinpitt]

Two ostree failures, let's retry to see if they are flakes.

[martinpitt]

Debugging check-ostree OstreeRestartCase.testOstree: Screenshot says open(O_TMPFILE): No such file or directory. This is reflected in the journal:

Initiated txn DownloadUpdateRpmDiff for caller :1.36: /org/projectatomic/rpmostree1/fedora_atomic
libostree pull from 'local' for fedora/28/x86_64/atomic-host complete
Txn DownloadUpdateRpmDiff on /org/projectatomic/rpmostree1/fedora_atomic failed: open(O_TMPFILE): No such file or directory

I don't have an immediate idea what this does (there is a lot of setup in the test case to create a local repo). I asked @cgwalters on IRC, and I'll deep-dive into what this does.

[cgwalters]

Seems like a libcurl change FWIW.

[cgwalters]

I filed coreos/rpm-ostree#1360

[martinpitt]

I pushed a workaround for coreos/rpm-ostree#1360; this fixes OstreeCase.testRemoteManagement and gets further in OstreeRestartCase.testOstree; but that now fails like this:

Traceback (most recent call last):
  File "test/verify/check-ostree", line 333, in testOstree
    b.wait_present('table.listing-ct tbody:nth-child(3) i.fa-circle')
  File "/home/martin/upstream/c3/test/common/testlib.py", line 270, in wait_present
    return self.wait_js_func('ph_is_present', selector)
  File "/home/martin/upstream/c3/test/common/testlib.py", line 264, in wait_js_func
    self.wait_js_cond("%s(%s)" % (func, ','.join(map(jsquote, args))))
  File "/home/martin/upstream/c3/test/common/testlib.py", line 261, in wait_js_cond
    self.raise_cdp_exception("timeout\nwait_js_cond", cond, result["exceptionDetails"], trailer)
  File "/home/martin/upstream/c3/test/common/testlib.py", line 166, in raise_cdp_exception
    raise Error("%s(%s): %s" % (func, arg, msg))
Error: timeout
wait_js_cond(ph_is_present("table.listing-ct tbody:nth-child(3) i.fa-circle")): condition did not become true
warning: error: [7] Couldn't connect to server

5 lines before the end.. :-/

After reboot it looks like this:

The test says "After reboot upgrade but no rollback target" - but there actually is. It seems that after upgrading to base.2 and rolling back (to base.1), it now offers a rollback to "base.2", which would actually be a "roll forward"? This sounds like some ostree behaviour change, which might actually be desired? @cgwalters, is rpm-ostree rollback supposed to flip back and forth between the two latest revisions?

On F27 it instead looks like this, i. e. the order is reversed and no rollback option:

On the CLI the behaviour between F27 and F28 are similar:

# rpm-ostree status
State: idle; auto updates disabled
Deployments:
● ostree://local:fedora/28/x86_64/atomic-host
                   Version: cockpit-base.1 (2018-05-09 16:13:32)
                    Commit: c2f8ecaf112c4ba559789198065c8cb30b355c4104f3c9e2b20cef9e3dbb8ff3
              GPGSignature: Valid signature by 95A8BA1754D0E95E2B3A98A7EE15015654780CBD

  ostree://local:fedora/28/x86_64/atomic-host
                   Version: cockpit-base.2 (2018-05-09 16:16:58)
                    Commit: 2f3f0373ccea3b9cbfb332721d571bdb68a6acb413cdd70422d85f499ccf4a2a
              GPGSignature: Valid signature by 95A8BA1754D0E95E2B3A98A7EE15015654780CBD

# rpm-ostree rollback 
Moving '81527dbd85f9033a0be44213b1656664ead2d199bb308262dbd1082cee95c440.0' to be first deployment
Transaction complete; bootconfig swap: no deployment count change: 0
Upgraded:
  cockpit-machines 167.x-1.wip.fc28 -> 99999-2
Downgraded:
  cockpit-ostree 167.x-1.wip.fc28 -> wip-0.1
Removed:
  cockpit-docker-167.x-1.wip.fc28.x86_64
Added:
  empty-1-0.noarch
Run "systemctl reboot" to start a reboot

# rpm-ostree status
State: idle; auto updates disabled
Deployments:
  ostree://local:fedora/28/x86_64/atomic-host
                   Version: cockpit-base.2 (2018-05-09 16:45:33)
                    Commit: 81527dbd85f9033a0be44213b1656664ead2d199bb308262dbd1082cee95c440
              GPGSignature: Valid signature by 95A8BA1754D0E95E2B3A98A7EE15015654780CBD

● ostree://local:fedora/28/x86_64/atomic-host
                   Version: cockpit-base.1 (2018-05-09 16:13:32)
                    Commit: c2f8ecaf112c4ba559789198065c8cb30b355c4104f3c9e2b20cef9e3dbb8ff3
              GPGSignature: Valid signature by 95A8BA1754D0E95E2B3A98A7EE15015654780CBD

I. e. on both images rollback just flips between the two versions. But with the API/JSON, F27 retains a constant order of the revisions while on F28 the order changes.

@cgwalters, does that ring a bell by any chance?

[cgwalters]

rollback has always behaved that way, though I can't say it was really well thought through as far as the UX. The implementation is basically to just swap the bootloader order; the idea here is if something went wrong we don't want to do complex operations.

Stef helped add the concept of deploy partly because of this - it's predictable, whereas rollback takes you to something that depends on history.

Desynchronzation between the CLI and DBus sounds like:

• daemon: Add new Reload D-Bus method coreos/rpm-ostree#1311
• app: Call Reload right after getting sysroot proxy coreos/rpm-ostree#1312

Basically there were some cases where we were either missing a reload or it was racy.

I guess the bottom line is the test cases will probably have to work both ways?

[martinpitt]

I guess the bottom line is the test cases will probably have to work both ways?

Yes, I agree. Either behaviour makes sense, so let's accept both. Thanks Colin!

[martinpitt]

Adjusted the tests for new rollback behaviour.

[larskarlitski]

Thanks a lot! Sorry that there was so much breakage here.