Cirrus: new API clobbers past (flake) runs

[edsantiago]

The new logformatter URL mechanism, after #14608, has a serious flaw: flake URLs are lost. Reason: the new URLs include the BuildID, and TaskName, but not the TaskID. TaskID is the crucial one for linking to a flake log. Without TaskID, we have something like /build/12345/int this that, but the int this that on a successful run clobbers the one of the flaked run.

@cevich let's talk about this upon your return please.

[cevich]

Oof. Yes, the Cirrus-API will always reflect the "latest" artifact, so a re-run of a task for any reason will effectively clobber API access to any "old" runs. So if I understand correctly, you need a way to link to the annotated log output from a specific run of a task, preferable by TaskID. The tricky part is now we've got multiple clouds storing both logs and artifacts, so we need some kind of proxy or API to unify the URLs...or we need to store the annotated logs ourselves in a dedicated location.

Some ideas:

1. Ask Cirrus nicely, if they would update their REST API to support URLs with <TASK NAME, ID, OR ALIAS>. Since $CIRRUS_TASK_ID is available at runtime, you can populate your links using that.
2. Use (a hopefully working podman) in an always: script instruction for each task. In that, run the gcsupload container image and stuff the annotated log into a dedicated GCS bucket using a naming scheme of our own choosing. Update all links to reference the dedicated bucket and our own object naming scheme.
3. Give up on referencing the annotated version of flake logs, instead just point at https://cirrus-ci.com/task/<$CIRRUS_TASK_ID>?logs=main#L<LINE NUMBER> raw log. Not ideal, but simple.

That's all I've got for now, but I'll keep thinking about it.