Ignore timestamp labels during sorting and release of images

[hiddeco]

In #2176 a whitelist option for the usage of image timestamps from
labels was implemented. This PR did however not remove the
CreatedTS method, with as a result that some elements of Flux like
fluxctl would still consult the timestamp parsed from labels, rather
than relying on the cache decorated info.CreatedAt.

This commit removes the method, and replaces any calls made to it
with info.CreatedAt, so that a timestamp from a label is not
mistakenly used while the image has not been whitelisted.

[2opremio]

Can you add a test which would had showed the problem this PR is fixing?

[hiddeco]

@2opremio I will try to find some time.

[hiddeco]

@2opremio the problem is fairly simple, labels should always be ignored and only be used by the cache decorator in case an image has been whitelisted to replace the CreatedAt date. I have updated the OrderByCreationDate test to show the labels are now ignored while present in the metadata.

[AMoghrabi]

Hey @hiddeco -- Sorry to bother but I have a question regarding this change. We recently upgraded from flux version 1.12.3 to 1.16.0. We are receiving the following error because our timestamp label does not follow RFC3339:

ts=2020-04-16T03:05:43.739832359Z caller=repocachemanager.go:260 component=warmer canonical_name=xxx.azurecr.io/xxx auth={map[]} err="failed to parse 1 timestamp label(s) as RFC3339 (org.opencontainers.image.created); ask the repository administrator to correct this as it conflicts with the spec" ref=xxx.azurecr.io/xxx:xxxxx

and also:

 container=xxx repo=xxx.azurecr.io/xxx pattern=glob:develop-* current=xxx.azurecr.io/xxxxx warning="image with zero created timestamp" current="xxx.azurecr.io/xxxxx (0001-01-01 00:00:00 +0000 UTC)" latest="xxx.azurecr.io/xxxxx (2020-04-15 21:02:36.516841899 +0000 UTC)" action="skip container"

We have several images that are producing this error which stops them from being deployed by Flux. We are trying to fix them but this is blocking us from upgrading flux quite a bit and fixing them all will take a while.

This change looks like it's for fluxctl but I thought I'd double check I'm not overlooking something. Is there some sort of flag and/or annotation we can add to fluxctl to turn this behavior off?

Thanks a lot!

[hiddeco]

@AMoghrabi the former log message is just a mere warning, it has no impact on Flux' automation. The latter log message is however a problem for Flux, as it needs the timestamps from all images to be able to perform automation operations.

If the image has a creation timestamp but Flux says it doesn't, can you please try clearing the image metadata cache (scale up/down memcached) to see if this resolves the issue?

[AMoghrabi]

Thanks a lot for your response @hiddeco. I've deleting the memcached pod and haven't had any luck. I could try scaling it that would produce different results (though I don't think that's the case).

I've checked another service that is having the same issue. Inspecting the image which is reported to have a zero-timestamp, I see this:

  "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }

But in an issue reported on this repo, one of the developers mentioned Flux does not look at LastTagTime, so we can likely ignore this.

Inspecting the labels we add to the image during a docker build, I see this:

"org.opencontainers.image.created": "2019-08-30T15:34:45"

I think this is what Flux is complaining about as it isn't following the RFC spec, and maybe Flux does not know how to parse and read this format so it results in a zero-timestamp. Does that sound about right?

Otherwise, on the image I also see this field: "Created": "2019-08-30T15:35:47.746126694Z",, which seems to be in the correct format.

It seems like our only option is to fix these timestamps publish a new image. Afterwards, we need to manually redeploy the image without the help of Flux as Flux would continue to struggle, as the currently running image does not have a proper timestamp and cannot do a comparison. What are your thoughts?

Let me know if you'd like me to open an issue instead of going forward in this PR.

Thanks again!

Edit: I forgot to check... I think this particular service may still be able to deploy, I haven't looked into it. I think it causes a problem when we update the timestamp label to something that conforms to the RFC spec. But I'll need to test that still. I'l open to any ideas you may have though, thanks.