Flux automation released untagged image

[squaremo]

https://github.com/weaveworks/service-conf/commit/01bb051

[squaremo]

It straight away corrected it in the next automation run, but it is strange indeed that it did it in the first place.

[rade]

another one, this time for grafana (the earlier one was scope).

[squaremo]

The logs show fluxd straight-up deciding that the untagged image is the right one:

ts=2018-03-24T10:06:59.884426041Z caller=images.go:62 component=sync-loop service=monitoring:deployment/dev-grafana container=grafana currentimage=quay.io/weaveworks/grafana:master-a830190 msg="added image to changes" newimage=quay.io/weaveworks/grafana

This suggests to me that it's getting bogus information from the cache; either a bug in the code that assembles the available images when automation asks, or a bug in the metadata fetching and storing. Asking LIstImages directly doesn't yield a suspicious result, but it has been updated several times since it the problem occurred.

[rade]

We've seen this in the sock shop too, so users will likely run into it as well.

[squaremo]

See fluxcd/flux#1051

[squaremo]

First time this appears to have happened is https://github.com/weaveworks/service-conf/commit/d4b2a9775, according to

git log --oneline | grep "/[.a-z0-9-]* to automated"

[squaremo]

Full list, showing the commit dates too:

3d8da50e1 Wed Apr 4 01:56:07 2018 +0000 Release quay.io/weaveworks/logging to automated
8b9025d7d Mon Apr 2 07:19:37 2018 +0000 Release quay.io/weaveworks/billing-uploader to automated
cc383bf39 Sun Apr 1 20:26:56 2018 +0000 Release quay.io/weaveworks/grafana to automated
4338e548e Fri Mar 30 16:33:05 2018 +0000 Release quay.io/weaveworks/billing-api to automated
c5af9adb5 Thu Mar 29 20:04:06 2018 +0000 Release quay.io/weaveworks/billing-api to automated
b020930d2 Thu Mar 29 10:03:42 2018 +0000 Release quay.io/weaveworks/scope, quay.io/weaveworks/authfe, quay.io/weaveworks/users to automated
92cc66b08 Thu Mar 29 01:45:09 2018 +0000 Release quay.io/weaveworks/billing-ingester to automated
52a3c7b7c Wed Mar 28 21:42:36 2018 +0000 Release quay.io/weaveworks/users, quay.io/weaveworks/notification-eventmanager to automated
590b0fc9b Wed Mar 28 20:54:46 2018 +0000 Release quay.io/weaveworks/gcp-launcher-webhook, quay.io/weaveworks/billing-api to automated
3f22ad886 Wed Mar 28 10:04:59 2018 +0000 Release quay.io/weaveworks/launcher-service to automated
92bda22b7 Tue Mar 27 19:01:43 2018 +0000 Release quay.io/weaveworks/grafana to automated
002bcd7c3 Tue Mar 27 19:00:36 2018 +0000 Release quay.io/weaveworks/scope, quay.io/weaveworks/launcher-service to automated
1f48e3c7c Tue Mar 27 18:59:29 2018 +0000 Release quay.io/weaveworks/grafana to automated
0b3277c18 Tue Mar 27 00:17:29 2018 +0000 Release quay.io/weaveworks/launch-generator to automated
383446466 Sat Mar 24 10:07:00 2018 +0000 Release quay.io/weaveworks/grafana to automated
01bb0513e Thu Mar 22 17:44:23 2018 +0000 Release quay.io/weaveworks/scope to automated
9d25fcb68 Tue Mar 20 15:52:05 2018 +0000 Release quay.io/weaveworks/authfe to automated
50e3bbd99 Tue Mar 20 15:19:27 2018 +0000 Release weaveworks/kubediff to automated
270bcadb6 Mon Mar 19 10:19:40 2018 +0000 Release quay.io/weaveworks/launch-generator to automated
ae3c39f0c Mon Mar 19 06:01:55 2018 +0000 Release quay.io/weaveworks/grafana to automated
029d4c9e7 Thu Mar 15 06:22:48 2018 +0000 Release quay.io/weaveworks/grafana to automated
bf69a1b23 Mon Mar 12 06:49:24 2018 +0000 Release quay.io/weaveworks/metrics to automated
e1088394b Sun Mar 11 12:44:29 2018 +0000 Release quay.io/weaveworks/billing-ingester to automated
1c867c869 Sun Mar 4 22:27:47 2018 +0000 Release quay.io/weaveworks/billing-ingester to automated
84e66615d Sun Mar 4 08:49:56 2018 +0000 Release quay.io/weaveworks/scope to automated
d5e5de5c9 Thu Feb 8 09:04:42 2018 +0000 Release quay.io/weaveworks/logging to automated
b4f3d2776 Sun Feb 4 15:39:35 2018 +0000 Release quay.io/weaveworks/billing-ingester to automated
74a8b897a Sun Feb 4 13:48:23 2018 +0000 Release quay.io/weaveworks/grafana to automated
bff2e4764 Sat Feb 3 14:28:24 2018 +0000 Release quay.io/weaveworks/grafana to automated
6c12c6aa9 Fri Feb 2 14:29:20 2018 +0000 Release quay.io/weaveworks/scope to automated
1fd3f51c3 Sun Jan 21 21:01:11 2018 +0000 Release quay.io/weaveworks/metrics to automated
0c1b3ced1 Wed Jan 17 11:19:18 2018 +0000 Release quay.io/weaveworks/scope to automated
64eacbdf9 Sun Jan 14 02:30:13 2018 +0000 Release quay.io/weaveworks/logging to automated
58f708e62 Fri Jan 5 22:01:38 2018 +0000 Release quay.io/weaveworks/billing-aggregator to automated
294ef7b6e Fri Jan 5 22:00:59 2018 +0000 Release quay.io/weaveworks/notebooks to automated
d4b2a9775 Wed Dec 27 13:51:32 2017 +0000 Release quay.io/weaveworks/grafana to automated

Sometimes happens a few times on a day, sometimes doesn't happen for about a month :/

[squaremo]

The registry rewrite (fluxcd/flux#851) was merged not long before the first such commit, as were fluxcd/flux#876 and fluxcd/flux#880, which were follow-ons from that. It's likely the problem lies in those, somewhere.

[rade]

There is good chance this got fixed by fluxcd/flux#1251

[rade]

There is good chance this got fixed by fluxcd/flux#1251

We haven't seen it since. -> closing