Flux uses registry credentials from other namespaces

[bootc]

Describe the bug
Flux appears to be using credentials from other namespaces when accessing our private registry, and is thus failing to check for image updates.

The logs below show the relevant (and slightly redacted) lines for one of our environments with 5 applications in 5 different namespaces, all pulling from the same registry and with the same Secret name. app1:Secret/gitlab-registry can only see the registry.example.com/tcl/app1 tags, and so on. For some reason Flux is picking one of our apps (app5 in the example below) and using that credential to query the tags on all the images including app1 through app4. In effect the credential is bleeding across namespaces.

To Reproduce
Steps to reproduce the behaviour:
0. Create two namespaces with two different Docker registry secrets, with the same name and same registry host.

1. Deploy something to give Flux some images to look for.
2. Notice that Flux is making queries to the registry with only one set of credentials for both namespaces.

Expected behavior
It should use the credential only from the relevant namespace.

Logs

ts=2020-01-06T11:12:05.871370995Z caller=warming.go:180 component=warmer canonical_name=registry.example.com/tcl/app1 auth="{map[registry.example.com:<registry creds for [email protected], from app5:secret/gitlab-registry>]}" err="requesting tags: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
ts=2020-01-06T11:12:21.692306974Z caller=warming.go:180 component=warmer canonical_name=registry.example.com/tcl/app2 auth="{map[registry.example.com:<registry creds for [email protected], from app5:secret/gitlab-registry>]}" err="requesting tags: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
ts=2020-01-06T11:12:31.457550778Z caller=warming.go:180 component=warmer canonical_name=registry.example.com/tcl/app3 auth="{map[registry.example.com:<registry creds for [email protected], from app5:secret/gitlab-registry>]}" err="requesting tags: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
ts=2020-01-06T11:12:42.844377867Z caller=warming.go:180 component=warmer canonical_name=registry.example.com/tcl/app4 auth="{map[registry.example.com:<registry creds for [email protected], from app5:secret/gitlab-registry>]}" err="requesting tags: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"

Additional context
Add any other context about the problem here, e.g

• Flux version: 1.17.0 (also seen with 1.16.0, but 1.15.0 is fine)
• Helm Operator version: [not relevant]
• Kubernetes version: 1.14 (via EKS) and 1.15 (via RKE)
• Git provider: GitLab
• Container registry provider: GitLab

[2opremio]

I think that the problem was introduced in #2520

I am looking into it.

[2opremio]

BTW, thanks a lot for the detailed bug report and for pinpointing in what version the bug was introduced.

[2opremio]

@bootc I have just created a fix for this. See #2728 . It will be included in the next release.

[bootc]

Thanks @2opremio! Is there (or will there be) an image of Flux I can test with before it's released?

[2opremio]

I can create one for you. Give me a few minutes

[2opremio]

You can use 2opremio/flux:scope-imagepullsecrets-correctly-a802915a to test that it works

[bootc]

Yes, that looks like it fixes the issue for me. Many thanks! 👍

[2opremio]

@bootc The fix for this issue has been released with 1.17.1