[stress] Sync namespace federated credentials periodically and on startup #9063
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The original implementation was skipping lookups to see if credentials already existed, on the assumption that deletion was always being handled. This resulted in multiple credentials being created for the same namespace.
Instead we can simply poll to make sure deleted namespaces don't continue to take up a federated identity credential slot as a failsafe. We also sync credential state on startup and keep an in-memory cache, so that namespace deletions/creations re-use existing credentials. There could be some edge cases here with the cache, but I think it's a better trade-off than always querying the managed identity pool every time a namespace is created, as the operation is slow and can increase stress deploy times (we need to wait on completion before deploying otherwise there is a race condition with the credential being created and the deploy container running).
EDIT: my original changes assumed DELETE events weren't being caught, but it's actually because I had added the below code. Now that is removed and the delete event handler method has been added back.