keyvault: populating the cache using both the Key Vault List and Resources API to workaround incomplete/stale data being returned

[tombuildsstuff]

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave "+1" or "me too" comments, they generate extra noise for PR followers and do not help prioritize for review

Description

Through a significant amount of testing and validation - plus open issues, we've identified an issue when populating the available Key Vaults for the cache, where the Key Vaults List API returns stale data. Having spend a significant amount of time investigating this issue, the API itself appears to only add and remove from the cache - rather than invalidating the cache when something changes - meaning that if there's an issue on the Azure side (or slow replication, etc) the list of available Key Vaults is invalid, and thus Key Vaults which exist are not returned through this API.

Digging into the Azure CLI implementation, the command az keyvault list returns a list of both Key Vaults and Managed HSMS (by default together, but this can be filtered to either type using an argument). Notably however, when loading Key Vaults this is done using an old version of the Resources API (2015-11-01) which is curious - and having spent a decent chunk of time digging into the Azure CLI source code, I can't find a documented reason for this, particularly for an API version this old. By contrast, when obtaining a list of Managed HSMs the Managed HSMs List API is used (that is /subscriptions/XXX/providers/Microsoft.KeyVaults/managedHsms).

Whilst for the life of me I can't find any meaningful context for why the Azure CLI is using this older API, since that's what the Azure CLI is using, it feels reasonable for us to also use that API version (so that users have a consistent experience when querying the available data).

Therefore.. this PR updates the Key Vault caching logic to first query using the Key Vaults List API (that is, /subscriptions/XXX/providers/Microsoft.KeyVaults/vaults) and then to query using the Resources API to populate the list of Key Vaults within the current Subscription. This means we are making an extra API call, but given that both of these APIs (in addition to the Resource Graph API) have known caching issues - I'm honestly not sure what we else we can do here, so this feels like a reasonable compromise.

In my testing this resolves the issues that we've been seeing (now that we can reproduce this) where there's 10 Key Vaults currently in our Subscription, but only 2 are returned when using the Key Vaults List API (even when crossing all pages) - and all 10 are returned via the Resources API - and as such I believe this'll also solve #25548.

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• keyvault - populating the Key Vaults cache using both the Key Vaults List and Resources API to workaround incomplete/stale data being returned from the API [keyvault: populating the cache using both the Key Vault List and Resources API to workaround incomplete/stale data being returned #26070]

Related Issue(s)

Fixes #25548

Dependent on #26069

[manicminer]

Thanks @tombuildsstuff, this LGTM ⚓

[danielino]

Hello @tombuildsstuff , i might have found a bug within the refresh after apply because an error is raised:

i have a disk encryption set that uses a key from a keyvault named xxxxxxxxxxxx (in the logs).

an error is raised because another keyvault (named zzzzzzzzzzzz in logs) is not found.
i don't know why this vault are returned from the api, because it doesn't exists anymore.

│ Error: validating Key Vault Key "https://xxxxxxxxxxxx.vault.azure.net/keys/xxxxxxxxxxxxxxxx" for Disk Encryption Set: retrieving the Resource ID the Key Vault at URL "https://xxxxxxxxxxxxxxxx.vault.azure.net/": populating the Key Vaults cache for Subscription (Subscription: "my_subscription"): retrieving Key Vault (Subscription: "my_subscription"
│ Resource Group Name: "yyyyyyyyyyyyyyy"
│ Key Vault Name: "zzzzzzzzzzzz"): unexpected status 404 (404 Not Found) with error: ResourceGroupNotFound: Resource group 'OTHER_RG' could not be found.

@aronoeldb

[tombuildsstuff]

@danielino thanks for reporting this - I've opened #26199 which should resolve that

[danielino]

@danielino thanks for reporting this - I've opened #26199 which should resolve that

@tombuildsstuff you're awesome, thanks!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.