AAD Stopped Working

[Silvenga]

What happened: AAD managed cluster no longer accepts authentication, even with admin keys.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Occurs with both the admin credentials and user credentials:

rm ~/.kube/config
az aks get-credentials --name aks --overwrite-existing
kubectl get nodes

The behavior of this command has been altered by the following extension: aks-preview
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code BLAH to authenticate.
error: You must be logged in to the server (Unauthorized)

Anything else we need to know?:

We've also "Upgraded to AKS-managed Azure AD Integration" - https://docs.microsoft.com/en-us/azure/aks/managed-aad#upgrading-to-aks-managed-azure-ad-integration

Considering the docs just changed, and this just started today, I'm assuming something was changed on aks.

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"c02cd11cc8bced1391937fe271c3b9c9fe9befa0", GitTreeState:"clean", BuildDate:"2020-06-24T19:57:20Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

• Size of cluster (how many worker nodes are in the cluster?) 2
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.) HTTP Microservices
• Others:

[aristosvo]

Because AKS-managed AAD is GA since yesterday, it might have something to do with that.

Maybe removing the aks-preview extension for az cli and updating az cli to the latest version will solve the problem.

[Silvenga]

So (lightly embarrassing).

I've been using WSL2 for everything Linux related (as the rest of my team). Turns out there's a clock drift problem (my WSL2 instance was over 2 days off from the host's clock) - microsoft/WSL#5324. I assumed the standard Hyper-V Time synchronization services would be enabled, they are not (and of course ntp cannot run as a service under WSL2, yet...).

Forcing a re-sync via ntp does allow everything to work again.

So very bad timing on our side, it was definitely one of those "correlation is not causation" things. My clock just happened to be just enough off for my requests to become invalid, right when the feature went GA.