KIAM 3.3+ fails to grant role in Azure

[joseph-burns]

Problem:

KIAM images above version v3.2 do not function as expected in Azure environments.

Environment:

K8s Version: 1.19
OS: Centos 7
CNI: Weave

Multiple types of requests (S3, secretsmanager) return a 400 when attempting to authenticate. The agent logs show:
{"addr":"192.168.192.2:33546","duration":1,"headers":{"Content-Length":["11"],"Content-Type":["text/html"],"Date":["Fri, 02 Jul 2021 13:29:16 GMT"],"Server":["Microsoft-IIS/10.0"]},"level":"info","method":"PUT","msg":"processed request","path":"/latest/api/token","status":400,"time":"2021-07-02T13:29:16Z"}
When running the same configurations in AWS, we see that the requests are successfully authenticated through the KIAM agents.

When using the same manifest for KIAM server & agent, but using v3.2 images, the requests are successful. However, we're unable to do so due to security vulnerabilities in that version. IPTables, and everything else we could think to check seemed to be in working order (and, again, only changing the version of KIAM back to 3.2, everything functions).

Is there some setting we need to apply post v3.3 when in Azure, perhaps? We've tried adjusting gateway timeouts, but from the time of the test run it is clear these are not being reached.

Interesting to note as well the size of the request - when running in AWS it is always 56, but the failed requests in Azure are always 11, and return a 400.

[udoniyor]

We faced a similar issue when deploying Kiam on our Azure cluster. This happens due to Azure having a meta-server with the same IP address and API pattern. Azure meta service expects a specific header that is not present when AWS SDK makes the query, and and as a result returns 400. AWS SDK can function fine if you give it 403 or 404, but for some reason the developers decided to make it stop working if it gets 400 error.

Kiam has a pass through for that specific token end point, which is located here. So to get around the issue, I removed the lines, to prevent the pass-through. Then just built the image. Kiam has working fine for us for couple of months since.

[joseph-burns]

@udoniyor thank you. We'll give it a try. For clarity, I guess this means you have 1 image you'd use just in Azure, and one you'd use in AWS or elsewhere?

[udoniyor]

We have only one cluster on Azure where we are using Kiam, at the moment we are not using Kubernetes on AWS. Based on my experience, you will need the custom image only on Azure, maybe GCP as well, but have not tried on them.

[joseph-burns]

@udoniyor is it just the remoteAddr="" part that needs removed?
Can we force it to not call that proxy handler? I'd like to implement this in a clean(ish) manor, as well as not require a separate image for different clouds. I guess ideally it'd be configurable as to whether to have this new behavior or not. Willing and able to commit this back, I'm just wondering how you had done it.

[udoniyor]

@joseph-burns
The change I made on our end is a bit on hacky side, ideally this should probably a command line argument to enable the pass-through or not. I just removed the code responsible to allowing the pass-through.

[joseph-burns]

I attempted this (struggled a lot with the dockerfile not working as intended) - it didn't seem to change the behavior at all. Again, though, I did struggle with the Dockerfile, it would not run docker build correctly.