Always return 404 when used with Ambassador Auth Service

[ReeSilva]

Describe the bug

I don't know if I missing something. I'm trying to implement Oathkeeper + Ambassador. I first have followed the blog post but now I'm trying to change to Oathkeeper v0.16. It works well and get my access rules from files through Kubernetes config map (which I'll put below, there is no sensitive data since it's an internal PoC implementation).

If I do not enable Auth Service on Ambassador I can get my rules from Oathkeeper, so they are all there. But, if I activate Auth Service in Ambassador, it returns 404 for all my endpoints, including the Oathkeeper, a behavior that looks like I have no access rules configured. When I was config following the Ory Blog Post about Oathkeeper and Ambassador, it works well (except for the fact that if fails when I was trying to use Oauth2 Introspection)
Here is the files for my config map for access rules and also the yml for oathkeeper api in Kubernetes: https://gist.github.com/ReeSilva/4fbd581368a1f2e52f98eed04ed401da

Reproducing the bug

1. Run a Kubernetes Cluster
2. Deploy Ambassador v0.70.0
3. Create an Ambassador Service to expose API Gateway
4. Deploy Oathkeeper v0.16.0 with Ambassador Mapping
5. Run curl http://${AmbassadorServiceAddress[:AmbassadorServicePort]/${oathkeeper-mapping}/health/ready} and it should works ok
6. Create a rule with any authenticator/authorizer/mutator for Oathkeeper
7. Enable AuthService on Ambassador pointing to Oathkeeper Service and /decisions endpoint
8. Run curl http://${AmbassadorServiceAddress[:AmbassadorServicePort]/${oathkeeper-mapping}/health/ready} and it should returns 404

Server logs

It doesn't have any log for API Calls

Take security seriously and subscribe to the ORY Security Newsletter. Stay on top of new patches and security insights.                                                                                                

>> Subscribe now: http://eepurl.com/di390P <<
time="2019-05-23T20:52:05Z" level=info msg="Authenticator oauth2_token_introspection did not specify pre-authorization which is thus disabled"
time="2019-05-23T20:52:05Z" level=info msg="Software quality assurance features are enabled. Learn more at: https://www.ory.sh/docs/ecosystem/sqa"
time="2019-05-23T20:52:05Z" level=info msg="TLS has not been configured for proxy, skipping"
time="2019-05-23T20:52:05Z" level=info msg="Listening on http://:4455"
time="2019-05-23T20:52:05Z" level=info msg="TLS has not been configured for api, skipping"
time="2019-05-23T20:52:05Z" level=info msg="Listening on http://:4456"

Server configuration

Here are the .yaml for configmap with Access Rules and also the .yaml for Oathkeeper K8s Service: https://gist.github.com/ReeSilva/4fbd581368a1f2e52f98eed04ed401da

Expected behavior

It should authenticate, authorize and mutate and returns the response from the service

Environment

• Version: Ambassador v0.70.0, Oathkeeper v0.16.0
• Environment: K8s in Minikube

[aeneasr]

If Oathkeeper doesn't log anything then it's not receiving any requests, indicating something else is wrong which does not really relate to Oathkeeper itself.

[ReeSilva]

@aeneasr Thanks for ur answer. Ok, that's a good way to start, I was thinking that no log on API was some implementation for v0.16. So I'll see if there's anything wrong with comunication inside K8s network and give a try. Thank u again.

[ReeSilva]

@aeneasr Well, after some hours of test I still have no success. The most strange thing is if I take the same configuration but, instead of use the Oathkeeper I use a example auth service given by Ambassador, it works ok. And, if I curl /health/ready endpoint of Oathkeeper from Ambassador pods it works ok as well.

I'm still trying but if u remember something that resemble like that, I'll appreciate.

Another info that might help is that if I use the same K8s files but instead of use v0.16 I use v0.13.8 and instead of point to /decisions I point to judge, it works like a charm. And I've followed all changelogs from v0.13.8 to v0.16.0 and there is nothing than the change from /judge to /decisions

[aeneasr]

Sorry, I missed updates on this (thanks for the ping on discord). I'll investigate this in the next days. What would help me is if you set up a step-by-step guide using e.g. curl that "fakes"/imitates the request from Ambassador, that way I can quickly reproduce the behaviour and potentially find the flaw at hand.

[ReeSilva]

No problem, man. Sure, as soon as I get in home I'll try to use a proxy to capturate the request from Ambassador tô Oathkeeper to send to u. Also I'll send the entre Kubernetes config so u can, if u need, reproduce the whole chain

[aeneasr]

I was able to reproduce the issue where the log does not show any requests to /decisions

[aeneasr]

I've identified the issue and fixed it and added an e2e test to make sure this doesn't regress at some point. The root cause was a mis-use of a negroni middleware function (using With instead of Use to chain middlewares) which is also why this wasn't detected by the integration and unit tests (hence e2e).

Thank you for the report, fix is going to be merged on master today and will be released soon after.

[ReeSilva]

Yay, good news. I'll give a try ASAP and will return. Thank u a lot, @aeneasr

[ReeSilva]

Works like a charm \o