OIDC authentication fails if used with proxy

[satyachillale]

Environment Details

• Helidon Version: 2.5.1
• Helidon MP
• JDK version: 11
• OS:Linux
• Docker version (if applicable):

---

Problem Description

OIDC identity provider is deployed in a public network.
When connected Helidon MP application behind a corporate proxy, OIDC fails to reach the identity provider, even with proxy configurations.

The problem can be consistently reproduced.

stack trace

[trmqa@slc14xtz console]$ java ${JAVA_FLAGS}  -jar target/otmm-console.jar
2022.07.14 21:28:41 INFO io.helidon.common.LogConfig Thread[main,5,main]: Logging at initialization configured using classpath: /logging.properties
2022.07.14 21:28:42 WARNING org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider Thread[main,5,main]: Class class [Ljava.lang.String; has null package
2022.07.14 21:28:42 WARNING org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider Thread[main,5,main]: Class class [Ljava.lang.String; has null package
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.util.concurrent.CompletionException: io.helidon.webclient.WebClientException: http://129.80.8.224:8080/realms/tmmdev/protocol/openid-connect/certs
	at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)
	at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376)
	at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1093)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158)
	at io.helidon.webclient.WebClientRequestBuilderImpl.lambda$invoke$20(WebClientRequestBuilderImpl.java:605)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:262)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: io.helidon.webclient.WebClientException: http://129.80.8.224:8080/realms/tmmdev/protocol/openid-connect/certs
	... 17 more
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /129.80.8.224:8080
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
	... 8 more

Steps to reproduce

1. Provide OIDC identity configurations of an identity provider in public network.
2. Connect to a VPN/Corporate network that cannot reach public resource without a proxy.
3. Configure proxy settings as below.
4. Run the application

Expectation: The application run without any errors.
Actual: Above reported error is thrown

I have used the below configurations.
Configurations:

server:
  static.classpath:
    location: "/WEB"
    context: "/consoleui"

client:
  proxy:
    use-system-selector: false
    host: "www-proxy.us.oracle.com"
    port: 80
    no-proxy: [ "localhost" ,"127.0.0.1", ".oraclecorp.com" , ".docker.oraclecorp.com" ,".us.oracle.com" , ".oci.oraclecorp.com" ]

security:
  config.require-encryption: false
  jersey:
    enabled: true
  providers:
    - oidc:
        proxy-protocol: "http"
        proxy-host: "www-proxy.us.oracle.com"
        proxy-port: 80

[klustria]

I'm not getting the Timeout Error:

Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /129.80.8.224:8080

The timeout error is a proxy error and has nothing to do with Helidon.

Instead I'm getting this error:

Exception in thread "main" io.helidon.common.Errors$ErrorMessagesException: FATAL: Failed to load metadata: java.util.concurrent.CompletionException: io.helidon.webclient.WebClientException: Request failed with code 404 from https://150.136.243.50:8877/realms/helidon-course/.well-known/openid-configuration at java.util.concurrent.CompletionException: io.helidon.webclient.WebClientException: Request failed with code 404
        at io.helidon.common.Errors.checkValid(Errors.java:183)
 

and on keycloak, I'm seeing this problem:

2022-09-10 08:00:49,204 DEBUG [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-0) Error response 404: javax.ws.rs.NotFoundException: RESTEASY003210: Could not find resource for full path: https://150.136.243.50:8877https://150.136.243.50:8877/realms/helidon-course/.well-known/openid-configuration

Only happens when proxy is set.

This issue will focus on investigating the 404 http error as the timeout error originally mentioned as the problem is not a Helidon issue. The 404 http error I'm getting means that my environment is able to work with proxy and is able to reach the Identity Server, albeit just failing.

[klustria]

Using Wireshark I have observed this request and response when using proxy (which fails):

GET http://localhost:8877/realms/helidon-course/.well-known/openid-configuration HTTP/1.1
HTTP/1.1 404 Not Found , JavaScript Object Notation (application/json)

while without proxy, it looks like this (which succeeds:

GET /realms/helidon-course/.well-known/openid-configuration HTTP/1.1 
HTTP/1.1 200 OK , JavaScript Object Notation (application/json)

From the message request, it can be seen that the one with proxy uses absolute URL while the one without uses relative URL. This seems to be similar issue as this: #3577 which was fixed using a flag in this PR: #3614.

[klustria]

How to reproduce Using Oracle Corp Proxy:

1. Install KeyCloak as OIDC Server using Chapter 8 Practice 2 of the internal Helidon Course on a remote OCI instance. The setup however needs to be modified to use SSL port as it won't work otherwise since the installation is remote. Use Server Installation - Setting Up SSL Guide to help with the setup. When starting KeyCloak use --https-port to define the SSL port to use. Also add --log-level=DEBUG as an option to KeyCloak to be able to see the error.
2. Use the Application describe in the same Helidon Course Practice but set it up to use the server certificate created from previous step by importing it into the Application's default JDK KeyStore.
3. Set up the application to use the proxy by adding this in the application's config under security.providers.oidc:

        force-https-redirects: true
        proxy-protocol: "http"
        ## Oracle Proxy
        proxy-host: "www-proxy-hqdc.us.oracle.com"
        proxy-port: 80

5. Build and run the application.

[klustria]

How to reproduce Using locally installed mitmproxy:

1. Install KeyCloak as OIDC Server using Chapter 8 Practice 2 of the internal Helidon Course on local machine. Add --log-level=DEBUG as an option to KeyCloak to be able to see the error.
2. Install mitmproxy in your machine: https://mitmproxy.org/#mitmproxy. When ran, this will start a proxy on localhost:8080.
3. Set up Application as describe in the same Helidon Course and add mitmproxy's host and port on the application's config under security.providers.oidc:

        proxy-protocol: "http"
        ## mitmproxy
        proxy-host: 127.0.0.1
        proxy-port: 8080

4. Build and run the application.
5. This approach will be easier to debug as all the pieces are in local environment and no SSL communication was setup to complicate investigation.
6. Use WireShark to intercept the communication between the proxy and the OIDC server by using a capture filter of tcp port 8080 and host 127.0.0.1. This how the behavior described in OIDC authentication fails if used with proxy  #4644 (comment) was observed.

[klustria]

The behavior looks identical as what was observed in #3577. It was fixed via PR: #3614 using a flag but seems like not ideal for an MP app. The goal is now to investigate why with proxy, absolute URI is used in the PATH (which fails) while without proxy, it uses relative URI (which succeeds).

[klustria]

As a test, I commented lines 752 759, 760 from

helidon/webclient/webclient/src/main/java/io/helidon/webclient/WebClientRequestBuilderImpl.java

Lines 751 to 761 in 78db0f7

	static URI relativizeNoProxy(URI finalUri, Proxy proxy, boolean relativeUris) {
	if (proxy == Proxy.noProxy() || proxy.noProxyPredicate().apply(finalUri) || relativeUris) {
	String path = finalUri.getRawPath();
	String fragment = finalUri.getRawFragment();
	String query = finalUri.getRawQuery();
	StringBuilder sb = new StringBuilder();
	constructRelativeURI(sb, path, query, fragment);
	return URI.create(sb.toString());
	}
	return finalUri;
	}

to remove the condition where it will only give relative URI when there is no Proxy or if the host is is in the no-proxy list or if forced via the flag/config property added via 3614. This change made the communication between Application->Proxy->OIDC to work.

[klustria]

Using curl to send the same request via proxy also works:

curl -k -v --http1.1 "http://127.0.0.1:8877/realms/helidon-course/.well-known/openid-configuration"

and WireShark showed this request/response result:

GET /realms/helidon-course/.well-known/openid-configuration HTTP/1.1
HTTP/1.1 200 OK , JavaScript Object Notation (application/json)

[klustria]

Bottomline, question is why do we use absolute URI path when using proxy as this causes the recipient to receive the URL with hostname and port duplicated, hence causing a 404:

2022-10-25 18:54:32,995 DEBUG [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-0) Error response 404: javax.ws.rs.NotFoundException: RESTEASY003210: Could not find resource for full path: http://localhost:8877http://localhost:8877/realms/helidon-course/.well-known/openid-configuration

[klustria]

The use of absolute request URI on Proxy and relative request URI on non-proxy were reported and changed in #2302 and #3438. The relevant information of using absolute URI when there is proxy is mentioned in 5.1.2 Request-URI of https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html which states the following:

The absoluteURI form is REQUIRED when the request is being made to a proxy. The proxy is requested to forward the request or service it from a valid cache, and return the response. Note that the proxy MAY forward the request on to another proxy or directly to the server

specified by the absoluteURI. In order to avoid request loops, a proxy MUST be able to recognize all of its server names, including any aliases, local variations, and the numeric IP address. An example Request-Line would be:

       GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.1
To allow for transition to absoluteURIs in all requests in future versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI form in requests, even though HTTP/1.1 clients will only generate them in requests to proxies.

[klustria]

The goal now is to see if PR #3614 which uses relative-uris config flag to allow forcing of request URIs to use relative path can be used for OIDC config.

[klustria]

PR #5267 is the potential fix for this issue. Basically, the goal is to add relative-iris boolean flag in OIDC config to force internal webclient in OIDC to use relative URI if needed.

[klustria]

PR #5267 has been reviewed and merged.