Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InvocationTargetException with java.net.http.HttpClient on v1.2+ #1621

Closed
hxtmdev opened this issue Sep 5, 2023 · 17 comments
Closed

InvocationTargetException with java.net.http.HttpClient on v1.2+ #1621

hxtmdev opened this issue Sep 5, 2023 · 17 comments

Comments

@hxtmdev
Copy link

hxtmdev commented Sep 5, 2023

version

Docker image, works up to 1.1.173, fails from 1.1.174-SNAPSHOT onward

platform

Linux 5.4.0-149-generic Kubernetes pod, nodes run Ubuntu

problem

I am working with a Kubernetes cluster where babashka.http-client/get throws an InvocationTargetException inside pods (no access to the bare node).

babashka/babashka:1.1.173 works
babashka/babashka:1.1.174-SNAPSHOT does not
babashka/babashka:1.3.184 does not
babashka/babashka:1.3.185-SNAPSHOT does not
babashka/babashka:1.3.185-SNAPSHOT-alpine does not

So my first guess would be GraalVM 22.3.1.

The new versions work locally and in a Docker container on my dev machine.

The cluster can run other code like (+ 1 2) just fine, even with the new versions. It's just the combination of whatever http-client does, newer versions, and this environment that seems to be the problem. That's why I'm reporting it here, the http-client code does what it should in other environments so I figured it's probably something in babashka itself.

repro

kubectl run -ti --rm --image babashka/babashka:1.3.185-SNAPSHOT bb -- sh -c "sleep 2; bb '(babashka.http-client/get \"https://babashka.org/\")'"

----- Error --------------------------------------------------------------------
Type:     java.lang.InternalError
Message:  java.lang.reflect.InvocationTargetException
Location: <expr>:1:1

----- Context ------------------------------------------------------------------
1: (babashka.http-client/get "https://babashka.org/")
   ^--- java.lang.reflect.InvocationTargetException

----- Stack trace --------------------------------------------------------------
babashka.http-client.internal/client    - <built-in>
babashka.http-client.internal/fn--27450 - <built-in>
clojure.core/deref                      - <built-in>
babashka.http-client.internal/request   - <built-in>
babashka.http-client/request            - <built-in>
babashka.http-client/get                - <built-in>
user                                    - <expr>:1:1

expected behavior

kubectl run -ti --rm --image babashka/babashka:1.1.173 bb -- sh -c "sleep 2; bb '(babashka.http-client/get \"https://babashka.org/\")'"

{:status 200, :body "[...]", [...], :method :get}}
@borkdude
Copy link
Collaborator

borkdude commented Sep 5, 2023

Does this only happen inside kubectl? Are you able to provide a repro using just Docker that I can run locally? I don't have kubectl on my machine.

@hxtmdev
Copy link
Author

hxtmdev commented Sep 5, 2023

It only happens when running inside this specific Kubernetes Cluster, that's the problem.

The command I provided is just the shortest self-contained command that evaluates (babashka.http-client/get "https://babashka.org/") inside a pod(=~container) in whatever Kubernetes cluster you have configured as your default.

I just tried it inside another cluster and it works just fine. I have not found any other environment, be it containerized, directly on my machine or otherwise where it fails. It's just this specific cluster.

Given the nature of the exception, is this even something where Babashka itself could be at fault or is the root problem almost certainly inside GraalVM and we can't do much more than track it here?

@borkdude
Copy link
Collaborator

borkdude commented Sep 5, 2023

Hard to say. You could try the JVM version of babashka.http-client in the cluster and perhaps it will give a better error message? Does httpkit work for the same request? babashka.http-client is based on java.net.http.

@hxtmdev
Copy link
Author

hxtmdev commented Sep 5, 2023

I found oracle/graal#7182 now, and can confirm that the environment in question has lines with more than 2 : in them in /proc/self/cgroup while the working environments don't. So I'll assume that as the distinguishing feature for now.

I will try to narrow it down with the JVM version tomorrow.

@hxtmdev
Copy link
Author

hxtmdev commented Sep 6, 2023

kubectl run -ti --rm --image clojure:temurin-20-bullseye pod-name -- bash -c "
cat >deps.edn <<.
{:deps{org.babashka/http-client {:mvn/version \"0.3.11\"}}}
.
clj <<.
(require 'babashka.http-client)
(babashka.http-client/get \"https://babashka.org/\")
.
"

succeeds with temurin-{20,17,11}-bullseye, fails temurin-8-bullseye with

user=> Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:387).
java.net.http.HttpClient

Even Docker tags from over a year ago (openjdk-17-tools-deps-1.11.1.1105-bullseye), temurin-19-tools-deps-1.11.1.1165-bullseye (9 months old), and the newest temurin-19 tag temurin-19-tools-deps-bullseye (5 months old) work.

Interestingly, httpkit has the same problem in the same versions: kubectl run -ti --rm --image babashka/babashka:1.3.184 bb -- sh -c "sleep 2; bb '@(org.httpkit.client/get \"https://babashka.org/\")'"

----- Error --------------------------------------------------------------------
Type:     java.lang.InternalError
Message:  java.lang.reflect.InvocationTargetException
Location: <expr>:1:2

----- Context ------------------------------------------------------------------
1: @(org.httpkit.client/get "https://babashka.org/")
    ^--- java.lang.reflect.InvocationTargetException

----- Stack trace --------------------------------------------------------------
org.httpkit.client/make-client        - <built-in>
babashka.impl.httpkit-client/fn--8850 - <built-in>
clojure.core/force                    - <built-in>
org.httpkit.client/request            - <built-in>
babashka.impl.httpkit-client/request  - <built-in>
babashka.impl.httpkit-client/get      - <built-in>
user                                  - <expr>:1:2
clojure.core/deref                    - <built-in>
user                                  - <expr>:1:1

The good news is that with your hint in the direction of java.net.http I managed to reproduce it independently of the wrapping library with the same affected versions

kubectl run -ti --rm --image babashka/babashka:1.3.184 bb -- sh -c "sleep 2; bb '
(-> (java.net.http.HttpClient/newHttpClient)
    (.send (-> (java.net.http.HttpRequest/newBuilder)
               (.uri (java.net.URI/create \"https://babashka.org/\")) .build)
           (java.net.http.HttpResponse\$BodyHandlers/ofString))
    .body)'"
----- Error --------------------------------------------------------------------
Type:     java.lang.InternalError
Message:  java.lang.reflect.InvocationTargetException
Location: <expr>:2:1

----- Context ------------------------------------------------------------------
1: 
2: (-> (java.net.http.HttpClient/newHttpClient)
   ^--- java.lang.reflect.InvocationTargetException
3:     (.send (-> (java.net.http.HttpRequest/newBuilder)
4:                (.uri (java.net.URI/create "https://babashka.org/")) .build)
5:            (java.net.http.HttpResponse$BodyHandlers/ofString))
6:     .body)

----- Stack trace --------------------------------------------------------------
user - <expr>:2:1

To reproduce locally one would likely need to find a way to start a process so that it sees lines with more than 2 : in /proc/self/cgroup which Docker's --cgroup-parent does not do on my machine (apart from refusing names with : to begin with). cgexec did not work on my machine, if it helps I can investigate if such a thing can be done with a bunch of systemd units.

My next thought was trying to rebuild the babashka image with a newer GraalVM but 22.3.X is already the newest one in https://www.graalvm.org/release-notes/.

Anything I can do to clear this up further or do we just wait for a new GraalVM release and that graal issue?

@hxtmdev hxtmdev changed the title InvocationTargetException with http-client/get on v1.2+ InvocationTargetException with java.net.http.HttpClient on v1.2+ Sep 6, 2023
@borkdude
Copy link
Collaborator

borkdude commented Sep 6, 2023

Please try to reproduce this with a JVM as well, to exclude the possibility that this is a bb issue.

@hxtmdev
Copy link
Author

hxtmdev commented Sep 6, 2023

The first repro in my last message was all JVM, no Babashka. The bb in the invocation was just the pod name. I have edited those two characters for clarity.

I could not reproduce with JVM on

clojure:temurin-20-bullseye
clojure:temurin-17-bullseye
clojure:temurin-11-bullseye
clojure:openjdk-17-tools-deps-1.11.1.1105-bullseye (April 2022)
clojure:temurin-19-tools-deps-1.11.1.1165-bullseye (9 months ago)
clojure:temurin-19-tools-deps-bullseye (5 months ago)

I now also tried the direct HttpClient repro and could not reproduce that on clojure:temurin-19-tools-deps-bullseye either (with only JVM, no Babashka)

kubectl run -ti --rm --image clojure:temurin-19-tools-deps-bullseye pod-name -- sh -c "sleep 2; clj -e '
(-> (java.net.http.HttpClient/newHttpClient)
    (.send (-> (java.net.http.HttpRequest/newBuilder)
               (.uri (java.net.URI/create \"https://babashka.org/\")) .build)
           (java.net.http.HttpResponse\$BodyHandlers/ofString))
    .body)'"
WARNING: Implicit use of clojure.main with options is deprecated, use -M
"<!DOCTYPE html>........"

@borkdude
Copy link
Collaborator

borkdude commented Sep 6, 2023

Failure with JDK8 is expected since java.net.http wasn't there at the time.

Try the linux static binary instead to see if that helps. This should be the same binary that goes into the alpine image. You can download it from Github releases as well.

Also invoke bb with bb --debug to see more error output.

@hxtmdev
Copy link
Author

hxtmdev commented Sep 6, 2023

bb --debug on 184

----- Error --------------------------------------------------------------------
Type:     java.lang.InternalError
Message:  java.lang.reflect.InvocationTargetException
Location: <expr>:2:1

----- Context ------------------------------------------------------------------
1: 
2: (-> (java.net.http.HttpClient/newHttpClient)
   ^--- java.lang.reflect.InvocationTargetException
3:     (.send (-> (java.net.http.HttpRequest/newBuilder)
4:                (.uri (java.net.URI/create "https://babashka.org/")) .build)
5:            (java.net.http.HttpResponse$BodyHandlers/ofString))
6:     .body)

----- Stack trace --------------------------------------------------------------
user - <expr>:2:1

----- Exception ----------------------------------------------------------------
clojure.lang.ExceptionInfo: java.lang.reflect.InvocationTargetException
{:type :sci/error, :line 2, :column 1, :message "java.lang.reflect.InvocationTargetException", :sci.impl/callstack #object[clojure.lang.Volatile 0x2346162 {:status :ready, :val ({:line 2, :column 1, :ns #object[sci.lang.Namespace 0x43ede8c0 "user"], :file "<expr>"})}], :file "<expr>"}
 at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:135)
    sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:40)
    sci.impl.interpreter$eval_string_STAR_.invokeStatic (interpreter.cljc:66)
    sci.impl.interpreter$eval_string_STAR_.invoke (interpreter.cljc:57)
    sci.impl.interpreter$eval_string_STAR_.invokeStatic (interpreter.cljc:59)
    sci.core$eval_string_STAR_.invokeStatic (core.cljc:270)
    babashka.main$exec$fn__31991$fn__32029$fn__32030.invoke (main.clj:1043)
    babashka.main$exec$fn__31991$fn__32029.invoke (main.clj:1043)
    babashka.main$exec$fn__31991.invoke (main.clj:1033)
    clojure.lang.AFn.applyToHelper (AFn.java:152)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.core$with_bindings_STAR_.invokeStatic (core.clj:1990)
    clojure.core$with_bindings_STAR_.doInvoke (core.clj:1990)
    clojure.lang.RestFn.invoke (RestFn.java:425)
    babashka.main$exec.invokeStatic (main.clj:823)
    babashka.main$main.invokeStatic (main.clj:1168)
    babashka.main$main.doInvoke (main.clj:1125)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    babashka.main$_main.invokeStatic (main.clj:1201)
    babashka.main$_main.doInvoke (main.clj:1193)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    babashka.main.main (:-1)

bb --debug on 184-alpine only differs below the (core.clj:667) line

----- Error --------------------------------------------------------------------
Type:     java.lang.InternalError
Message:  java.lang.reflect.InvocationTargetException
Location: <expr>:2:1

----- Context ------------------------------------------------------------------
1: 
2: (-> (java.net.http.HttpClient/newHttpClient)
   ^--- java.lang.reflect.InvocationTargetException
3:     (.send (-> (java.net.http.HttpRequest/newBuilder)
4:                (.uri (java.net.URI/create "https://babashka.org/")) .build)
5:            (java.net.http.HttpResponse$BodyHandlers/ofString))
6:     .body)

----- Stack trace --------------------------------------------------------------
user - <expr>:2:1

----- Exception ----------------------------------------------------------------
clojure.lang.ExceptionInfo: java.lang.reflect.InvocationTargetException
{:type :sci/error, :line 2, :column 1, :message "java.lang.reflect.InvocationTargetException", :sci.impl/callstack #object[clojure.lang.Volatile 0x1d1f5b4f {:status :ready, :val ({:line 2, :column 1, :ns #object[sci.lang.Namespace 0x72ff274f "user"], :file "<expr>"})}], :file "<expr>"}
 at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:135)
    sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:40)
    sci.impl.interpreter$eval_string_STAR_.invokeStatic (interpreter.cljc:66)
    sci.impl.interpreter$eval_string_STAR_.invoke (interpreter.cljc:57)
    sci.impl.interpreter$eval_string_STAR_.invokeStatic (interpreter.cljc:59)
    sci.core$eval_string_STAR_.invokeStatic (core.cljc:270)
    babashka.main$exec$fn__31991$fn__32029$fn__32030.invoke (main.clj:1043)
    babashka.main$exec$fn__31991$fn__32029.invoke (main.clj:1043)
    babashka.main$exec$fn__31991.invoke (main.clj:1033)
    clojure.lang.AFn.applyToHelper (AFn.java:152)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.core$with_bindings_STAR_.invokeStatic (core.clj:1990)
    clojure.core$with_bindings_STAR_.doInvoke (core.clj:1990)
    clojure.lang.RestFn.invoke (RestFn.java:425)
    babashka.main$exec.invokeStatic (main.clj:823)
    babashka.main$main.invokeStatic (main.clj:1168)
    babashka.main$main.doInvoke (main.clj:1125)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    babashka.main$_main$f__32104__auto____32114.invoke (main.clj:1207)
    clojure.lang.AFn.run (AFn.java:22)
    java.lang.Thread.run (Thread.java:1589)
    com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine (PlatformThreads.java:775)
    com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine (PosixPlatformThreads.java:203)

@hxtmdev
Copy link
Author

hxtmdev commented Sep 6, 2023

Rearranged things so Clojure gives a reflection warning without the type hint but does not with the hint, results are the same

kubectl run -ti --rm --image babashka/babashka:1.3.184-alpine bb -- sh -c "sleep 2; bb --debug '
(defn f [^java.net.http.HttpClient c]
  (-> c
      (.send (-> (java.net.http.HttpRequest/newBuilder)
                 (.uri (java.net.URI/create \"https://babashka.org/\")) .build)
             (java.net.http.HttpResponse\$BodyHandlers/ofString))))
(f (java.net.http.HttpClient/newHttpClient))'"

@hxtmdev
Copy link
Author

hxtmdev commented Sep 6, 2023

Tossing the currently advertised GraalVM version's native-image at a plain Java

public static void main(String[] args) throws Exception {
    System.out.println(
            HttpClient.newHttpClient()
                    .send(HttpRequest.newBuilder().uri(URI.create("https://babashka.org/")).build(),
                            HttpResponse.BodyHandlers.ofString()));
}

works fails for the "17" and the "20" versions.

big stacktrace
Exception in thread "main" java.lang.InternalError: java.lang.reflect.InvocationTargetException
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:67)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Container.metrics(Container.java:44)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.ContainerInfo.<init>(ContainerInfo.java:34)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.Containers.activeProcessorCount(Containers.java:88)
      at [email protected]/java.lang.Runtime.availableProcessors(Runtime.java:337)
      at [email protected]/java.util.concurrent.ForkJoinPool.<init>(ForkJoinPool.java:2767)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.ForkJoinPoolCommonAccessor.initializeCommonPool(RecomputedFields.java:385)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.ForkJoinPoolCommonAccessor.get(RecomputedFields.java:377)
      at [email protected]/java.util.concurrent.ForkJoinPool.getCommonPoolParallelism(ForkJoinPool.java:330)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.CompletableFutureFieldHolder.<clinit>(Target_java_util_concurrent_CompletableFuture.java:66)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.CompletableFutureAsyncPoolAccessor.get(Target_java_util_concurrent_CompletableFuture.java:58)
      at [email protected]/java.util.concurrent.CompletableFuture.defaultExecutor(CompletableFuture.java:2659)
      at [email protected]/jdk.internal.net.http.HttpClientImpl.<clinit>(HttpClientImpl.java:916)
      at [email protected]/jdk.internal.net.http.HttpClientBuilderImpl.build(HttpClientBuilderImpl.java:143)
      at [email protected]/java.net.http.HttpClient.newHttpClient(HttpClient.java:163)
      at Test.main(Test.java:9)
Caused by: java.lang.reflect.InvocationTargetException
      at [email protected]/java.lang.reflect.Method.invoke(Method.java:578)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:63)
      ... 15 more
Caused by: java.lang.ExceptionInInitializerError
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:78)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupMetrics.getInstance(CgroupMetrics.java:164)
      ... 17 more
Caused by: java.lang.NullPointerException
      at [email protected]/java.util.Objects.requireNonNull(Objects.java:233)
      at [email protected]/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:297)
      at [email protected]/java.nio.file.Path.of(Path.java:148)
      at [email protected]/java.nio.file.Paths.get(Paths.java:69)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupUtil.lambda$readStringValue$0(CgroupUtil.java:57)
      at [email protected]/java.security.AccessController.executePrivileged(AccessController.java:147)
      at [email protected]/java.security.AccessController.doPrivileged(AccessController.java:571)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupUtil.readStringValue(CgroupUtil.java:59)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:66)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemController.getLongValue(CgroupSubsystemController.java:125)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.getLongValue(CgroupV1Subsystem.java:269)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.getHierarchical(CgroupV1Subsystem.java:215)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.setSubSystemControllerPath(CgroupV1Subsystem.java:203)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.initSubSystem(CgroupV1Subsystem.java:111)
      at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.<clinit>(CgroupV1Subsystem.java:47)
      ... 19 more

So it appears that while the problem also occurs in Babashka, it's not specific to Babashka's use of GraalVM but rather a general GraalVM issue.

Do you want to keep this open until a working GraalVM version is used or close it?

@borkdude
Copy link
Collaborator

borkdude commented Sep 6, 2023 via email

@hxtmdev
Copy link
Author

hxtmdev commented Sep 6, 2023

In case you don't get edits via mail: I tested that wrong, it can be reproduced with plain Java+GraalVM.

The stacktrace looks similar to the GraalVM issue.

@borkdude
Copy link
Collaborator

borkdude commented Sep 6, 2023

Do you want to keep this open until a working GraalVM version is used or close it?

Let's keep it open but I recommend submitting your repro to the Graal issue (oracle/graal#7182) as well.

@borkdude
Copy link
Collaborator

bb is compiled using GraalVM 21 nowadays, can you check if this was fixed?

@borkdude
Copy link
Collaborator

I'll re-open if you confirm this needs attention again.

@hxtmdev
Copy link
Author

hxtmdev commented Dec 15, 2023

Confirmed working starting from babashka/babashka:1.3.185-SNAPSHOT as changelog/1.3.185 would suggest 👍🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants