Support CLJ_JVM_OPTS when downloading the clojure-tools.

[ikappaki]

Hi @borkdude,

The Clojure tools scripts have recently introduced a change to pass the value of the CLJ_JVM_OPTS to the java invocation, so as to configure the JVM while mainly downloading dependencies from the internet. A use case for this, whereby the JVM is directed to switch to the MS-Windows certificate store to connect to the internet as described at https://ask.clojure.org/index.php/12190/certificate-exception-downloading-dependencies-firewall.

This has been recently implemented in deps.clj with #60, but there is an additional unique case that deps.clj that also needs to support: the downloading of the ClojureTools from the Clojure website.

This can't work out of the box, because deps.clj does not invoke the java executable when downloading the tools (so as to pass the CLJ_JVM_OPTS at the java invocation) but rather uses the clojure-tools-jar-download fn in-process to do so:

deps.clj/src/borkdude/deps.clj

Lines 255 to 265 in 864d6c5

	(defn clojure-tools-jar-download
	"Downloads clojure tools jar into deps-clj-config-dir."
	[deps-clj-config-dir]
	(let [dir (io/file deps-clj-config-dir)
	zip (io/file deps-clj-config-dir "tools.zip")]
	(.mkdirs dir)
	(download (format "https://download.clojure.org/install/clojure-tools-%s.zip" @version)
	zip)
	(unzip zip (.getPath dir))
	(.delete zip))
	(warn "Successfully installed clojure tools!"))

Thus, in this particular example case, when the JVM has to use an alternative certificate store, there is no way for the downloading logic to switch to an alternative certificate store, and deps.clj fails right at the beginning.

I think what I'm advocating here is for the tools downloading logic to make use of the CLJ_JVM_OPTS options, and this to work across all potential uses, i.e. when deps.clj is called as a library dependency, or as a native executable or from babashka either natively or via the deps.[.clj|bat] scripts.

Do you have any suggestions how to achieve this? I have an idea that appears to have good potential that I tested it to work in a limited scope, but I don't want to scare you off with it yet :)

Thanks

[borkdude]

@ikappaki When using the deps.exe binary, you can provide system properties like this. This works for any GraalVM binary, also for babashka.

deps -Djavax.net.ssl.trustStoreType=Windows-ROOT <clojure-opts>

However, I'm open to respecting CLJ_JVM_OPTS while downloading, if that helps.

This could potentially work by parsing this environment variable and setting the system properties while running deps.clj. Was that also your idea?

but I don't want to scare you off with it yet

Thanks for being so considerate :)

[ikappaki]

@ikappaki When using the deps.exe binary, you can provide system properties like this. This works for any GraalVM binary, also for babashka.

deps -Djavax.net.ssl.trustStoreType=Windows-ROOT <clojure-opts>

Hi @borkdude, I don't think this would be good enough, since we want to set the CLJ_JVM_OPTS once and work seamlessly across all deps.clj invocations. The user would have to know or remember that they need special handling of deps.clj to make it work the first time (it won't just work telling someone configure CLJ_JVM_OPTS to access the internet and expect them to start using clojure right away). Besides that, I think it might not even work out of the box from tools like Calva, either perhaps because they don't have a configuration options to pass command line options or a user has to go the extra mile to figure this out.

However, I'm open to respecting CLJ_JVM_OPTS while downloading, if that helps.

This could potentially work by parsing this environment variable and setting the system properties while running deps.clj. Was that also your idea?

I think this might work in most cases. Java advanced network configuration (such as proxies and certificates configuration) is done via java properties, so parsing the env variable and setting all -D properties could work (I suppose we also need to restore the properties after we're done). Graalvm appears to allow setting the certificate store at runtime. But I know from experience once these properties are set once in the JVM and an ssl context has been created, these cannot be changed a second time.

The latter, calling deps.clj as a library on the JVM, could cause issues. If the program has already tried to open an ssl context (e.g. attempt to connect to the internet) and then call on deps.clj to download the tools, setting the properties will have no effect (they have already been configurated otherwise). Not sure whether this can be a common case, but we need to keep in mind it is possible to fail at least in one circumstance.

but I don't want to scare you off with it yet

So, here is the simple idea: the clojure-tools download is done by a java invocation passing in as options the contents of the CLJ_JVM_OPTS, as we already do for the other cases :)

But what program do we call with java to perform the download? Clojure is not installed yet so as to run a simple script to download the dependencies, nor do we have any jars with us to call upon; deps.clj in is simplest form is just a simple standalone script that can be invoked from the command line.

The idea is to base64 encode a jar that we build with bb and def it as a blob in the script, that when decoded and saved on disk (say in ~/.deps.clj can be invoked from java with two arguments, the version number and the target path to download the tools to. I've tested the concept to work with an unoptimized jar and it only takes 2k base64 encoded string to put into a def at the bottom of the script, which is mostly outside of the readers view.

So, I can see two options so far

1. At runtime, before downloading the clojure tools, parse CLJ_JVM_OPTS and set all -D properties with the hope that they will take effect. Restore the previous properties after we are done with the download.
2. Embed a tiny jar fie as a base64 string, that when unpacked can be called with java + CLJ_JVM_OPTS to download the clojure-tools. This will require some extra build work so as to make sure that there is a babashka task to build the jar from java source code, and also update the borkdude/deps.clj with the resulting base64 encoded binary.

I think #1 could work in most of the cases unless testing or someone else demonstrates otherwise. #2 is the ultimate solution on par what we do with dependencies download, that is likely to be future proof.

What do you recommend?

Thanks,

[borkdude]

To get a sense of the importance of this issue: is this a problem you are currently experiencing, or just a problem that might happen to someone, theoretically?

Alternatives:

• We could compile a Java program on the fly and execute that, iff CLJ_JVM_OPTS is set and when a download needs to happen. From Java 9+ on Java even allows you to run single .java files directly. I prefer keeping the code readable as embedding some obfuscated code might seem suspicious.

• Previously deps.clj used Powershell or curl to download. I later implemented that in Clojure because it was less OS-specific. Would it help anything to go back?

• Do nothing: deps.clj already prints what file should be downloaded to which location.

[ikappaki]

To get a sense of the importance of this issue: is this a problem you are currently experiencing, or just a problem that might happen to someone, theoretically?

As an example, I've seen this issue happening in Calva when the CLJ_JVM_OPTS has been correctly set to use the proxy settings/certificate store, but Calva can't bootstrap deps.clj because it can't download tools.zip the first time it needs it.

Alternatives:

• We could compile a Java program on the fly and execute that, iff CLJ_JVM_OPTS is set and when a download needs to happen. From Java 9+ on Java even allows you to run single .java files directly. I prefer keeping the code readable as embedding some obfuscated code might seem suspicious.

Sure, this sounds like an all encompassing solution and much clearer than embedding a blob. The only minor disadvantage is that it does not support Java 1.8.

• Previously deps.clj used Powershell or curl to download. I later implemented that in Clojure because it was less OS-specific. Would it help anything to go back?

I don't think so, assuming that CLJ_JVM_OPTS has been configured on the user's PC so as to allow access to the internet, we should not be using any other tooling (such as PS) that might require additional configuration to connect to the internet.

• Do nothing: deps.clj already prints what file should be downloaded to which location.

This would be frustrating to newcomers that have no experience with Clojure and not sure why the need this. Ideally, I would like to tell to people with no previous experience in Clojure, to set up this variable to let Clojure tooling connect to the internet, download Calva and that's all they need to get them started. Calva is just given as an example here.

Thanks

[borkdude]

@ikappaki I'm open to an experimental and minimal PR for downloading the tools.jar stuff, which:

• Writes a Java program (which only uses standard JDK classes) to a temporary dir/file on disk
• Compiles that file (to support Java 8 we still have to do this)
• Runs that file, with the CLJ_JVM_OPTS set

I think we only have to do this if the CLJ_JVM_OPTS environment variable is non-nil. In other cases, we can still rely on the old method.

Are there any systems around that don't have javac but only java? In that case, we have to fall back to the old method as well.

[ikappaki]

Hi @borkdude,

let me take a step back and explain the driver behind all this, so you have a better view of why we are doing this in case you have other better ideas.

It is common nowadays in large organisations to control internet traffic and this results to extra setup required for applications to connect to the wider internet. If we are to support clojure tools in these enclaves (which to me is very important for extending the clojure reach) then we need to have a way to address the network configuration part. in the tools The traditional way to address this was to introduce specific handling for specific parts of the configuration, e.g. http[s]_proxy env variable parsing. This has worked good so far until the traffic controllers have become much stricter nowadays requiring even more low level configuration (i.e. access to private root certificates stored elsewhere). I've presented the issue to the ask clojure forum and suggested a few options how to set the certificate store (as a new command line option, in a global tools' .edn file, in an environment variable). Puredanger felt that introducing a new environment variable, CLJ_JVM_OPTS, is the better option. This has the advantage that it's relatively future proof, since we can pass any other options to it in the future, and that it is something you configure once in the user's workstation and applies to all clojure tools invocation, so the user doesn't have to configure the tools for each project resulting in a much improved user experience.

Having solved this issue for the official clojure tools, I'm trying to apply the same solution to the other most essential tools like deps.clj and (later) babashka. deps.clj is almost identical to the official clojure tools script, except this part we are addressing now (just a reminder the goal here is for the tools to be able to connect to the internet from the enclave without any further configuration other than consulting the CLJV_JM_OPTS variable).

@ikappaki I'm open to an experimental and minimal PR for downloading the tools.jar stuff, which:

* Writes a Java program (which only uses standard JDK classes) to a temporary dir/file on disk

* Compiles that file (to support Java 8 we still have to do this)

* Runs that file, with the `CLJ_JVM_OPTS` set

I think we only have to do this if the CLJ_JVM_OPTS environment variable is non-nil. In other cases, we can still rely on the old method.

Are there any systems around that don't have javac but only java? In that case, we have to fall back to the old method as well.

This is getting slightly more complicated than I first envisaged, but I have no issue going with it. Another option perhaps could be that we use the java9+ runtime compilation facilities and fallback to the old way and if that fails to a message like "please download and install the clojure tools manually" on java1.8? I think the user codebase using java1.8 must be minimal and ready to deal with any shortcomings manually. This will address the javac presence. I don't know if there are any installations without javac (does old JREs come with javac?).

Thanks!

[borkdude]

Sounds good.

We can assume Java 9+ going forward for the "CLJ_JVM_OPTS" download method and fall back to the old method otherwise.

[ikappaki]

Sounds good.

We can assume Java 9+ going forward for the "CLJ_JVM_OPTS" download method and fall back to the old method otherwise.

Hi @borkdude, could you point me to an example or documentation to compile java source at runtime with the Java9+ facility? The only relevant reference I found is using the javax.tools library, but this requires a javac compiler and can made to work in both jdk8 and jdk9 as per this contrived example: https://blog.jooq.org/how-to-compile-a-class-at-runtime-with-java-8-and-9/. Is this what you had in mind perhaps in the first place?

Thanks

[borkdude]

public class Foo {
    public static void main(String [] args) {
        System.out.println("Hello");
    }
}

java Foo.java.

That's it.

[borkdude]

Note: java, not javac