Deadlock while fetching dependencies

[pbzdyl]

Lein process is stuck while downloading dependencies with the following setup (I've minimized the reproduction case to that configuration):

We have the following Git projects (with links to their repos in Gitlab):

• lib-a - standalone, independent project
• lib-b - refers lib-a as its direct dependency
• lib-c - refers lib-a and lib-b as its direct dependencies
• app - refers lib-a, lib-b and lib-c as its direct dependencies

Empty ~/.m2/repository and ~/.gitlibs directories (a common situation in clean CI builds without any caches).

Steps to reproduce:

1. Clone the app repository

git clone https://gitlab.com/lein-git-down-test/app.git

2. Make sure local m2 repo ~/.m2/repository and ~/.gitlibs cache are empty:

mv ~/.m2/repository ~/.m2/repository_backup
mv ~/.gitlibs ~/.gitlibs_backup

3. Try to build the app project

cd app
lein jar

Output:

Retrieving reifyhealth/lein-git-down/0.4.0/lein-git-down-0.4.0.pom from clojars
Retrieving org/clojure/tools.gitlibs/1.0.100/tools.gitlibs-1.0.100.pom from central
Retrieving org/clojure/pom.contrib/0.3.0/pom.contrib-0.3.0.pom from central
Retrieving org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.pom from central
Retrieving org/eclipse/jgit/org.eclipse.jgit-parent/4.10.0.201712302008-r/org.eclipse.jgit-parent-4.10.0.201712302008-r.pom from central
Retrieving com/jcraft/jsch/0.1.54/jsch-0.1.54.pom from central
Retrieving org/sonatype/oss/oss-parent/6/oss-parent-6.pom from central
Retrieving com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.pom from central
Retrieving org/sonatype/oss/oss-parent/5/oss-parent-5.pom from central
Retrieving com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy/0.0.9/jsch.agentproxy-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.pom from central
Retrieving net/java/dev/jna/jna/4.1.0/jna-4.1.0.pom from central
Retrieving net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.pom from central
Retrieving com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.pom from central
Retrieving com/jcraft/jsch/0.1.49/jsch-0.1.49.pom from central
Retrieving com/jcraft/jzlib/1.1.3/jzlib-1.1.3.pom from central
Retrieving lein-pprint/lein-pprint/1.3.2/lein-pprint-1.3.2.pom from clojars
Retrieving com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar from central
Retrieving org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.jar from central
Retrieving org/clojure/tools.gitlibs/1.0.100/tools.gitlibs-1.0.100.jar from central
Retrieving com/jcraft/jsch/0.1.54/jsch-0.1.54.jar from central
Retrieving com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.jar from central
Retrieving net/java/dev/jna/jna/4.1.0/jna-4.1.0.jar from central
Retrieving net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.jar from central
Retrieving com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.jar from central
Retrieving com/jcraft/jzlib/1.1.3/jzlib-1.1.3.jar from central
Retrieving reifyhealth/lein-git-down/0.4.0/lein-git-down-0.4.0.jar from clojars
Retrieving lein-pprint/lein-pprint/1.3.2/lein-pprint-1.3.2.jar from clojars
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
Retrieving org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
Retrieving lib-a/lib-a/master/lib-a-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-a/lib-a/b3140d4146072fb056cbe32ce3320ca6600e2885/pom.xml
Retrieving lib-b/lib-b/master/lib-b-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-b/lib-b/8c6e91809ea51ba8fcfdb43cc12a8a472668b8fc/pom.xml
Retrieving lib-c/lib-c/master/lib-c-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-c/lib-c/56eadc7cc0e24b883a294a6a52a9ecea908606d6/pom.xml
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.jar from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
Retrieving lib-a/lib-a/master/lib-a-master.jar from gitlab
Retrieving lib-b/lib-b/master/lib-b-master.jar from gitlab
Retrieving lib-c/lib-c/master/lib-c-master.jar from gitlabRetrieving reifyhealth/lein-git-down/0.4.0/lein-git-down-0.4.0.pom from clojars
Retrieving org/clojure/tools.gitlibs/1.0.100/tools.gitlibs-1.0.100.pom from central
Retrieving org/clojure/pom.contrib/0.3.0/pom.contrib-0.3.0.pom from central
Retrieving org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.pom from central
Retrieving org/eclipse/jgit/org.eclipse.jgit-parent/4.10.0.201712302008-r/org.eclipse.jgit-parent-4.10.0.201712302008-r.pom from central
Retrieving com/jcraft/jsch/0.1.54/jsch-0.1.54.pom from central
Retrieving org/sonatype/oss/oss-parent/6/oss-parent-6.pom from central
Retrieving com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.pom from central
Retrieving org/sonatype/oss/oss-parent/5/oss-parent-5.pom from central
Retrieving com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy/0.0.9/jsch.agentproxy-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.pom from central
Retrieving net/java/dev/jna/jna/4.1.0/jna-4.1.0.pom from central
Retrieving net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.pom from central
Retrieving com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.pom from central
Retrieving com/jcraft/jsch/0.1.49/jsch-0.1.49.pom from central
Retrieving com/jcraft/jzlib/1.1.3/jzlib-1.1.3.pom from central
Retrieving lein-pprint/lein-pprint/1.3.2/lein-pprint-1.3.2.pom from clojars
Retrieving com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar from central
Retrieving org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.jar from central
Retrieving org/clojure/tools.gitlibs/1.0.100/tools.gitlibs-1.0.100.jar from central
Retrieving com/jcraft/jsch/0.1.54/jsch-0.1.54.jar from central
Retrieving com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.jar from central
Retrieving net/java/dev/jna/jna/4.1.0/jna-4.1.0.jar from central
Retrieving net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.jar from central
Retrieving com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.jar from central
Retrieving com/jcraft/jzlib/1.1.3/jzlib-1.1.3.jar from central
Retrieving reifyhealth/lein-git-down/0.4.0/lein-git-down-0.4.0.jar from clojars
Retrieving lein-pprint/lein-pprint/1.3.2/lein-pprint-1.3.2.jar from clojars
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
Retrieving org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
Retrieving lib-a/lib-a/master/lib-a-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-a/lib-a/b3140d4146072fb056cbe32ce3320ca6600e2885/pom.xml
Retrieving lib-b/lib-b/master/lib-b-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-b/lib-b/8c6e91809ea51ba8fcfdb43cc12a8a472668b8fc/pom.xml
Retrieving lib-c/lib-c/master/lib-c-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-c/lib-c/56eadc7cc0e24b883a294a6a52a9ecea908606d6/pom.xml
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.jar from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
Retrieving lib-a/lib-a/master/lib-a-master.jar from gitlab
Retrieving lib-b/lib-b/master/lib-b-master.jar from gitlab
Retrieving lib-c/lib-c/master/lib-c-master.jar from gitlab

and the process is stuck with this output (there is no progress made).

Thread dump attached.

I suspect that this is related to file locking because when I deleted the ~/.m2/repository when the process was stuck, suddently it unblocked and continued with the following output:

Could not find artifact lib-a:lib-a:jar:master in central (https://repo1.maven.org/maven2/)
Could not find artifact lib-a:lib-a:jar:master in clojars (https://repo.clojars.org/)
Could not transfer artifact lib-a:lib-a:jar:master from/to gitlab (git://gitlab.com): /Users/piotrek/.m2/repository/lib-a/lib-a/master/lib-a-master.jar.part.lock (No such file or directory)
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
Retrieving org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.jar from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
Created /Users/piotrek/.gitlibs/libs/lib-a/lib-a/b3140d4146072fb056cbe32ce3320ca6600e2885/target/lib-a-0.0.0-SNAPSHOT.jar
Retrieving lib-a/lib-a/master/lib-a-master.pom from gitlab
Retrieving lib-b/lib-b/master/lib-b-master.pom from gitlab
Retrieving lib-a/lib-a/master/lib-a-master.jar from gitlab
Retrieving lib-b/lib-b/master/lib-b-master.jar from gitlab

and it got blocked again.

[manderson202]

Thanks for the detailed issue description and reproduction. This issue occurs, as you show, when you have transitive dependencies also using lein-git-down. The only current workaround is to download the transitive dependencies first and have them cached, which is, of course, less than ideal. Will use this issue for status tracking.

[pbzdyl]

Is there a chance this issue will be fixed? (e.g. is there a potential fix that could resolve this)

[manderson202]

We know the cause of the issue and we should be able to fix it, however, the fix may be a bit involved. Will need to do some more investigation to determine the full scope.

[pbzdyl]

Great - thank you for the information. Please, let me know if I can help (for example with testing).