Julia won't pull from https behind a corporate proxy

[mogthesprog]

Hi,

I've been experiencing some dificulty using Pkg.init() and Pkg.add() behind a corporate proxy. See below...

[31510:31508 - 0:1029] 01:16:24 [root@gb-slo-svb-0139:o +1] /opt $ julia
Version 0.5.0-dev
x86_64-redhat-linux
julia> Pkg.init()
INFO: Initializing package repository /opt/julia/lib/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ERROR, Class:OS, Failed to connect to github.com: Connection refused)
 [inlined code] from libgit2/error.jl:98
 in clone at libgit2/repository.jl:189
 in __clone#364__ at libgit2.jl:310
 in anonymous at pkg/dir.jl:49
 in cd at file.jl:47
 in init at pkg/dir.jl:47
 in init at pkg.jl:22
 in eval at ./boot.jl:263

julia>
[31510:31508 - 0:1030] 01:16:32 [root@gb-slo-svb-0139:o +1] /opt $ cd /tmp/
[31510:31508 - 0:1031] 01:16:38 [root@gb-slo-svb-0139:o +1] /tmp $ git clone https://github.com/JuliaLang/METADATA.jl
Cloning into 'METADATA.jl'...
remote: Counting objects: 52190, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 52190 (delta 3), reused 0 (delta 0), pack-reused 52178
Receiving objects: 100% (52190/52190), 6.68 MiB | 3.47 MiB/s, done.
Resolving deltas: 100% (13081/13081), done.
[31510:31508 - 0:1032] 01:16:52 [root@gb-slo-svb-0139:o +1] /tmp $

I've tried the git config --global https.proxy <comeproxy> and also git config --global url."git://".insteadOf https://" , i've tried configuring environment variables in the shell. I've even tried issuing Pkg.init() with the full url, replacing https with git or http. I'm a bit stuck at the moment. I can't see any reason why this shouldn't work behind our proxy.

Any ideas guys?

Cheers,

Morgan

[lobingera]

looking at the above i get the impression, you are using Pkg with libgit2, so there is a possiblity that your configuration of the command line git (calling git config) is not applied

[mogthesprog]

Thanks for the quick reply :)

Yeah I think so too... Any idea how I can change the libgit2 calls in julia? is there a Pkg method for changing the config?

[lobingera]

There seems to be some config in Libgit2 and it might be a idea to connect to this
#13879 where the topic of Libgit2 using the 'global' git config was touched.

[tkelman]

duplicate of #13472, see #13952 for a workaround. Or use release-0.4, you should only use master for submitting pull requests.

[axsk]

I am having a similar problem with master on my cluster at work:
ERROR: GitError(Code:ERROR, Class:OS, Failed to connect to github.com: No route to host)

On 0.4 the git config trick fixed the issue.
I would assume that this should not be necessary since #11312, so maybe the git:// vs https:// issue is not even the problem here?

Edit: I also tried @vtjnash's suggestion from #13952 , USE_SYSTEM_LIBGIT2=1 followed by a make clear && make but this did not help either.

[tkelman]

it's make clean not make clear, though you may need a make cleanall and/or make -C deps distclean-libgit2 to get the dependency change to propagate to the build result

[ViralBShah]

Should be fixed in #17783. Please try the latest nightlies tomorrow.

[gersonjferreira]

Is this fixed? Version 0.5.0 is now on the Ubuntu PPA and I got in trouble yesterday because I'm behind a proxy. So I had to revert to 0.4.5 version on the official Ubuntu repos and lock the version. I'll keep it lock until the PPA version gets the proxy fixed.

[ViralBShah]

Please do not use ubuntu PPAs and use the generic linux binaries instead.

[ViralBShah]

@staticfloat Do you think on the ubuntu PPA page, we can put a word about using the generic linux binaries for such cases?

Alternately, perhaps our custom PPAs could repackage all the necessary libraries and not use the system ones.

[gersonjferreira]

@ViralBShah , I don't understand this. You do announce the PPA on the Downloads section of the Julia webpage under Platform Specific Instructions. If the PPAs are not meant for the general user, this (and the reason) should be explicit somewhere. Or simply change the PPA name to Debian-like "testing" and "unstable", so that it is clear that only the generic binaries are the ones considered stable. In any case, I would like to have a stable PPA distribution available to help me keep it updated. Or you could try snap packaging... ?

[StefanKarpinski]

Unfortunately, issues with standard configurations of many libraries, combined with distro policies requiring the use of shared versions of libraries when they exist, make it essentially impossible to have a fully working Julia in most distros, including Debian and Ubuntu. We'll continue to work towards improving this situation, but don't expect it to be smooth in the near future.

[StefanKarpinski]

To get a sense of how many libraries need patches to work correctly, check out this directory of patches that we apply when building Julia's dependencies from source:

• https://github.com/JuliaLang/julia/tree/master/deps/patches

We do upstream these patches, but it takes a while for projects to review them, let alone for the fixes to make it into distros. Upstream projects also vary in how willing they are to take patches.

[tkelman]

Not just patches, but Julia also has pretty strict requirements on consistent versions and options in how we build things for it to all work correctly. Sorry we haven't removed the PPA from the platform specific downloads page, we should have done that a while ago and replaced it with "download and untar the generic binaries."

Distro specific builds or packaging are maintained by volunteer contributors on a best effort level, and anywhere their build configuration causes problems we have to recommend that people not use them. There are many distros other than ubuntu out there, and our generic binaries work on the majority of them. Auto updating is a double edged sword, as downgrading to an old build is not possible from a PPA.

[nalimilan]

downgrading to an old build is not possible from a PPA.

But it's possible from Copr using dnf downgrade. :-)

[tkelman]

Yeah, the level of comfort I have with recommending distro packages is directly proportional to how actively involved their maintainer is with Julia development. Your Fedora packages are usually in pretty good shape because you stay on top of things, like Elliot used to have much more time to do than he has recently.

There's a licensing issue with the system libgit2 in debian and openssl, see discussion at libgit2/libgit2#3462 (comment) and #17661. Frustratingly there's code that implements a solution that everyone other than the maintainers of libgit2 has said they would be happy with, but the libgit2 maintainers have been very quiet in response to the question of "would you reconsider a reopened version of this."

[staticfloat]

@gersonjferreira The PPA, like many PPAs, is a community-supported effort, which is a fancy way of saying that it's not an official release and does not go through the same testing as the generic linux binaries available from the julia download page, which is why @ViralBShah suggests using the generic Linux binaries. Yes, there are advantages/disadvantages to both, and I will certainly be working on debugging why exactly the PPA version doesn't seem to honor proxy settings properly, but for the short-term, the generic linux binaries are the best solution.

Returning to the matter at hand, I'm having difficulty reproducing the proxy issue on my local machine, mostly because I'm not certain I've ever actually used a proxy with git. The main issue for this is here, feel free to follow this thread to see when it will be fixed: staticfloat/julia-debian#3

[gersonjferreira]

ok, I understand the difficulties. But them you should warn people about it. Once the proxy had fail here for me, I had to hurry and tell my students (all new to Linux) to lock the version on the apt config.

What about snap packages for the generic binary files? Could be a much better way to distribute them. But this is out of topic. Regarding the PPA and proxy issue, I thank you all for the attention. I'll go back to the generic binaries for now. Cheers.

[tkelman]

how many distros support snap packages? are they installable on centos 6 without having root access? if not, we're likely going to stick with the tarball approach. maybe switch to a better compression than gzip at some point.

[tkelman]

Reading up on it, looks like it would be pretty trivial to wrap the existing generic Linux binaries into a snap package. Unless there's a lot of demand for this format I don't think we're about to start building and distributing these ourselves, but it would be an easy contribution for someone in the community to do - as long as there's some mechanism of verifying the included contents have the same checksum as our official posted binaries, anyway.

[ViralBShah]

I have updated the platform specific instructions page about the pitfalls of using the distro packages.

JuliaLang/www_old.julialang.org@ab16b68

I think the right thing for the distro packages that are provided via ppa and copr is to bundle everything the generic binaries do.

[ViralBShah]

snap looks nice!

[nalimilan]

I think the right thing for the distro packages that are provided via ppa and copr is to bundle everything the generic binaries do.

@ViralBShah I disagree. Things are working pretty well in Fedora/RHEL7 without bundling libraries other than libuv and libunwind. I don't see the point of bundling everything when there are no serious bugs in distro packages. Else there's little point in providing packages in the first place.

[ViralBShah]

What about LLVM patches?