The Documenter script doc/make.jl needs Internet connection, which breaks build at server farms #19966

ronisbr · 2017-01-10T15:46:37Z

Hi guys!

I am the maintainer of julia in openSUSE. Besides the stable version, I keep building master so that I can track bugs. I do these steps to obtain the code:

Clone git repository.
make light-source-dist.

All the other dependencies are either already installed in openSUSE or downloaded and added to srccache by the RPM script. However, it turns out that since December, 2016 the building is failing with the following error:

[  554s] /home/abuild/rpmbuild/BUILD/julia/usr/bin/julia make.jl -- deploy
[  555s] INFO: Initializing package repository /home/abuild/rpmbuild/BUILD/julia/doc/deps/v0.6
[  555s] INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
[  555s] ERROR: LoadError: GitError(Code:ERROR, Class:Net, curl error: Could not resolve host: github.com
[  555s] )
[  555s] Stacktrace:
[  555s]  [1] macro expansion at ./libgit2/error.jl:99 [inlined]
[  555s]  [2] clone(::String, ::String, ::Base.LibGit2.CloneOptions) at ./libgit2/repository.jl:190
[  555s]  [3] #clone#106(::String, ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractCredentials}, ::Function, ::String, ::String) at ./libgit2/libgit2.jl:327
[  555s]  [4] (::Base.LibGit2.#kw##clone)(::Array{Any,1}, ::Base.LibGit2.#clone, ::String, ::String) at ./<missing>:0
[  555s]  [5] (::Base.Pkg.Dir.##4#6{String,String})() at ./pkg/dir.jl:49
[  555s]  [6] cd(::Base.Pkg.Dir.##4#6{String,String}, ::String) at ./file.jl:69
[  555s]  [7] init(::String, ::String) at ./pkg/dir.jl:47
[  555s]  [8] init() at ./pkg/pkg.jl:70
[  555s]  [9] include_from_node1(::String) at ./loading.jl:532
[  555s]  [10] include(::String) at ./sysimg.jl:14
[  555s]  [11] process_options(::Base.JLOptions) at ./client.jl:308
[  555s]  [12] _start() at ./client.jl:374
[  555s] while loading /home/abuild/rpmbuild/BUILD/julia/doc/make.jl, in expression starting on line 3
[  555s] make[2]: *** [Makefile:34: html] Error 1
[  555s] make[2]: Leaving directory '/home/abuild/rpmbuild/BUILD/julia/doc'
[  555s] make[1]: *** [Makefile:115: docs] Error 2
[  555s] make[1]: Leaving directory '/home/abuild/rpmbuild/BUILD/julia'
[  555s] make: *** [Makefile:59: /home/abuild/rpmbuild/BUILD/julia/doc/_build/html/en/index.html] Error 2
[  555s] error: Bad exit status from /var/tmp/rpm-tmp.1IrPYW (%install)

Notice that the documentation is already built in doc/_build/html because of light-source-dist. So, I verified that the script doc/make.jl is being called and the first thing it does it Pkg.init(), which requires internet connection, breaking the build at server farms.

Is there a way to circumvent this problem besides deleting and touching make.jl?

The text was updated successfully, but these errors were encountered:

ronisbr · 2017-01-10T16:17:30Z

If anyone wants to build using light-source-dist (and probably full-source-dist) on a server farm now, then do:

Download http://www.unicode.org/Public/9.0.0/ucd/UnicodeData.txt and copy to doc/.
rm doc/make.jl and touch doc/make.jl.

nalimilan · 2017-01-10T16:38:32Z

See #18588 (comment) and #19725. My priority was to be able to build the tarball without Internet access. Do you really need to also generate that tarball without Internet access? I guess we could track UnicodeData.txt in git, it's always sounded a kind of weird to me that we would need this to build docs and yet get it from elsewhere. But it's a bit large, maybe we should compress it first.

nalimilan · 2017-01-10T16:43:10Z

Ah, I think I misread your explanation. The tarball generated by make light-source-dist already builds fine without Internet since #19539 and #19705. Are you sure you're using the latest git master?

ronisbr · 2017-01-10T16:53:44Z

Hi @nalimilan,

I don't need to build the tarball without internet. It is fine if make light-source-dist download things and put in correct directories.

Yes, I cloned git repository in the morning (commit 6b1b4f6). Let me show the build options I am using:

NO_GIT=1 \
USE_SYSTEM_LLVM=0 \
USE_SYSTEM_LIBUNWIND=1 \
USE_SYSTEM_READLINE=1 \
USE_SYSTEM_PCRE=1 \
USE_SYSTEM_LIBM=0 \
USE_SYSTEM_OPENLIBM=1 \
USE_SYSTEM_OPENSPECFUN=1 \
USE_SYSTEM_DSFMT=1 \
USE_SYSTEM_BLAS=1 \
LIBBLASNAME=libblas.so.3 \
USE_SYSTEM_LAPACK=1 \
LIBBLASNAME=liblapack.so.3 \
USE_SYSTEM_FFTW=1 \
LIBFFTWNAME=libfftw3_threads.so.3 \
LIBFFTWFNAME=libfftw3f_threads.so.3 \
USE_SYSTEM_GMP=1 \
USE_SYSTEM_MPFR=1 \
USE_SYSTEM_ARPACK=1 \
USE_SYSTEM_SUITESPARSE=1 \
USE_SYSTEM_ZLIB=1 \
USE_SYSTEM_GRISU=1 \
USE_SYSTEM_RMATH=0 \
USE_SYSTEM_LIBUV=0 \
USE_SYSTEM_UTF8PROC=0 \
USE_SYSTEM_PATCHELF=1 \
USE_SYSTEM_LIBGIT2=1 \
USE_MKL=0 \
USE_BLAS64=0 \
USE_LLVM_SHLIB=0 \
VERBOSE=1 \
USECLANG=0

The problem I mentioned was at make install. I saw your commit, and I have this at Makefile:

# Build the HTML docs (skipped if already exists, notably in tarballs)
$(BUILDROOT)/doc/_build/html/en/index.html: $(shell find $(BUILDROOT)/base $(BUILDROOT)/doc \( -path $(BUILDROOT)/doc/_build -o -path $(BUILDROOT)/doc/deps -o -name *_constants.jl -o -name *_h.jl \) -prune -o -type f -print)
	@$(MAKE) docs

I can't understand what this is doing, but I have the doc/_build directory. However I do not have the doc/deps directory.

ronisbr · 2017-01-10T16:59:06Z

Here is the script that builds julia:

make %{?_smp_mflags} MARCH=%{julia_march} \
                     prefix=%{_prefix} \
                     bindir=%{_bindir} \
                     libdir=%{_libdir} \
                     libexecdir=%{_libexecdir} \
                     datarootdir=%{_datarootdir} \
                     includedir=%{_includedir} \
                     sysconfdir=%{_sysconfdir} \
                     %{juliabuildopts} \
                     release
make %{?_smp_mflags} MARCH=%{julia_march} \
                     prefix=%{_prefix} \
                     bindir=%{_bindir} \
                     libdir=%{_libdir} \
                     libexecdir=%{_libexecdir} \
                     datarootdir=%{_datarootdir} \
                     includedir=%{_includedir} \
                     sysconfdir=%{_sysconfdir} \
                     %{juliabuildopts} \
                     debug
%install
make install DESTDIR=%{buildroot} \
             MARCH=%{julia_march} \
             prefix=%{_prefix} \
             bindir=%{_bindir} \
             libdir=%{_libdir} \
             libexecdir=%{_libexecdir} \
             datarootdir=%{_datarootdir} \
             includedir=%{_includedir} \
             sysconfdir=%{_sysconfdir} \
             %{juliabuildopts}

nalimilan · 2017-01-10T16:59:38Z

I don't have doc/deps in my tarballs either. The only special thing I am doing before calling make light-source-dist is calling make -C doc. Does that make a difference for you?

Does it work it you use the tarball included in this SRPM?

nalimilan · 2017-01-10T17:01:25Z

Also, what's the %prep step in the .spec file? Maybe the timestamps are set to the time when the tarball is unpacked, which could be a problem for make. Can you check them?

ronisbr · 2017-01-10T17:04:17Z

If I execute make -C doc I get:

make -C doc                                                                                                                                               15:00:49  ☁  master ☂ ✭
make: Entering directory '/media/data/packaging/julia/doc'
/media/data/packaging/julia/deps/tools/jlchecksum UnicodeData.txt
Building HTML documentation.
/media/data/packaging/julia/usr/bin/julia make.jl -- deploy
make: /media/data/packaging/julia/usr/bin/julia: Command not found
make: *** [Makefile:34: html] Error 127
make: Leaving directory '/media/data/packaging/julia/doc'

My %prep section is:

%prep
%setup -q -n julia

# remove .gitignore
find . -name ".git*" -exec rm {} \;

pushd deps/
mkdir -p srccache/
pushd srccache/
cp %{SOURCE10} ./
cp %{SOURCE11} ./
cp %{SOURCE12} ./
cp %{SOURCE13} ./
popd
popd

pushd doc/
cp %{SOURCE14} ./
popd

# Remove julia script that builds the documentation.
# See: https://github.com/JuliaLang/julia/issues/19966
rm doc/make.jl
touch doc/make.jl

Those sources are dependencies that I want to build with julia instead of use from openSUSE repos.

ronisbr · 2017-01-10T17:05:39Z

Let me try to delete my julia tree and clone again.

ronisbr · 2017-01-10T17:08:19Z

Yeah, seems that I am doing something very wrong :D Sorry...

The problem is that the _build directory was created at another version. Now, I am having the aforementioned error because the script is looking for julia at a strange directory. Let me see if I can find why.

nalimilan · 2017-01-10T17:09:19Z

And what happens if you don't touch rm doc/make.jl? Ironically, it's one of the dependencies of the docs, so touching it will trigger a rebuild.

make -C doc needs the JULIA_EXECUTABLE variable to point to the resulting julia binary. That's a bit annoying since it means you need to call make just to call make light-source-dist.

ronisbr · 2017-01-10T17:13:16Z

Ah, it only works if I touch doc/make.jl. That was the only way I found to avoid accessing internet on the server farm. I will do what you told me (build julia to build the tarball) and see if I can use the generated tarball on the server farm.

ronisbr · 2017-01-10T17:56:40Z

@nalimilan sorry for the noise! It turns out that the problem was caused because my directory with julia tree already had an old version of the documentation. Hence, for some reason, it was not being built with make light-source-dist. After cloning a new tree and following your steps (building julia, build the documentation, and then the tarball), then everything worked fine.

Thanks!

nalimilan · 2017-01-10T18:09:06Z

No worries, good to know it works for you too.

tkelman · 2017-01-10T18:10:42Z

Hm, sounds like the makefile rules didn't correctly rebuild the stale docs like they should have then?

ronisbr · 2017-01-10T18:12:38Z

@tkelman no, they didn't. I should have saved the old tree to debug that :( sorry...

ronisbr closed this as completed Jan 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The Documenter script doc/make.jl needs Internet connection, which breaks build at server farms #19966

The Documenter script doc/make.jl needs Internet connection, which breaks build at server farms #19966

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

nalimilan commented Jan 10, 2017

nalimilan commented Jan 10, 2017

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

nalimilan commented Jan 10, 2017

nalimilan commented Jan 10, 2017

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

nalimilan commented Jan 10, 2017

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

nalimilan commented Jan 10, 2017

tkelman commented Jan 10, 2017

ronisbr commented Jan 10, 2017

The Documenter script doc/make.jl needs Internet connection, which breaks build at server farms #19966

The Documenter script doc/make.jl needs Internet connection, which breaks build at server farms #19966

Comments

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

nalimilan commented Jan 10, 2017

nalimilan commented Jan 10, 2017

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

nalimilan commented Jan 10, 2017

nalimilan commented Jan 10, 2017

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

nalimilan commented Jan 10, 2017

ronisbr commented Jan 10, 2017

ronisbr commented Jan 10, 2017

nalimilan commented Jan 10, 2017

tkelman commented Jan 10, 2017

ronisbr commented Jan 10, 2017