Idiosyncratic failure building Julia on macOS

[mnemnion]

I don't know if this actually belongs here, because I can't imagine everyone with my system is having this problem, but: for a couple weeks I've been trying to build Julia on macOS 12.6.5, off the master branch, using gmake 4.4.1, and the build consistently fails in the same way.

I asked for help on the Zulip, which folks there were quite willing to provide, but we didn't get it working and I'm stumped. So on the off chance that this is either a known problem or in fact a bug, I'm filing this report.

The step which fails in gmake VERBOSE=1 is the following:

mv /path/to/julia/usr-staging/libwhich-99a0ea12689e41164456dba03e93bc40924de880.tar.tmp /path/to/julia/usr-staging/libwhich-99a0ea12689e41164456dba03e93bc40924de880.tar
[ ! -e /path/to/julia/usr/manifest/libwhich ] || gmake uninstall-libwhich
/usr/bin/tar -xmUf /path/to/julia/usr-staging/libwhich-99a0ea12689e41164456dba03e93bc40924de880.tar -C /path/to/julia/usr
echo 'libwhich-99a0ea12689e41164456dba03e93bc40924de880 staged-uninstaller' > /path/to/julia/usr/manifest/libwhich
gmake[1]: Leaving directory '/path/to/julia/deps'
gmake[1]: Entering directory '/path/to/julia/stdlib'
Makefile:53: *** empty variable name.  Stop.
gmake[1]: Leaving directory '/path/to/julia/stdlib'
gmake: *** [Makefile:79: julia-stdlib] Error 2

I've consulted the platform specific build notes, and [DY]LD_LIBRARY_PATH aren't set.

It isn't a bad-state issue in my copy of the repo, which I've download fresh every day I've tried. The problem appears to be internal to the build process, so not a missing tool or the like. It seems rather unlikely that fresh builds from a clone are broken on all macOS systems, or even all macOS 12.x systems, but still: it's not building.

Any help would be greatly appreciated, I know the bug tracker isn't for tech support but the odds that this is a bug seem high enough to file the issue and hope for the best.

[mkitti]

I was able to build master today using the following.

% make -j8

Base  ────────── 23.872582 seconds
FileWatching  ──  2.695978 seconds
Libdl  ─────────  0.002505 seconds
Artifacts  ─────  0.185451 seconds
SHA  ───────────  0.123926 seconds
Sockets  ───────  0.174381 seconds
LinearAlgebra  ─  4.337545 seconds
Random  ────────  0.505880 seconds
Stdlibs total  ─  8.029323 seconds
Sysimage built. Summary:
Base ────────  23.872582 seconds 74.829%
Stdlibs ─────   8.029323 seconds 25.168%
Total ───────  31.902868 seconds
    JULIA usr/lib/julia/sys-o.a
Collecting and executing precompile statements
└ Collect (Basic: ✓ 843) => Execute ✓ 613
Outputting sysimage file...
Output ──────  22.123140 seconds
    LINK usr/lib/julia/sys.dylib
...
    JULIA stdlib/REPLExt.release.image
 │ Attempted to find missing packages in package registries but no registries are installed.
 └ Use package mode to install a registry. `pkg> registry add` will install the default registries.

% sw_vers
ProductName:            macOS
ProductVersion:         13.6.3
BuildVersion:           22G436

% which make
/usr/bin/make

% make -v
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

[mbauman]

gmake 4.4.1

Why gmake? Have you tried using the macOS' default /usr/bin/make?

[mnemnion]

I've tried it both ways, I figured a newer version was better, and that the error messages were somewhat more informative.

Here's what I get attempting to build 71f68b4ce9189e64f320631f3f74ffb3dd10e875 with the building macOS make VERBOSE=1:

/usr/bin/tar -xmUf /path/to/julia/usr-staging/libwhich-99a0ea12689e41164456dba03e93bc40924de880.tar -C /path/to/julia/usr
echo 'libwhich-99a0ea12689e41164456dba03e93bc40924de880 staged-uninstaller' > /path/to/julia/usr/manifest/libwhich
/path/to/julia/deps/tools/jldownload /path/to/julia/stdlib/srccache/Pkg-.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   285  100   285    0     0   3659      0 --:--:-- --:--:-- --:--:--  3701
100   361  100   361    0     0   3501      0 --:--:-- --:--:-- --:--:--  3501
100   513  100   513    0     0   4000      0 --:--:-- --:--:-- --:--:--  4000
100   817  100   817    0     0   5300      0 --:--:-- --:--:-- --:--:--  5300
100  1425  100  1425    0     0   7940      0 --:--:-- --:--:-- --:--:--  7940
100  2641  100  2641    0     0  12842      0 --:--:-- --:--:-- --:--:-- 12842
100  5073  100  5073    0     0  21785      0 --:--:-- --:--:-- --:--:-- 21785
  0   157    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 502
curl: Failed to extract a sensible file name from the URL to use for storage
curl: (3) URL using bad/illegal format or missing URL
make[1]: *** [/path/to/julia/stdlib/srccache/Pkg-.tar.gz] Error 3
make: *** [julia-stdlib] Error 2

So same failure point, with the details of the error different.

@mkitti, is that a build from a fresh clone? It doesn't look like it's doing the initial downloads, which is where this fails for me. I also notice you're on macOS 13.x, I'm still on 12.x. I'm not quite at the upgrade-the-os stage to get this to build, but I'm getting there.

[mkitti]

The problem seems to be with the Pkg library download and a missing hash / version.

/path/to/julia/deps/tools/jldownload /path/to/julia/stdlib/srccache/Pkg-.tar.gz

I believe this was a fresh clone since I usually do not develop Julia on macOS, but I can check.

[mnemnion]

The problem seems to be with the Pkg library download and a missing hash / version.

That was more-or-less what we concluded on the Zulip, and it looked enough like a bug in the build process to be worth filing an issue over. Could still be some weirdness in my system, but I can't figure out how that might be.

[mkitti]

xref: https://julialang.zulipchat.com/#narrow/stream/274208-helpdesk-.28published.29/topic/Building.20Julia.20on.20macOS.2012.2E6.2E3.20fails/near/421291189

[mkitti]

Here is what the relevant section should look like via make VERBOSE=1.

echo 'utf8proc-34db3f7954e9298e89f42641ac78e0450f80a70d staged-uninstaller' > /Users/<user>/Documents/src/julia/usr/manifest/utf8proc
[ ! -e /Users/<user>/Documents/src/julia/usr/manifest/libwhich ] || /Library/Developer/CommandLineTools/usr/bin/make uninstall-libwhich
cd /Users/<user>/Documents/src/julia/usr && rm -fv -- $(/usr/bin/tar -tf /Users/<user>/Documents/src/julia/usr-staging/libwhich-99a0ea12689e41164456dba03e93bc40924de880.tar | grep -v '/$')
./tools/libwhich
rm -f /Users/<user>/Documents/src/julia/usr/manifest/libwhich
/usr/bin/tar -xmUf /Users/<user>/Documents/src/julia/usr-staging/libwhich-99a0ea12689e41164456dba03e93bc40924de880.tar -C /Users/<user>/Documents/src/julia/usr
echo 'libwhich-99a0ea12689e41164456dba03e93bc40924de880 staged-uninstaller' > /Users/<user>/Documents/src/julia/usr/manifest/libwhich
/Users/<user>/Documents/src/julia/deps/tools/jldownload /Users/<user>/Documents/src/julia/stdlib/srccache/Pkg-1f16df404a2fbe8642ea3eecc9f4d7064c400a73.tar.gz https://api.github.com/repos/JuliaLang/Pkg.jl/tarball/1f16df404a2fbe8642ea3eecc9f4d7064c400a73
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   477  100   477    0     0   1131      0 --:--:-- --:--:-- --:--:--  1141
100  518k  100  518k    0     0   950k      0 --:--:-- --:--:-- --:--:--  950k

[mkitti]

What is the output of cat stdlib/Pkg.version?

Mine is now as follows.

% cat stdlib/Pkg.version 
PKG_BRANCH = master
PKG_SHA1 = 1f16df404a2fbe8642ea3eecc9f4d7064c400a73
PKG_GIT_URL := https://github.com/JuliaLang/Pkg.jl.git
PKG_TAR_URL = https://api.github.com/repos/JuliaLang/Pkg.jl/tarball/$1

That's consistent with https://github.com/JuliaLang/julia/blob/master/stdlib/Pkg.version

[mkitti]

Here's a traceback of the relevant Makefile lines:

julia/deps/tools/git-external.mk

Line 63 in 77c0672

$$(JLDOWNLOAD) $$@ $$(call $2_TAR_URL,$$($2_SHA1))

julia/deps/tools/stdlib-external.mk

Line 15 in 77c0672

$$(eval $$(call git-external,$1,$2,,,$$(BUILDDIR)))

julia/stdlib/Makefile

Line 53 in 77c0672

$(foreach module, $(STDLIBS_EXT), $(eval $(call stdlib-external,$(module),$(shell echo $(module) | tr a-z A-Z))))

[mkitti]

% make -C stdlib print-PKG_TAR_URL
PKG_TAR_URL=https://api.github.com/repos/JuliaLang/Pkg.jl/tarball/
% make -C stdlib print-PKG_SHA1   
PKG_SHA1=1f16df404a2fbe8642ea3eecc9f4d7064c400a73
% make -C stdlib print-PKG_SRC_DIR
PKG_SRC_DIR=Pkg-1f16df404a2fbe8642ea3eecc9f4d7064c400a73

[DilumAluthge]

What shell do you use, and what version of that shell?

(I doubt it's relevant, but I figured I'd ask, for completeness.)

[DilumAluthge]

Another stray thought: do you have Homebrew installed on your machine?

[DilumAluthge]

Also, can you give us the output of which -a $tool and $tool --version for each of the following values of $tool:

1. curl
2. wget
3. fetch
4. git

[mnemnion]

• fish 3.6.4 (I've tried it under zsh 5.8.1 as well)
• curl 8.4.0 (/usr/bin)
• wget: GNU Wget 1.21.4 built on darwin21.6.0. (homebrew)
• fetch... originally, no fetch! (weird). Alas, installing it doesn't fix the issue, but now it's fetch v0.4.6 in the homebrew directory
• git 2.43.0 (homebrew)

>  cat stdlib/Pkg.version                                                                                                                    (base)
PKG_BRANCH = master
PKG_SHA1 = 1f16df404a2fbe8642ea3eecc9f4d7064c400a73
PKG_GIT_URL := https://github.com/JuliaLang/Pkg.jl.git
PKG_TAR_URL = https://api.github.com/repos/JuliaLang/Pkg.jl/tarball/$1

➜  make -C stdlib print-PKG_TAR_URL        
PKG_TAR_URL=https://api.github.com/repos/JuliaLang/Pkg.jl/tarball/
➜  make -C stdlib print-PKG_SHA1                   
PKG_SHA1=1f16df404a2fbe8642ea3eecc9f4d7064c400a73
➜  make -C stdlib print-PKG_SRC_DIR         
PKG_SRC_DIR=

That last line seems significant!

[DilumAluthge]

What do which -a make and make --version give?

[DilumAluthge]

Also, what's the output of make -C stdlib print-PKG_SRC_FILE?

[DilumAluthge]

Also:

1. Do you have a Make.user file? (For most people, the answer will be no.)
2. Is the environment variable DEPS_GIT defined in your environment? (For most people, the answer will be no.)

[mnemnion]

PKG_SRC_DIR is also blank:

➜  make -C stdlib print-PKG_SRC_DIR   
PKG_SRC_DIR=

make is the standard /usr/bin version 3.81. I've also tried this with homebrew gmake, version 4.4.1. Only Make.user.* files are in the Julia distribution, DEPS_GIT is not defined.

[mnemnion]

I think that was a bad copypaste on my part, PKG_SRC_FILE is also blank:

✗  make -C stdlib print-PKG_SRC_FILE                                                                                                         
PKG_SRC_FILE=

[mkitti]

Could it be a shell issue? I usually use bash everywhere.

[mnemnion]

I've tried it with fish and zsh, the latter being the default shell on macOS. Just tried bash because why not, but same error.

[mnemnion]

As of commit 2e876fc7f665cd660, the build process continues to error out in the same place. In case it might be useful, I made a gist of the full build output, with the path scrubbed.

Am I right in thinking no one has been able to reproduce this problem?

[mbauman]

It isn't a bad-state issue in my copy of the repo, which I've download fresh every day I've tried.

Are you downloading GitHub's bundled-up .zip? The build process does expect to have state from a git clone... but I can successfully build from the zip, too.

I've never experienced or seen a build failure like this, no. What ENV vars do you have set? Can you try unsetting everything?

[mnemnion]

Are you downloading GitHub's bundled-up .zip?

No, I'll try that next, thanks. I was following the README and using git clone, and in fact didn't realize there were release .zips. But since it's the bundle download process that's failing, this is certainly worth trying.

[mkitti]

git clone is how I usually do it.

[mkitti]

Could you try hardcoding PKG_SRC_FILE somewhere below line 60 in git-external.mk:

julia/deps/tools/git-external.mk

Line 60 in 77c0672

$2_SRC_DIR := $1-$$($2_SHA1)

$2_SRC_DIR := $1-$$($2_SHA1)
PKG_SRC_DIR := Pkg-6859d68579e7970daf00720901cf487563dca0da
$2_SRC_FILE := $$(SRCCACHE)/$$($2_SRC_DIR).tar.gz
$$($2_SRC_FILE): | $$(SRCCACHE)
	$$(JLDOWNLOAD) $$@ $$(call $2_TAR_URL,$$($2_SHA1))

6859d68579e7970daf00720901cf487563dca0da here should be match the output of make -C stdlib print-PKG_SHA1

[mnemnion]

So, in trying to get julia to build with env -i PATH="$PATH" make, the command threw a brand new error showing that something in the build process was trying to call a binary with the same name as one of my custom tools. I haven't figured out which one, or why, or indeed, why stripping ENV down to PATH only made the error show up, but moving the directory containing all the commands which could be triggering it got the build to complete.

So thanks everyone for your help! I feel bad using the issue tracker here but I had long-since run out of theories for how my environment could be causing it. I'll close the issue now, and might attach a comment or two with a more specific postmortem of the build failure, for posterity, and just in case it proves useful for someone, somewhere. You never know.