normalize deps Makefile targets and add uninstall

[vtjnash]

using a consistent mapping between target names reduces the coupling between targets
and should make it easier to add features like reinstall, uninstall, etc.

and it fixes #17671 and #17282

while the diff is large, this essentially just a mechanical change.

all targets (except get) now use secondary file targets, giving a simple mapping from the PHONY targets, and allows another target to reference the manifest file directly rather than using the OBJ_TARGET variable:
extract: srccache/LIB/source-extracted
configure: bulid/LIB/build-configured
compile: build/LIB/build-compiled
check: build/LIB/build-checked
install: usr/manifest/LIB

this allows more accurate tracking of when a build stage has completed, and a predictable way for the user to force a build stage to restart (by removing or touching the corresponding file)

(also killed of the atlas target, since it hasn't been maintained in several years)

[tkelman]

extract and extract-* should probably be here?

[tkelman]

You know what would do all of this boilerplate for us? ExternalProject_Add.

[vtjnash]

should still delete the shared library

it's not really that useful to delete one random file from the install, that's more a vestige of the old Makefile. in part 2 of this PR series, I'll update my PR that does a full uninstall (like autotools make uninstall, but functional even for handwritten makefiles and cmake)

[tkelman]

very misleading to have clean-foo not actually clean anything from the installed copy of foo. otherwise you're leaving in place the piece that julia needs, so after clean-foo you'll have an old copy of the shared library sitting around and you won't notice that fact in julia tests

[tkelman]

how about get a handwritten start on uninstall with at least the existing removals of the shared libraries

[vtjnash]

Anyone opposed to merging this right now, while the queues are quiet?

[tkelman]

This is going to make everyone rebuild everything, the bugs it fixes are relatively minor.

I've also yet to see it pass on CI.

[vtjnash]

fwiw, I expect the x86_64 CI linalg/arnoldi here will fail, since I corrupted the arpack cache on my previous push. I can reset the cache, but then the PR build will take longer, and the cache shouldn't transfer to master after merging, does it?

@tkelman it fixes my out-of-tree build from rebuilding several things quite frequently, letting me use out-of-tree builds more heavily without the current conflicts.

[tkelman]

Has this PR ever successfully finished a 32 bit build within the time limit?

[tkelman]

are you positive these won't keep recaching?

[vtjnash]

they shouldn't, since we should no longer be touching & re-running the rules that were causing this (install should be getting handled by usr-staging.tgz now and completely ignoring deps/scratch)

and confirmed by CI: https://travis-ci.org/JuliaLang/julia/jobs/157787509#L1631

[vtjnash]

The 32-bit build wouldn't finish on master either if it was held to the same time limit (https://travis-ci.org/JuliaLang/julia/jobs/153542507)

[tkelman]

The from-scratch build has gone through in under 2 hours before to repopulate a clean cache, and if it doesn't on this branch then it may be making the build slower by adding more steps. Need to find a way to fix that otherwise every build will fail.

[vtjnash]

ok, I don't know how to search for the last full cache build

[vtjnash]

anyways, full deps build for x86 have been taking ~65min on this PR. with typical travis variance, that gives about a 20% chance of failure given that the cache build times range from 45-80 minutes. and only need one build to pass to populate the cache for a long time after.

this also implements support for doing our own cache. E.g. Our buildbots can upload their usr-staging folders and travis can use those instead, eliminating that issue and the difficulties we've had with resetting corrupted caches. Plus we can enable that for everyone, so that everyone downloading the julia src repo can get lightning fast builds.

[tkelman]

You're neglecting the fact that compilers and glibc versions need to match to make that actually work.

[tkelman]

Did you test this in clean compiles on all platforms?

/usr/bin/tar: ./bin: Cannot open: File exists
/usr/bin/tar: Exiting with failure status due to previous errors
make[1]: *** [/home/Tony/julia32/deps/dsfmt.mk:44: /home/Tony/julia32/usr/manifest/dsfmt] Error 2

[tkelman]

make distcleanall reconfigures and recompiles libuv, openlibm, openspecfun, llvm and probably others. This really wasn't ready to be merged, which is why when you asked "Anyone opposed to merging this right now" I responded with objections.

[tkelman]

should this be a tab, just for visual consistency?

[vchuravy]

I am compiling with llmv 3.9 and it seems that the files are not getting install in the right place:

tar tvf /home/wallnuss/src/julia/usr-staging/llvm-3.9.0/build_Release+Asserts.tgz
drwxr-xr-x wallnuss/wallnuss 0 2016-09-07 01:22 ./

I am still looking into what causes the issue, but it seems the build is ignoring the ($build_prefix) in https://github.com/JuliaLang/julia/blob/master/deps/tools/common.mk#L146

[tkelman]

why is this deleting the libuv manifest?

[vtjnash]

thanks @vchuravy. fixed by b3541bc#diff-c0f89475cb4381b2b7a2e7228fe5e39cR518

[vchuravy]

@vtjnash thanks! I got to the same fix and was just waiting for a build finishing before submitting it. This solves my problem :)