Make bootstrap use _checkouts

[zuiderkwast]

This makes it possible to run the bootstrap offline by placing the deps
in the already documented _checkouts directory in advance.

[bjosv]

LGTM

[zuiderkwast]

@ferd In the EEF meeting you agreed to merge this. Any thoughts?

[ferd]

I was supposed to test various vendoring strategies but went to vacation during the holidays without having had time to try that first.

[ferd]

Ran a quick Demo with John.

The short version of it: Rebar3 can seemingly support the type of vendoring the Ericsson team requires in the following way:

1. Declare an umbrella project (with code in apps/<libname>)
2. Declare all external dependencies in each of the top-level apps' rebar.config file (e.g. apps/myapp/rebar.config contains {deps, [elli, recon]}) but no dependencies are declared in the root's rebar.config file (these definitions get inherited by all sub-applications, and when we do the vendor step, it creates a cycle)
3. call rebar3 get-deps to fetch but not build dependencies. These should now be in _build/default/lib
4. mkdir lib at the root, and then mv _build/default/lib/<app-to-vendor> lib/
5. to upgrade libraries, remove it from lib/ and use rebar3 upgrade <appname> like usual to fetch a new build, and move it back into the lib/ directory

This will make the vendored app be tracked locally in source and rebar3 will not even attempt to fetch it.

The caveats we have:

• This does not prevent unvendored app from being fetched (whether with hex or git), which should be done with capabilities or possibly by further patches
• We'll need to generate a rebar3 build step that fetches deps, modifies its own structure to support that vendor format, and adds it to build artifacts so one can just download a vendored rebar3 build (useful for package maintainers)
• It's possible this scheme is gonna fail on plugins, and it will require us to add support to load local plugins (as project_plugins to be safe) from a local directory the way we support doing it from libraries.
• If you are fetching git deps, you may want to delete the .git directory when moving them to lib/

[zuiderkwast]

That's right. I've tried to put them all under lib/ too and it works. I did that after our meeting. But this PR is specifically about bootstrapping rebar3 itself. Are you saying that the lib/ method works for this too?

[zuiderkwast]

This does not prevent unvendored app from being fetched (whether with hex or git), which should be done with capabilities or possibly by further patches

This is what the other PR #2643 is for.

[ferd]

Yeah, #2643 may be more useful, though I'll need to take another look to properly cover network stuff there, to properly prevent all sorts of dep fetches and plugin uses. It's a bit broader than what is currently defined and there may be better hook points for it. Oh and things like hooks and plugins themselves can call the network and can't be covered by any of this either. This is always going to be an imperfect solution compared to permissions work.

[zuiderkwast]

I still need to bootstrap rebar3 (without the undocumented way of manually copying the deps into _build/default/lib/). I don't think the bootstrap script handles lib/.

[zuiderkwast]

@ferd After today's EEF meeting, I thought quickly about how using lib/ for boostrapping could look like (...since you don't seem to like the _checkouts idea).

Idea:

1. Organize all deps as top-level libs under lib/, including rebar3 itself and its deps. (For us, they would all be checked in to the same monorepo.)

   lib/
       rebar/
       getopt/ (and other rebar3 deps)
       myproduct/ (the actual application or product you're building)
       eredis/ (other application deps, which could overlap with rebar3's deps)
   _build/
   

2. Bootstrap rebar3 by calling lib/rebar/bootstrap --libdir=. -o ./rebar3 or similar. (Pardon my ignorance of the actual bootstrap syntax.) This could be implemented as simply copying the required libs (deps) into the _build directory. This is probably the top-level _build directory (but it could also be bootstrapped in lib/rebar/_build/ if you prefer).

3. Build the rest of the applications (libs) by running rebar3 on the top level: ./rebar3

WDYT?

[ferd]

See #2689 for my attempt at automating the first step.

This would on its own handle most cases, and could work in conjunction with an offline mode switch that could be added separately.

[garazdawi]

I did a hack that at least makes it possible to bootstrap rebar3 in its current state, though the method is as mentioned not documented nor supported so could break at any time. My hack does this:

On a machine with internet access run this:

git clone https://github.com/erlang/rebar3
cd rebar3
REBAR_CACHE_DIR=$(pwd)/cache ./bootstrap
rm -rf _build/default/lib/rebar _build/default/lib/*/ebin
git add -f _build/default/lib cache
git commit -m "Add bootstrap dependencies to repository"

And then you push this to your own repo and when on the machine without internet access you just do:

REBAR_CACHE_DIR=$(pwd)/cache ./bootstrap --offline

and it should create rebar3 without having to access the internet. Just thought I'd share what I did until this can be done in a supported way.

[debalance]

FWIW, @weiss created https://github.com/weiss/rebar3 so that I can get a tarball with all required sources included that I can use for the Debian package, which is required to build rebar3 from source completely offline.

[zuiderkwast]

Good to know there are more workarounds.

This PR will most likely be closed soon. The way forward is that rebar3 will vendor its deps using the new vendoring mechanism added in #2689. This means the rebar3 repo (and tar.gz releases) will contain all rebar3's deps. I haven't found any PR or issue tracking this though, so I don't know the progress of it.

[ferd]

We're currently trying to see if we can release a Relx version, then we can release Rebar3 3.19, and then I start the vendoring mode in main.

[zuiderkwast]

Closed in favor of #2720.