Experimental vendoring provider

[ferd]

The new experimental vendoring provider is called as rebar3 experimental vendor,
and works by doing the following:

• Detecting whether the app is an umbrella application (only works on
  them)
• Fetching all dependencies for the project
• Moving all dependencies to the top-level vendor/ directory
• Telling the user to configure {project_app_directories, ...} to
  include the vendored path
• Telling the user to move all {deps, ...} entries away from the
  top-level rebar.config file and into sub-applications' files

This then works by changing all non-checkout dependencies into top-level
apps stored elsewhere than the user-supplied ones.

This should also implicitly empty the lock file on the next compilation.

When called again, the provider:

• Makes itself ignore the previous {project_app_directories, ...}
  values
• Moves whatever the content of vendor/ was into _vendor/ (which is
  pre-emptively emptied) to capture errors of any kind.
• Re-fetches the dependencies (the ebin/ directories aren't cleared in
  case the dependency ships pre-built beam files)
  • Since the dependencies were unlikely to be locked, they are
    upgraded
• Tells the user to re-update their vendoring configuration

This is a proposed experimental value that would enshrine a suggested
vendoring mechanism for offline builds for Ericsson et al. The idea would be
that at some point, Rebar3 itself would be able to reuse that pattern (after
migrating to an umbrella project structure) in order to safely vendor deps and
allowing builds from official release tarballs on offline systems.

Currently not handled:

• Tests
• Support for any plugins whatsoever.
  • this would require adding support for project-internal plugins, which might actually be cool, but I couldn't tackle in a single night.
• Dealing with stray .git directories (and other source control artifacts) from dependencies that have been fetched
• User-friendly error-handling (including loud warnings of circular dependencies when deps stay at the top-level)

[ferd]

CC @tsloughter @zuiderkwast @KennethL -- this is following the last B&P discussion about this being a possible step to deal with vendoring and accepting a later offline mode to rebar3.

[zuiderkwast]

• Fetching all dependencies for the project

(...)

• Re-fetches the dependencies

@ferd Nice to see progress! However, the "fetch" steps will most likely prevent us from using the feature. I suppose we will have to organize everything manually.

[ferd]

• Fetching all dependencies for the project

(...)

• Re-fetches the dependencies

@ferd Nice to see progress! However, the "fetch" steps will most likely prevent us from using the feature. I suppose we will have to organize everything manually.

Those should only be called whenever you feel like vendoring the first time or for upgrading. I’m not sure why they’re a blocker, though I could imagine that if you maintain a patch set it works clobber some (hence the _vendor backup)

[KennethL]

@ferd I don't think the fetch functionality here is a blocker but I don't think it will be used in our scenario.
I imagine that we will and want to fetch all the dependencies ourselves and not necessarily from the same location as the dep in the rebar.config points to.
The dependencies we have fetched are then to be put in suitable places for rebar3 to fetch locally.

Having rebar3 to tell us what dependencies we need to fetch and put in place would be helpful. I think it sort of already can do that, but maybe not without accessing the network. Of course it would miss nested dependencies then but that should work better next attempt when we have put the first level of deps in place.

Of course finding out and fetching of deps can also be done by manually reading the rebar.conf

[zuiderkwast]

Those [fetching of deps] should only be called whenever you feel like vendoring the first time or for upgrading. I’m not sure why they’re a blocker, though I could imagine that if you maintain a patch set it works clobber some (hence the _vendor backup)

@ferd Then I misunderstood you. As long as it works without those online steps (fetching & upgrading) then I guess it's all good, as Kenneth explained.

[ferd]

The failing shelltest is due to github deprecating the git://github.com/user/repo way of fetching without default ports and dependencies suffering through that. It's unrelated to this branch.

[tsloughter]

Approving since I think it is likely worth it to have a task for doing this with deps, but I worry this particular vendoring won't be acceptable by those asking for vendoring the most. But this is still probably a step in that direction.

[ferd]

Yeah, I'll merge it. It's tagged as experimental and can be removed/changed at any point in time.

Namely to rework will include:

• Tests
• Support for any plugins whatsoever.
  • this will require adding support for project-internal plugins, which might actually be cool
• Dealing with stray .git directories (and other source control artifacts) from dependencies that have been fetched
• User-friendly error-handling (including loud warnings of circular dependencies when deps stay at the top-level)

There is also an assumption that:

• The rebar3 repo can at some point be shaped to work that way as an experiment to help package managers
• That we can pair this with the pending/submitted PRs from Ericsson folks to bypass network calls and go offline fully

[zuiderkwast]

Support for any plugins whatsoever.

Hi Fred! As we've talked about before, an umbrella project can override deps and deps-of-deps by placing them in lib/ or apps/. An app included in an umbrella will take precedence over any dep with the same name. (I still haven't found this in the docs but I hope my understanding is right.)

Is something similar possible with plugins? Let's say rebar3_fmt that create custom actions or a compile time dep, something that's not to be included in our release. We want the local version to be used rather than downloading one.

[ferd]

I believe this is already supported yes. If it isn't, there may be a bug in plugin handling code, following these two PRs:

• Add support for project-local plugins #2697
• Support plugins in experimental vendor provider #2702

[zuiderkwast]

Thanks!

So for the record, the place to put them is plugins/ (or config project_plugin_dirs) in the umbrella project.

[ferd]

yep.