Allow dispatch & overrides of global macros via packages

[clausherther]

Describe the feature

As a dbt package developer, I'd like to be able to override a built-in (global) macro like create_table_as or is_incremental in a package and have projects that import the package transparently use the built-in macro without any changes to their code.
One way to accomplish this would be to allow dbt to apply its package dispatch functionality to global macros.
@jtcohen6 has suggested the following syntax in this Slack thread.

dispatch:
  - macro_namespace: dbt
    search_order: ['my_package', 'dbt']

This would effectively inject my_package in the dispatch search_path for global macros in the dbt project.
It would also maintain the current behavior where dbt looks for overrides of global macros in the current root project and should follow the precedence described for materializations here.

So, e.g. if someone

• overwrote create_table_as locally in the root project only, dbt would use the local version
• imported a package that overwrote create_table_as only, dbt would use the package version (over the global version)
• overwrote create_table_as locally in the root project and imported a package that overwrote create_table_as, dbt would use the local version in the root project, over the package version and over the global version.

Another use case, in addition to overrides of the default__ version of macros, would be for a package to supply alternate versions of global macros for specific platforms. Ideally such platform specific versions of global macros should belong in the respective plugin's global project, but there may be use cases for company-specific overrides to particular platforms that don't make sense in the plugin project.

Describe alternatives you've considered

An alternative would be to wrap the macro exposed by the package locally. This would require project owners to be aware of the override and maintain a local wrapper macro for each override.

Who will this benefit?

This would benefit package maintainers, particular those maintaining internal, company specific packages that override default behavior with customized implementations.

[jtcohen6]

Thanks for opening @clausherther, and for making such a clear statement of the problem. I agree with all the examples and use cases you've provided. To make it as pellucid as possible, there are two phenomena at play here:

1. When resolving "internal" macros, dbt looks to the root project and internal packages (dbt, dbt_postgres, dbt_presto, etc)
2. When resolving dispatched macros, dbt looks to the dispatch config and uses the search order defined there. Otherwise, it just searches in the package identified by the macro_namespace (or the package where the macro is defined, if no macro_namespace is provided)

We're talking about changing the behavior of the latter, but leaving the former in place.

I think this might be as simple as going through the dispatch calls within the global_project and setting the macro_namespace to 'dbt'. The thing I'd want to make sure is that various adapter plugins, whose macros live in a slightly different namespace (dbt_postgres, dbt_presto), would still have their implementations included in the search order by default. If that manages to work, we may get a lot of this "for free."

To go example by example:

So, e.g. if someone

• overwrote create_table_as locally in the root project only, dbt would use the local version

This would continue to work exactly as it does today.

• imported a package that overwrote create_table_as only, dbt would use the package version (over the global version)

In order for this to work, the package would need to reimplement either default__create_table_as or (e.g.) presto__create_table_as and be included in the dispatch search_order. Only the root package can totally override the original create_table_as macro.

• overwrote create_table_as locally in the root project and imported a package that overwrote create_table_as, dbt would use the local version in the root project, over the package version and over the global version.

This would continue to work exactly as it does today, where the root create_table_as is preferred over the internal one. In all likelihood, no dispatching occurs (unless the root version includes a call to dispatch). If instead, however, both the root project and the imported package define versions of default__create_table_as, then the search_order in the dispatch config would define the order of precedence.

One final note that came up in our Slack conversation: The way we've implemented the dispatch config in v0.19.2/v0.20.0 requires a full explicit list of all packages to search, provided to search_order. In the future, we may want to implicitly tack on the root project as the first search package and the macro namespace as the last one. I can't think of a case where someone wouldn't want to prefer their own root version, or wouldn't want to ultimately fall back on the original package (rather than raising an error). For the time being, we'll be erring on the side of too-explicit syntax.

[clausherther]

Also 💯 points for the (near) alliteration of "pellucid...possible...phenomena"