[CT-527] Unexpected macro resolution order for collect_freshness macro

[jtcohen6]

Split off from #4525

When running dbt source freshness, why does the definition of collect_freshness in a package silently take precedence over the global/default version in dbt-core, without the end user / root project "opting into" the override?

These codepaths are in execution/adapter land:

dbt-core/core/dbt/task/freshness.py

Line 145 in 979e1c7

freshness = self.adapter.calculate_freshness(

dbt-core/core/dbt/adapters/base/impl.py

Lines 1005 to 1006 in 979e1c7

	# run the macro
	table = self.execute_macro(FRESHNESS_MACRO_NAME, kwargs=kwargs, manifest=manifest)

But the real question here is around the macro resolver for collect_freshness, so I think Language team is best positioned to weigh in. For all other "built-in" macros (ones defined in the global project), only the root project can override them, or explicitly opt into a dispatch order (docs) that prefers macros from packages.

Reproduction case

Create a source for a nonexistent table in your project:

version: 2

sources:
  - name: fake_source
    loaded_at_field: current_timestamp
    freshness:
      warn_after: {period: day, count: 7}
    tables:
      - name: does_not_exist

Run dbt source freshness, fails because the table doesn't exist.

Create a local package with macros/collect_freshness.sql:

{% macro collect_freshness(source, loaded_at_field, filter) %}
  {% call statement('collect_freshness', fetch_result=True, auto_begin=False) -%}

  -- postgres syntax
  select
    current_timestamp as max_loaded_at,
    current_timestamp as snapshotted_at

  {% endcall %}
  {{ return(load_result('collect_freshness').table) }}
{% endmacro %}

Add that local package in packages.yml:

packages:
  - local: local_package

Run dbt source freshness, it passes—even though the end user never opted into the override.

Expected behavior

The user could override collect_freshness in their own project, to point to the package version instead:

{% macro collect_freshness(source, loaded_at_field, filter) %}
    {{ return(fivetran_utils.collect_freshness(source, loaded_at_field, filter) }}
{% endmacro %}

Or rearrange dispatch order to prefer candidate macros from the fivetran_utils package:

dispatch:
  - macro_namespace: collect_freshness
    search_order: ['fivetran_utils', 'dbt']

[nathaniel-may]

estimation:

• several moving parts
• tests may be difficult
• actual change likely to be small once the right places in the code are found

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.