[Bug] Microbatch: Calling ref() in a macro with a microbatch model name results in error #10928

brian-franklin · 2024-10-28T19:14:42Z

Is this a new bug in dbt-core?

I believe this is a new bug in dbt-core
I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When running a macro that executes ref() with a microbatch model name, an error occurs: 'Macro' object has no attribute 'config'

Expected Behavior

ref() should resolve to a relation with the 3-part name of the object.

Steps To Reproduce

using dbt-snowflake adapter version 1.9.0-b1 and dbt-core version 1.9.0-b2
Create a model using the microbatch incremental strategy
Create a new macro file with the code:

{% macro test_ref_microbatch(node) %}
  {{ log('Running for node: ' ~ node, info=True) }}
  {% set relation=ref(node) %}
  {{ log('relation: ' ~ relation, info=True) }}
{% endmacro %}

Call the macro passing the name of themicrobatch model created in step 2 (replace model_name

dbt run-operation test_ref_microbatch --args '{node: model_name}'

Relevant log output

============================== 15:06:47.975382 | ad60fa66-5fe6-4d70-a9ae-1cb7092779f7 ==============================
�[0m15:06:47.975382 [info ] [MainThread]: Running with dbt=1.9.0-b2
�[0m15:06:47.975382 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'fail_fast': 'False', 'debug': 'False', 'log_path': 'C:\\repos\\hmi_dbt_lib\\logs', 'profiles_dir': 'C:\\Users\\BFranklin\\.dbt', 'version_check': 'True', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'empty': 'None', 'log_format': 'default', 'static_parser': 'True', 'introspect': 'True', 'invocation_command': 'dbt run-operation test_ref_microbatch --args {node: content}', 'target_path': 'None', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'send_anonymous_usage_stats': 'True'}
�[0m15:06:48.263242 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': 'ad60fa66-5fe6-4d70-a9ae-1cb7092779f7', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001FF42684E90>]}
�[0m15:06:48.312158 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': 'ad60fa66-5fe6-4d70-a9ae-1cb7092779f7', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001FF430FDA90>]}
�[0m15:06:48.313226 [info ] [MainThread]: Registered adapter: snowflake=1.9.0-b1
�[0m15:06:48.497237 [debug] [MainThread]: checksum: 4cfbbe2a671212f062e88a43cba5322b30b7e3e8e91b953aa4de249eeaade60f, vars: {}, profile: , target: , version: 1.9.0b2
�[0m15:06:48.628653 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
�[0m15:06:48.628653 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
�[0m15:06:48.643270 [warn ] [MainThread]: [�[33mWARNING�[0m]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
- models.hmi_dbt_lib.landed
- models.hmi_dbt_lib.processed
�[0m15:06:48.669298 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': 'ad60fa66-5fe6-4d70-a9ae-1cb7092779f7', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001FF430CD3D0>]}
�[0m15:06:48.742067 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': 'ad60fa66-5fe6-4d70-a9ae-1cb7092779f7', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001FF421CCF20>]}
�[0m15:06:48.742067 [info ] [MainThread]: Found 12 models, 22 data tests, 10 sources, 637 macros
�[0m15:06:48.742067 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': 'ad60fa66-5fe6-4d70-a9ae-1cb7092779f7', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001FF42666420>]}
�[0m15:06:48.742067 [debug] [MainThread]: Acquiring new snowflake connection 'macro_test_ref_microbatch'
�[0m15:06:48.742067 [info ] [MainThread]: Running for node: content
�[0m15:06:48.742067 [debug] [MainThread]: Snowflake adapter: Error running SQL: macro test_ref_microbatch
�[0m15:06:48.757758 [debug] [MainThread]: Snowflake adapter: Rolling back transaction.
�[0m15:06:48.757758 [debug] [MainThread]: Opening a new connection, currently in state init
�[0m15:06:49.610115 [error] [MainThread]: Encountered an error while running operation: Runtime Error
  'Macro' object has no attribute 'config'
�[0m15:06:49.625892 [debug] [MainThread]: Traceback (most recent call last):
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\adapters\snowflake\connections.py", line 322, in exception_handler
    yield
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\adapters\base\impl.py", line 1151, in execute_macro
    result = macro_function(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt_common\clients\jinja.py", line 354, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt_common\clients\jinja.py", line 322, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\jinja2\runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\jinja2\runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 25, in macro
  File "C:\repos\dbt-env-beta\Lib\site-packages\jinja2\sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\jinja2\runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\context\providers.py", line 305, in __call__
    return self.resolve(name, package, version)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\context\providers.py", line 577, in resolve
    return self.create_relation(target_model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\context\providers.py", line 645, in create_relation
    return super().create_relation(target_model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\context\providers.py", line 614, in create_relation
    event_time_filter=self.resolve_event_time_filter(target_model),
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\context\providers.py", line 244, in resolve_event_time_filter
    and self.model.config.materialized == "incremental"
        ^^^^^^^^^^^^^^^^^
AttributeError: 'Macro' object has no attribute 'config'. Did you mean: 'Config'?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\task\run_operation.py", line 62, in run
    self._run_unsafe(package_name, macro_name)
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\task\run_operation.py", line 47, in _run_unsafe
    res = adapter.execute_macro(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\adapters\base\impl.py", line 1150, in execute_macro
    with self.connections.exception_handler(f"macro {macro_name}"):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\BFranklin\AppData\Local\Programs\Python\Python312\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\repos\dbt-env-beta\Lib\site-packages\dbt\adapters\snowflake\connections.py", line 361, in exception_handler
    raise DbtRuntimeError(str(e)) from e
dbt_common.exceptions.base.DbtRuntimeError: Runtime Error
  'Macro' object has no attribute 'config'

�[0m15:06:49.626475 [debug] [MainThread]: Command `dbt run-operation` failed at 15:06:49.626475 after 1.78 seconds
�[0m15:06:49.626475 [debug] [MainThread]: Connection 'macro_test_ref_microbatch' was left open.
�[0m15:06:49.626475 [debug] [MainThread]: On macro_test_ref_microbatch: Close
�[0m15:06:49.742282 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001FF4251B890>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001FF421CCF20>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001FF41B81040>]}
�[0m15:06:49.742282 [debug] [MainThread]: Flushing usage events
�[0m15:06:49.874262 [debug] [MainThread]: An error was encountered while trying to flush usage events

Environment

- OS: Windows 10 Enterprise
- Python: 3.12.7
- dbt: 1.9.0-b2
- dbt-snowflake: 1.9.0-b1

Which database adapter are you using with dbt?

snowflake

Additional Context

Here is my microbatch model config:

{{
  config(
    materialized='incremental',
    incremental_strategy='microbatch',
    event_time='landed_timestamp',
    begin='2023-01-01',
    batch_size='day',
    on_schema_change='append_new_columns',
    schema='inscape',
    tags=["inscape"]
    )
}}

The text was updated successfully, but these errors were encountered:

dbeatty10 · 2024-10-29T03:13:56Z

Thanks for reporting this @brian-franklin !

I couldn't reproduce the error you got. Could you try the following and see if it works for you or not?

Project files and commands

Create these files:

models/model_name.sql

{{
  config(
    materialized='incremental',
    incremental_strategy='microbatch',
    event_time='landed_timestamp',
    begin='2023-01-01',
    batch_size='day',
    on_schema_change='append_new_columns',
    schema='inscape',
    tags=["inscape"]
    )
}}

select 1 as id

macros/my_macro.sql

{% macro test_ref_microbatch(node) %}
  {{ log('Running for node: ' ~ node, info=True) }}
  {% set relation=ref(node) %}
  {{ log('relation: ' ~ relation, info=True) }}
{% endmacro %}

Run this command:

dbt run-operation test_ref_microbatch --args '{node: model_name}'

Log output:

$ dbt run-operation test_ref_microbatch --args '{node: model_name}'
03:05:17  Running with dbt=1.9.0-b2
03:05:18  Registered adapter: snowflake=1.9.0-b1
03:05:19  Found 1 model, 464 macros
03:05:19  Running for node: model_name
03:05:19  relation: analytics_dev.dbt_dbeatty_inscape.model_name

brian-franklin · 2024-10-29T13:46:00Z

@dbeatty10 I tried this and it worked, however, I realize I left out an aspect of my project setup that I now see it causing the issue. I am using an environment variable to set the database name. Here is a representative example of the section project yaml

dbt_project yaml

models:
  testing:
    database: "{{ env_var('DBT_CURATED_DB') }}"
    # Config indicated by + and applies to all files under models/example/
    example:
      +materialized: view

I am using a setting.json file to set the environment variable...

settings.json file

{
  "terminal.integrated.defaultProfile.windows": "PowerShell",
  "terminal.integrated.env.windows": {
    "DBT_CURATED_DB": "HMI_CURATED_CLONE_PROD_NS1_656",
    "DBT_EXPERIMENTAL_MICROBATCH": "True"
  }
}

Now when I run the macro, I am able to reproduce the error...
dbt run-operation test_ref_microbatch --args '{node: model_name}'

Log Output

(dbt-env-beta) PS C:\repos\testing> dbt run-operation test_ref_microbatch --args '{node: model_name}'
13:38:56  Running with dbt=1.9.0-b2
13:38:56  Registered adapter: snowflake=1.9.0-b1
13:38:56  Unable to do partial parsing because env vars used in dbt_project.yml have changed
13:38:56  Unable to do partial parsing because a project config has changed
13:38:57  Found 3 models, 4 data tests, 466 macros
13:38:57  Running for node: model_name
13:38:58  Encountered an error while running operation: Runtime Error
  'Macro' object has no attribute 'config'

dbeatty10 · 2024-10-29T22:00:25Z

Ah, I see what you are saying @brian-franklin 👍

All I needed to do to reproduce this was to start with #10928 (comment) and then set the DBT_EXPERIMENTAL_MICROBATCH environment variable to True.

Here's the place in the code where the error is triggered:

dbt-core/core/dbt/context/providers.py

Line 244 in 8df5c96

and self.model.config.materialized == "incremental"

MichelleArk · 2024-11-05T18:19:04Z

Should be a matter of more safely determining whether to resolve an event_time_filter. We should only be doing this for microbatch models, so an isinstance(self.model, ModelNode) should suffice there: https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/context/providers.py#L244C22-L244C27

brian-franklin added bug Something isn't working triage labels Oct 28, 2024

dbeatty10 added the microbatch Issues related to the microbatch incremental strategy label Oct 29, 2024

dbeatty10 added awaiting_response and removed triage labels Oct 29, 2024

github-actions bot added triage and removed awaiting_response labels Oct 29, 2024

dbeatty10 added pre-release Bug not yet in a stable release and removed triage labels Oct 29, 2024

graciegoheen mentioned this issue Nov 4, 2024

[EPIC] Incremental Model Improvements - Microbatch #10624

Open

MichelleArk mentioned this issue Nov 6, 2024

[Fix] assert resolver.model is ModelNode prior to resolving event_time_filter #10975

Merged

5 tasks

MichelleArk closed this as completed in #10975 Nov 6, 2024

MichelleArk self-assigned this Dec 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Microbatch: Calling ref() in a macro with a microbatch model name results in error #10928

[Bug] Microbatch: Calling ref() in a macro with a microbatch model name results in error #10928

brian-franklin commented Oct 28, 2024 •

edited by dbeatty10

Loading

dbeatty10 commented Oct 29, 2024

brian-franklin commented Oct 29, 2024

dbeatty10 commented Oct 29, 2024

MichelleArk commented Nov 5, 2024

[Bug] Microbatch: Calling ref() in a macro with a microbatch model name results in error #10928

[Bug] Microbatch: Calling ref() in a macro with a microbatch model name results in error #10928

Comments

brian-franklin commented Oct 28, 2024 • edited by dbeatty10 Loading

Is this a new bug in dbt-core?

Current Behavior

Expected Behavior

Steps To Reproduce

Relevant log output

Environment

Which database adapter are you using with dbt?

Additional Context

dbeatty10 commented Oct 29, 2024

Project files and commands

brian-franklin commented Oct 29, 2024

dbt_project yaml

settings.json file

Log Output

dbeatty10 commented Oct 29, 2024

MichelleArk commented Nov 5, 2024

brian-franklin commented Oct 28, 2024 •

edited by dbeatty10

Loading