Replace environment variable with a project flag to gate microbatch functionality #10799

MichelleArk · 2024-09-30T18:56:05Z

Resolves #10798

Problem

We had gated the new microbatch feature behind an environment variable in the initial implementation of microbatch (#10594). However, for a better experience, we want people to be setting a project flag (AKA behavior flag).

Solution

Gate microbatch functionality behind a project flag (AKA behavior flag) instead of an environment variable.

Checklist

I have read the contributing guide and understand what's expected of me.
I have run this code in development, and it appears to resolve the stated issue.
This PR includes tests, or tests are not required or relevant for this PR.
This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
This PR includes type annotations for new and modified functions.

github-actions · 2024-09-30T18:56:20Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

codecov · 2024-09-30T18:58:37Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.07%. Comparing base (30b8a92) to head (0f6b7fb).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10799      +/-   ##
==========================================
- Coverage   89.12%   89.07%   -0.06%     
==========================================
  Files         183      183              
  Lines       23592    23626      +34     
==========================================
+ Hits        21027    21044      +17     
- Misses       2565     2582      +17

Flag	Coverage Δ
integration	`86.37% <100.00%> (-0.14%)`	⬇️
unit	`62.79% <52.94%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Unit Tests	`62.79% <52.94%> (-0.03%)`	⬇️
Integration Tests	`86.37% <100.00%> (-0.14%)`	⬇️

…yDefault` as it was prior to a9df50f

…with behavior flag changes This is needed to get the tests to pass. This is only necessary until the changes in dbt-adapters are merged. Said another way: we need to merge the changes to dbt-adapters first, and then revert these dependency changes before merging these changes.

QMalcolm · 2024-10-01T21:30:03Z

Related PRs

Switch from environment variable to behavior flag for gating microbatch functionality dbt-adapters#323
Update test_incremental_microbatch to use behavior flag instead of environment variable dbt-bigquery#1358

QMalcolm · 2024-10-01T21:59:38Z

dev-requirements.txt

-git+https://github.com/dbt-labs/dbt-adapters.git@main
-git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-tests-adapter
+git+https://github.com/dbt-labs/dbt-adapters.git@microbatch-behavior-flag
+git+https://github.com/dbt-labs/dbt-adapters.git@microbatch-behavior-flag#subdirectory=dbt-tests-adapter


This is needed to get the tests to pass. This is only necessary until the changes in dbt-adapters are merged. Said another way: we need to merge the changes to dbt-adapters first, and then revert these dependency changes before merging this PR's changes.

…o `find_materialization_macro_candidate_by_name` This is necessary because in the following commit we're going to add a function to get whether or not a manifest should be run using the new microbatch functionality. Getting the MacroCandidate is necessary because we need to know the locality of the macro.

…ity should be used The new microbatch functionality is, unfortunately, potentially dangerous. That is it adds a new materalization strategy `microbatch` which an end user could have defined as a custom strategy previously. Additionally we added config keys to nodes, and as `config` is just a Dict[str, Any], it could contain anything, thus meaning people could already be using the configs we're adding for different purposes. Thus we need some intellegent gating. Specifically something that adheres to the following: cms = Custom Microbatch Strategy abms = Adapter Builtin Microbatch Strategy bf = Behavior flag umb = Use Microbatch Batching t/f/e = True/False/Error | cms | abms | bf | umb | | t | t | t | t | | f | t | t | t | | t | f | t | t | | f | f | t | e | | t | t | f | f | | f | t | f | t | | t | f | f | f | | f | f | f | e | (The above table assumes that there is a microbatch model present in the project) In order to achieve this we need to check that either the microbatch behavior flag is set to true OR microbatch materializaion being used is the _root_ microbatch materialization (i.e. not custom). The function we added in this commit, `use_microbatch_batches`, does just that.

…nction

…tom_microbatch_strategy`

In 0349968 I had done this for the function `find_materialization_macro_by_name`, but that wasn't the right function to do it to, and will be reverted shortly. `find_materialization_macro_by_name` is used for finding the general materialization macro, whereas `find_macro_by_name` is more general. For the work we're doing, we need to find the microbatch macro, which is not a materialization macro.

…_name` to `find_materialization_macro_candidate_by_name`" This reverts commit 0349968.

…tead of `root` Previously were were checking for a locality of `root`. However, a locality of `root` means it was provided by a `package`. We wnt to check for locality of `core` which basically means `builtin via dbt-core/adapters`. There is another locality `imported` which I beleive means it comes from another package.

…in boolean checks The method `use_microbatch_batches` is always invoked to evaluate an `if` statement. In most instances, it is part of a logic chain (i.e. there are multiple things being evaluated in the `if` statement). In `if` statements where there are multiple things being evaulated, `use_microbatch_batches` should come _last_ (or as late as possible). This is because it is likely the most costly thing to evaluate in the logic chain, and thus any shortcuts cuts via other evaluations in the if statement failing (and thus avoiding invoking `use_microbatch_batches) is desirable.

…ches

MichelleArk · 2024-11-08T22:18:08Z

dev-requirements.txt

-git+https://github.com/dbt-labs/dbt-adapters.git@main
-git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-tests-adapter
+git+https://github.com/dbt-labs/dbt-adapters.git@microbatch-behavior-flag
+git+https://github.com/dbt-labs/dbt-adapters.git@microbatch-behavior-flag#subdirectory=dbt-tests-adapter


reminder: let's make sure we bump the lower bound pin of dbt-adapters in setup.py as part of this PR once the dbt-adapters branches are merged.

MichelleArk · 2024-11-08T22:19:32Z

core/dbt/events/types.py

+        return "D020"
+
+    def message(self) -> str:
+        description = "The use of a custom microbatch macro outside of batched execution is deprecated. To use it with batched execution, set `flags.require_batched_execution_for_custom_microbatch_strategy` to `True` in `dbt_project.yml`. In the future this will be the default behavior."


super nit, just because this is user-facing:

"In the future this will be the default behavior." -> "In the future batched execution will be the default behaviour."

We could also mention the name of the custom macro that exists (e.g. get_incremental_microbatch_sql) as part of the message in case the user that's seeing this doesn't know about the custom strategy macro naming offhand.

MichelleArk

✅ Can't approve a PR I "authored" but here is my psuedo-approval!

Thank you for taking this across the finish line @QMalcolm, clean + precise work ✨

QMalcolm

Self approving because I didn't open the PR 🙈

… branch with behavior flag changes" This reverts commit 5b9d7f7.

mirnawong1 · 2024-11-11T13:20:00Z

hey @QMalcolm @MichelleArk , the docs team received this github issue.

i was reviewing this dbt-adapters PR and #10799 -- am i correct in thinking that the following needs to be added to the behavior flag doc:

the require_batched_execution_for_custom_microbatch_strategy flag defaults to False, which will raise a deprecation warning. Set it to True to allow custom microbatch strategies (like get_incremental_microbatch_sql) to use batched execution.

QMalcolm · 2024-11-11T14:53:17Z

hey @QMalcolm @MichelleArk , the docs team received this github issue.

i was reviewing this dbt-adapters PR and #10799 -- am i correct in thinking that the following needs to be added to the behavior flag doc:

the require_batched_execution_for_custom_microbatch_strategy flag defaults to False, which will raise a deprecation warning. Set it to True to allow custom microbatch strategies (like get_incremental_microbatch_sql) to use batched execution.

@mirnawong1 It's actually a little bit different than that. The flag require_batched_execution_for_custom_microbatch_strategy does default to False. However, a user only needs to set require_batched_execution_for_custom_microbatch_strategy to True if they have a custom microbatch macro. If they don't have a custom microbatch macro, everything will work. Additionally, they'll only get a depreciation warning IFF they have a custom microbatch strategy and the flag is False.

mirnawong1 · 2024-11-11T15:04:58Z

hey @QMalcolm oh ok i see! so require_batched_execution_for_custom_microbatch_strategy is for custom microbatch macros only.

so does this also mean we have another flag: require_builtin_microbatch_strategy (pr here), which replaces the DBT_EXPERIMENTAL_MICROBATCH env var? and they should set to true if they had the env_var already enabled?

QMalcolm · 2024-11-11T15:10:53Z

@mirnawong1 There is no other flag. I recognize that as slightly confusing. For context, the only reason we gated microbatch in the first place was the fear of breaking people's projects who already had a custom microbatch macro. What we've done here is make it so one only needs to set a flag if they have a custom microbatch macro, instead of making everyone set a flag because someone else's project might have a microbatch macro.

For a person setting up a microbatch model for the first time, there is no longer a flag or environment variable they need to set (assuming they don't have a custom microbatch macro)

mirnawong1 · 2024-11-11T15:35:26Z

oh ok got it! so i can update the callout here and replace it with content informing users of the behavior flag should they have a custom microbatch macro only. all other users that are setting up incremental microbatch strategy for the first time w no existing custom macro don't need to do anything apart from their config.

thank you so much for clarifying this!

first pass: replace os env with project flag

a9df50f

cla-bot bot added the cla:yes label Sep 30, 2024

QMalcolm added 6 commits October 1, 2024 11:26

Merge branch 'main' into microbatch-project-flags

96c4ccb

Fix TestMicrobatchMultipleRetries to not use os.env

704494f

Turn off microbatch project flag for `TestMicrobatchCustomUserStrateg…

bfecde1

…yDefault` as it was prior to a9df50f

Update BaseMicrobatchTest to turn on microbatch via project flags

d690dec

Add changie doc

1105e42

QMalcolm marked this pull request as ready for review October 1, 2024 21:25

QMalcolm requested a review from a team as a code owner October 1, 2024 21:25

QMalcolm mentioned this pull request Oct 1, 2024

[Microbatch] Update tests to use microbatch behavior flag dbt-labs/dbt-bigquery#1359

Open

QMalcolm changed the title ~~first pass: replace os env with project flag~~ Replace environment variable with a project flag to gate micorbatch functionality Oct 1, 2024

QMalcolm mentioned this pull request Oct 1, 2024

[Microbatch] Use behavior flag to gate microbatch functionality dbt-labs/dbt-adapters#327

Closed

QMalcolm reviewed Oct 1, 2024

View reviewed changes

QMalcolm requested a review from a team October 1, 2024 22:00

Merge branch 'main' into microbatch-project-flags

a3f2708

QMalcolm mentioned this pull request Oct 21, 2024

Update test to handle moving from microbatch environment variable to behavior flag dbt-labs/dbt-postgres#167

Closed

4 tasks

Merge branch 'main' into microbatch-project-flags

18d1938

MichelleArk changed the title ~~Replace environment variable with a project flag to gate micorbatch functionality~~ Replace environment variable with a project flag to gate microbatch functionality Nov 4, 2024

QMalcolm added 5 commits November 4, 2024 17:39

Fix functional tests after merging in main

4d908ad

Gate microbatch functionality by use_microbatch_batches manifest fu…

e77a13c

…nction

Rename microbatch behavior flag to `require_batched_execution_for_cus…

543e024

…tom_microbatch_strategy`

QMalcolm force-pushed the microbatch-project-flags branch from 9b3c122 to 543e024 Compare November 5, 2024 00:41

QMalcolm added 2 commits November 6, 2024 13:25

Merge branch 'main' into microbatch-project-flags

db3fba8

QMalcolm added 11 commits November 7, 2024 17:27

Use find_macro_candidate_by_name to find the microbatch macro

35b23a3

Revert "Extract most of the logic from `find_materialization_macro_by…

7808992

…_name` to `find_materialization_macro_candidate_by_name`" This reverts commit 0349968.

Merge branch 'main' into microbatch-project-flags

874d7c6

Drop behavior flag setting for BaseMicrobatchTest tests

dff25de

Rename 'env_var' to 'project_flag' in test_microbatch.py

3931b69

Update microbatch tests to assert when we are/aren't running with bat…

945bcff

…ches

Update test_resolve_event_time_filter to use use_microbatch_batches

63445b5

Fire deprecation warning for custom microbatch macros

26ba807

Add microbatch deprecation events to test_events.py

94363e8

MichelleArk commented Nov 8, 2024

View reviewed changes

QMalcolm approved these changes Nov 8, 2024

View reviewed changes

Revert "[REVERT BEFORE MERGE] Update dbt-adapters dev req to point to…

0f6b7fb

… branch with behavior flag changes" This reverts commit 5b9d7f7.

mirnawong1 mentioned this pull request Nov 11, 2024

Docs Changes Needed from dbt-adapters Issue #327 dbt-labs/docs.getdbt.com#6447

Closed

QMalcolm merged commit 89caa33 into main Nov 11, 2024
58 of 60 checks passed

QMalcolm deleted the microbatch-project-flags branch November 11, 2024 14:49

mirnawong1 mentioned this pull request Nov 11, 2024

add behavior flag for microbatch dbt-labs/docs.getdbt.com#6455

Merged

runleonarun mentioned this pull request Nov 12, 2024

Revert "Revert "add behavior flag for microbatch"" dbt-labs/docs.getdbt.com#6467

Merged

1 task

This was referenced Nov 13, 2024

Bump minimum dbt-adapters version to 1.9.0 #10996

Closed

Bump minimum dbt-adapters requirement to 1.9.0 #10997

Closed

Bump minimum dbt-adapters requirement to 1.9.0 #10998

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace environment variable with a project flag to gate microbatch functionality #10799

Replace environment variable with a project flag to gate microbatch functionality #10799

MichelleArk commented Sep 30, 2024 •

edited by QMalcolm

Loading

github-actions bot commented Sep 30, 2024

codecov bot commented Sep 30, 2024 •

edited

Loading

QMalcolm commented Oct 1, 2024

QMalcolm Oct 1, 2024

MichelleArk Nov 8, 2024

MichelleArk Nov 8, 2024 •

edited

Loading

MichelleArk Nov 8, 2024 •

edited

Loading

MichelleArk left a comment

QMalcolm left a comment

mirnawong1 commented Nov 11, 2024 •

edited

Loading

QMalcolm commented Nov 11, 2024

mirnawong1 commented Nov 11, 2024

QMalcolm commented Nov 11, 2024

mirnawong1 commented Nov 11, 2024

Replace environment variable with a project flag to gate microbatch functionality #10799

Replace environment variable with a project flag to gate microbatch functionality #10799

Conversation

MichelleArk commented Sep 30, 2024 • edited by QMalcolm Loading

Problem

Solution

Checklist

github-actions bot commented Sep 30, 2024

codecov bot commented Sep 30, 2024 • edited Loading

Codecov Report

QMalcolm commented Oct 1, 2024

QMalcolm Oct 1, 2024

Choose a reason for hiding this comment

MichelleArk Nov 8, 2024

Choose a reason for hiding this comment

MichelleArk Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

MichelleArk Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

MichelleArk left a comment

Choose a reason for hiding this comment

QMalcolm left a comment

Choose a reason for hiding this comment

mirnawong1 commented Nov 11, 2024 • edited Loading

QMalcolm commented Nov 11, 2024

mirnawong1 commented Nov 11, 2024

QMalcolm commented Nov 11, 2024

mirnawong1 commented Nov 11, 2024

MichelleArk commented Sep 30, 2024 •

edited by QMalcolm

Loading

codecov bot commented Sep 30, 2024 •

edited

Loading

MichelleArk Nov 8, 2024 •

edited

Loading

MichelleArk Nov 8, 2024 •

edited

Loading

mirnawong1 commented Nov 11, 2024 •

edited

Loading