Microbatch parallelism #10958

MichelleArk · 2024-10-31T14:22:56Z

Resolves #10853
Resolves #10855

Problem

Part of the benefit of microbatch, is updating a model in smaller (micro) batches. However, before the work in this PR, those batches were run sequentially. Having them run sequentially didn't negate all benefits of splitting into batches (partial instead of complete failures, better data guarantees, easier backfills, etc), however because they were running serially there weren't any speed gains. We wanted the ability to run batches in parallel. This PR does that.

Solution

First we pulled all the batch execution logic into a new MicrobatchModelRunner, which is called from the ModelRunner when a microbatch model is encountered. Handwaving all the complexities of creating a new Runner class, the class determines whether things can be run in parallel in the following.

First, if any of the following are true, a batch is run sequentially

The adapter doesn't support concurrent batches
The relation in the data warehouse for the model doesn't exist

If neither (1) nor (2) was hit, then we check if a config concurrent_batches is set for the model. If the value for that config is True then we run the batches in parallel, if False we run the batches sequentially.

If however concurrent_batches is None (i.e. not set), then we check if the model jinja contains a reference to this. If it references this then we run the batches sequentially. Otherwise, we run them in parallel.

Checklist

I have read the contributing guide and understand what's expected of me.
I have run this code in development, and it appears to resolve the stated issue.
This PR includes tests, or tests are not required or relevant for this PR.
This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
This PR includes type annotations for new and modified functions.

github-actions · 2024-10-31T14:23:14Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

codecov · 2024-10-31T14:29:29Z

Codecov Report

Attention: Patch coverage is 94.92386% with 10 lines in your changes missing coverage. Please review.

Project coverage is 89.10%. Comparing base (f080346) to head (b78f251).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10958      +/-   ##
==========================================
- Coverage   89.13%   89.10%   -0.04%     
==========================================
  Files         183      183              
  Lines       23646    23760     +114     
==========================================
+ Hits        21078    21171      +93     
- Misses       2568     2589      +21

Flag	Coverage Δ
integration	`86.40% <87.81%> (-0.14%)`	⬇️
unit	`62.12% <31.97%> (-0.65%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Unit Tests	`62.12% <31.97%> (-0.65%)`	⬇️
Integration Tests	`86.40% <87.81%> (-0.14%)`	⬇️

---- 🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

… will need to be re-written with new structure

…crobatch handling, TestMicrobatchWithInputWithoutEventTime failing

…batch execution We had a unit test. However that unit test broke upon our work to parallelize things. The parallelization work made it _possible_ for this test to be an integration test, which is actually what we always wanted for this functionality. So we removed the broken unit test, and added a new integration test :)

…ch model

…this` to determine batch parallelism Microbatch batches can be run in parallel, but won't _always_ be run in parallel. There are 3 general cases 1. The batch's model relation doesn't exist -> It can't be run in parallel 2. The relation exists and the model has a `this` invocation -> Assume it can't be run in parallel 3. The relation exists and the model _doesn't_ have a `this` invocation -> Assume it can be run in parallel However there are some exceptions: a. the `this` reference is actually safe (maybe the needed `this` data is guaranteed to exist) b. the batches are small enough (data wise) that it's actually faster to run sequentally Because of (a) and (b) we needed a way to escape out of (1) and (2). Thus we added the `parallel_batches` config. This config defaults to `None`. If it is set though, its value takes precedence over the presence of `this`.

core/dbt/task/run.py

…ity" This reverts commit 8bf1504.

QMalcolm · 2024-11-21T04:24:37Z

@cla-bot check

QMalcolm · 2024-11-21T04:25:05Z

The cla-bot is refusing to show up 🤔

QMalcolm · 2024-11-21T04:34:34Z

@cla-bot check

core/dbt/events/core_types_pb2.py

QMalcolm · 2024-11-21T04:47:53Z

@cla-bot check

MichelleArk · 2024-11-21T05:31:41Z

Merging and bypassing branch protection for verification/cla-signed, as the check seems broken across other branches as well.

MichelleArk added 2 commits October 30, 2024 18:34

first pass: microbatch parallelism: results + build command not working

2314f5e

merge batch-level results + log final result

34fdcf3

cla-bot bot added the cla:yes label Oct 31, 2024

MichelleArk and others added 17 commits October 31, 2024 21:49

Merge branch 'main' into microbatch-parallelism

330ca88

unit test for MicrobatchModelRunner: remove result builder test as it…

8cbd2c6

… will need to be re-written with new structure

move relation_exists to MicrobatchModelRunner

b3bc0b4

move batches to MicrobatchModelRunner

b3317c5

avoid adding microbatch results to run_results twice

c658be8

propagate previously successful batches to final result on retry

9e56650

simplify MicrobatchModelRunner state

92b9aa2

add tracking

3d09292

order batch_results prior to returning

801aede

Merge branch 'main' into microbatch-parallelism

f6c3860

refactor: remove batch_callback in favor of batch_results.append

8b9e669

refactor checkpoint: unify sequential and concurrent while loop in mi…

71bd3d7

…crobatch handling, TestMicrobatchWithInputWithoutEventTime failing

Merge branch 'main' into microbatch-parallelism

69c088d

refactor into RunTask.handle_microbatch_model

109fbd4

Use RunTask.handle_microbatch_model in BuildTask

62f93ab

Run batches in parallel dependent on if this is in jinja

69df6a2

Ensure KeyboardInterrupts/SystemExits get handled in call_runner

2c37abc

QMalcolm added the artifact_minor_upgrade To bypass the CI check by confirming that the change is not breaking label Nov 18, 2024

QMalcolm added 7 commits November 18, 2024 13:54

Add optional parallel_batches to node config

edaeb4c

Regenerate v12 manifest.json to include parallel_batches config

d21fdda

Validate parallel_batches config value in the context of a microbat…

05cfdb5

…ch model

Move _should_run_in_parallel onto MicrobatchModelRunner

4047b1c

Add unit test for all _should_run_in_parallel cases

281a8d8

QMalcolm requested review from a team as code owners November 20, 2024 04:16

QMalcolm requested review from PaulVPham and removed request for a team November 20, 2024 04:16

MichelleArk mentioned this pull request Nov 20, 2024

add base MicrobatchConcurrency capability dbt-labs/dbt-adapters#358

Merged

4 tasks

QMalcolm changed the title ~~[WIP] Microbatch parallelism~~ Microbatch parallelism Nov 20, 2024

gshank approved these changes Nov 20, 2024

View reviewed changes

MichelleArk commented Nov 20, 2024

View reviewed changes

core/dbt/task/run.py Outdated Show resolved Hide resolved

MichelleArk commented Nov 20, 2024

View reviewed changes

core/dbt/task/run.py Outdated Show resolved Hide resolved

QMalcolm added 6 commits November 20, 2024 16:13

Revert "dbt-adapters dev-requirements @microbatch-concurrency-capabil…

9779a41

…ity" This reverts commit 8bf1504.

Remove threading information from batch info logs

1ef88b3

Add MicrobatchExecutionDebug event type

f127b7d

Log whether a batch will be run in parallel/concurrently or not

7b3e54a

Move _build_run_microbatch_model_result to MicrobatchModelRunner

3981e7b

empty commit to retrigger full suite of github actions

12db4ec

QMalcolm removed the cla:yes label Nov 21, 2024

QMalcolm added the cla:yes label Nov 21, 2024

MichelleArk added the proto update update proto definitions in CI label Nov 21, 2024

MichelleArk commented Nov 21, 2024

View reviewed changes

core/dbt/events/core_types_pb2.py Outdated Show resolved Hide resolved

MichelleArk and others added 3 commits November 20, 2024 23:53

fix core_types_pb2.py to version 28.3

daa713e

Bump dbt-adapters minimum to 1.10.1

bb5e267

safer .name access

b78f251

MichelleArk merged commit fd6ec71 into main Nov 21, 2024
52 of 55 checks passed

MichelleArk deleted the microbatch-parallelism branch November 21, 2024 05:31

graciegoheen mentioned this pull request Nov 26, 2024

[Core] microbatch, running batches in parallel dbt-labs/docs.getdbt.com#6550

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Microbatch parallelism #10958

Microbatch parallelism #10958

MichelleArk commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024

codecov bot commented Oct 31, 2024 •

edited

Loading

QMalcolm commented Nov 21, 2024

QMalcolm commented Nov 21, 2024

QMalcolm commented Nov 21, 2024

QMalcolm commented Nov 21, 2024

MichelleArk commented Nov 21, 2024

Microbatch parallelism #10958

Microbatch parallelism #10958

Conversation

MichelleArk commented Oct 31, 2024 • edited Loading

Problem

Solution

Checklist

github-actions bot commented Oct 31, 2024

codecov bot commented Oct 31, 2024 • edited Loading

Codecov Report

QMalcolm commented Nov 21, 2024

QMalcolm commented Nov 21, 2024

QMalcolm commented Nov 21, 2024

QMalcolm commented Nov 21, 2024

MichelleArk commented Nov 21, 2024

MichelleArk commented Oct 31, 2024 •

edited

Loading

codecov bot commented Oct 31, 2024 •

edited

Loading