Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Make --event-end-time require --event-start-time and vice versa #10874

Closed
Tracked by #10624
QMalcolm opened this issue Oct 17, 2024 · 1 comment · Fixed by #10878
Closed
Tracked by #10624

[Feature] Make --event-end-time require --event-start-time and vice versa #10874

QMalcolm opened this issue Oct 17, 2024 · 1 comment · Fixed by #10878
Assignees
Labels
enhancement New feature or request microbatch Issues related to the microbatch incremental strategy
Milestone

Comments

@QMalcolm
Copy link
Contributor

QMalcolm commented Oct 17, 2024

Currently, we have five ways to run a microbatch model 🥴

  1. creating/recreating the model
    • batches are run from model's defined begin to "now"
  2. incrementally with no extra args
    • the batches run are determined by the current time and the lookback
    • f.x. if you have a batch_size=day, lookback=2, and it is "currently" 2024-10-17 13:04:39 then three batches are run: 2024-10-15, 2024-10-16, and 2024-10-17
  3. incrementally with both --event-time-start and --event-time-end specified
    • batches are run >= --event-time-start and < --event-time-end
    • f.x. if you have a batch_size=day, --event-time-start "2024-10-01", and --event-time-end "2024-10-04" then three batches will be run: 2024-10-01, 2024-10-02, 2024-10-03
  4. incrementally with only --event-time-start specified
    • batches will be run from --event-time-start to "now"
    • f.x. if you have a batch_size=day, --event-time-start "2024-10-13", and it's currently midday on 2024-10-17, then five batches will be run: 2024-10-13, 2024-10-14, 2024-10-15, 2024-10-16, and 2024-10-17
  5. incrementally with only --event-time-end specified
    • batches will run the --event-time-end batch plus prior batches according to the period
    • f.x. if you have a batch_size=day, lookback=2, and --event-time-end "2024-10-04", then three batches will be run: 2024-10-01, 2024-10-02, 2024-10-03

Through further discussion, we believe (4) and (5) are confusing, unexpected, and dangerous. Ways (4) and (5) weren't intentional implementations, but accidental realities created by implementations for (1), (2), and (3). As such, we should require that the specification of --event-time-end necessitates --event-time-start and vice versa.

@QMalcolm QMalcolm added microbatch Issues related to the microbatch incremental strategy enhancement New feature or request labels Oct 17, 2024
@QMalcolm QMalcolm added this to the v1.9 milestone Oct 17, 2024
@QMalcolm QMalcolm self-assigned this Oct 17, 2024
@mirnawong1
Copy link
Contributor

docs pr here to address this: https://github.com/dbt-labs/docs.getdbt.com/pull/6351/files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request microbatch Issues related to the microbatch incremental strategy
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants