Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Emit a warning if none of a microbatch model's ref/source inputs have an event_time defined #10926

Closed
Tracked by #10624
graciegoheen opened this issue Oct 28, 2024 · 0 comments · Fixed by #10929
Assignees
Labels
enhancement New feature or request microbatch Issues related to the microbatch incremental strategy

Comments

@graciegoheen
Copy link
Contributor

graciegoheen commented Oct 28, 2024

Describe the feature

An incremental model only filters it's upstream ref/source inputs if the upstream has an event_time config defined. We should emit a warning if none of a microbatch model's ref/source inputs have an event_time defined.

Why

We've found the following to not be an uncommon issue:

  1. Someone creates a microbatch model, model_a
  2. model_a has one or more input model/sources
  3. an event_time config is not added to any of model_a's inputs
  4. model_a gets run, but every batch has the same number of rows (which matches the combined size of the inputs)

This is a problem because it isn't immediately apparent to the person that created model_a what happened / went wrong.

@graciegoheen graciegoheen added enhancement New feature or request microbatch Issues related to the microbatch incremental strategy labels Oct 28, 2024
@QMalcolm QMalcolm changed the title emit a warning if none of a microbatch model's ref/source inputs have an event_time defined [Feature] Emit a warning if none of a microbatch model's ref/source inputs have an event_time defined Oct 28, 2024
@MichelleArk MichelleArk self-assigned this Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request microbatch Issues related to the microbatch incremental strategy
Projects
None yet
2 participants