-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert DagFileProcessor.execute_callbacks to Internal API #28900
Merged
+1,579
−683
Merged
Changes from 1 commit
Commits
Show all changes
118 commits
Select commit
Hold shift + click to select a range
609aebb
Start converting _execute_task_callbacks to internal API. get_templat…
vincbeck ac48800
Move get_serialized_dag and get_task_instance methods to model classes
vincbeck b57ddca
Convert handle_failure() method to internal API
vincbeck 917abfb
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 6dfebce
Remove comment
vincbeck 1867db4
Migrate _execute_dag_callbacks to internal API
vincbeck 6b110ea
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck c21fb81
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck d92fa83
Use cls.logger()
vincbeck cc52f47
Remove todo
vincbeck ee27659
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 68bfc72
Add back if callbacks
vincbeck 839dc6c
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 30d6ef9
Remove comments
vincbeck 9fb937d
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 81ee61b
Refactore _fetch_callback
vincbeck 3c300a7
Fix TaskInstance.get_task_instance
vincbeck dc6031e
Fix unit tests
vincbeck 12c908a
Fix unit tests
vincbeck ccf3a86
Add methods to rpc_endpoint
vincbeck 791ddbe
Add flag select_columns to get_task_instance
vincbeck 120a030
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 4f1ef33
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck e72652e
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 2fca2b3
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 48ef1cf
Use Pydantic-powered ORM models
vincbeck ee63263
Fix DagRunPydantic
vincbeck 3f5dc7f
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck f64ea0c
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck b61f9e6
Fix imports
vincbeck 7bed9bc
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck c8a8d20
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 9277e53
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck fea8db0
Convert methods in models to private functions
vincbeck 1fff88d
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 6b80508
Convert methods get_previous_scheduled_dagrun and get_previous_dagrun…
vincbeck c9ad169
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck dfc01a8
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 6d35f2f
Fix static checks
vincbeck 920aae6
Convert DagRun.get_task_instance to internal API
vincbeck 9492538
Fix circular dependencies
vincbeck 10638cc
Fix spellcheck
vincbeck 0179ba5
Disable type aliasing in pydantc models
vincbeck 995cab1
Fix unit test
vincbeck 03694b4
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 33640a9
Add custom_operator_name attribute to taskinstance Pydantic model
vincbeck 52dd4b0
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 0f042e9
Fix static checks
vincbeck f463837
Fix unit tests
vincbeck 8cf1fb1
Update reason in context
vincbeck 16c334b
Add annotations to `fetch_callback` method
vincbeck 7192075
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck d29dcd2
Resolve conflicts
vincbeck de30beb
Improve style of `handle_callback` method
vincbeck ed97834
Add default encoder
vincbeck 658be83
Address feedbacks from @uranusjr
vincbeck 62b9ba8
Add reason to context
vincbeck 640c819
Fix unit test
vincbeck 87b4fac
Fix params order
vincbeck 67f2ee6
Convert private functions to kwarg only
vincbeck 9914a36
Fix unit test
vincbeck 6881a51
Fix taskinstance unit tests
vincbeck 073ce8b
Fix pydantic unit tests
vincbeck d32a21e
Apply suggestion by uranujs@
vincbeck 927344a
Address feedbacks from uranusjr@
vincbeck b42390b
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck ea1d9cd
Apply D205 style rule
vincbeck d87346f
Fix get_serialized_dag method
vincbeck d3480a7
Skipping some tests if AIP-44 if disabled
vincbeck 638f149
Remove _get_task_instance function
vincbeck 3249ce4
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 9bb04fe
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 67fd268
Remove get_dagrun function
vincbeck c6cc066
Remove get_task_instances method
vincbeck e782d50
Remove get_task_instance method
vincbeck 9a5e1ab
Fix static checks
vincbeck bdd2370
Revert "Remove get_task_instance method"
vincbeck 4c1d4fa
Revert "Remove get_task_instances method"
vincbeck 7f73bc1
Revert "Remove get_dagrun function"
vincbeck 0ae55e1
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 72d106e
Fix static checks
vincbeck 10a511f
Fix tests
vincbeck c4d385d
Leave annotations
vincbeck c80105b
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck ae79b6b
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 46d2e25
Fix static checks
vincbeck 8a71857
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 4796186
Add type annotation to _stop_remaining_tasks()
vincbeck 9ac5d30
Add type annotation to handle_callback()
vincbeck 8ca799b
Pass `dagrun_id` to fetch_callback()
vincbeck e795915
Pass `dag_run_id` to fetch_task_instance()
vincbeck a14f941
Pass `dag_run_id` to get_previous_scheduled_dagrun()
vincbeck b2d5bc9
Remove `_set_duration` and introduce `set_end_date`
vincbeck 975eba8
Fix `set_end_date`
vincbeck adc10fd
Add TaskInstancePydantic.update_forward_refs()
vincbeck 7921dc5
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck f76bc56
Replace `save_to_db()` to `finish_task()`
vincbeck 4178b7c
Fix static checks
vincbeck 2a3edf1
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 8d35a1e
Fix static checks
vincbeck 2de2375
Cleanup
vincbeck 85743a8
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck 5882674
Fix tests
vincbeck 10183ac
Fix tests
vincbeck c6fb38b
Make pydantic models ignore TCH001 rule
vincbeck 5682a56
Use List instead of list in Pydantic models
vincbeck cbd4757
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck df2ff72
Fix tests
vincbeck 512872d
Add `refresh_from_db` after setting the duration
vincbeck a44ac1a
Merge branch 'main' into vincbeck/execute_callbacks
vincbeck dee5655
Fix import
vincbeck 8c60f21
Revert "Add `refresh_from_db` after setting the duration"
vincbeck 3923f2c
Revert "Cleanup"
vincbeck 989672e
Revert "Replace `save_to_db()` to `finish_task()`"
vincbeck d51371e
Revert "Fix `set_end_date`"
vincbeck 2b9014b
Revert "Remove `_set_duration` and introduce `set_end_date`"
vincbeck c6573c7
Add `dataset` property to `DatasetEventPydantic`
vincbeck 0653a33
Remove assert
vincbeck File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Move get_serialized_dag and get_task_instance methods to model classes
commit ac4880086909d3b8409397b0ef1f104d9720e190
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how does it behave in Internal API?
Does it release the rows when session is destroyed (I expect it) - so just after the results are returned
We may need to think about the scenario when the row-locking is important for potential race conditions or consistency, but I see it's outside of this PR (as we don't use it)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think a good idea would be to split that to two methods. We definitely do not want to have a case where we have lock_for_update=True over the internal API so rather than just leave it here, I think we could probably duplicate that method (with some DRY-internal method query) into: