Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AIP-44 decouple BaseJob ORM from Job logic #30294

Closed
potiuk opened this issue Mar 25, 2023 · 0 comments · Fixed by #30255
Closed

AIP-44 decouple BaseJob ORM from Job logic #30294

potiuk opened this issue Mar 25, 2023 · 0 comments · Fixed by #30255
Assignees
@potiuk
Labels
AIP-44 Airflow Internal API

Comments

@potiuk
Copy link
Member

potiuk commented Mar 25, 2023 •
edited
Loading

We should decouple the logic of jobs from the ORM models, so that we can run LocalTaskJob, TriggererJob, DagProcessorJob from the database.
@potiuk potiuk converted this from a draft issue Mar 25, 2023
@potiuk potiuk moved this from 🔖 Ready to 👀 In review in AIP-44 - Internal API Mar 25, 2023
@potiuk potiuk linked a pull request Mar 25, 2023 that will close this issue
Decouple "job runner" from BaseJob ORM model #30255
Merged
@potiuk potiuk mentioned this issue Mar 25, 2023
Merged
@potiuk potiuk added the AIP-44 Airflow Internal API label Mar 25, 2023
@potiuk potiuk self-assigned this Mar 25, 2023
potiuk added a commit to potiuk/airflow that referenced this issue Apr 9, 2023
@potiuk
Decouple "job runner" from BaseJob ORM model
bfe2465 
Originally BaseJob ORM model was extended and Polymorphism has been
used to tie different execution logic to different job types. This
has proven to be difficult to handle during AIP-44 implementation
(internal API) because LocalTaskJob, DagProcessorJob and TriggererJob
are all going to not use the ORM BaseJob model, but they should use
BaseJobPydantic instead. In order to make it possible, we introduce
a new type of object BaseJobRunner and make BaseJob use the runners
instead.

This way, the BaseJobRunners are used for the logic of each of the
job, where single, non-polimorphic BaseJob is used to keep the
records in the database - as a follow up it will allow to completely
decouple the job database operations and move it to internal_api
component when db-lesss mode is enabled.

Closes: apache#30294
@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in AIP-44 - Internal API Apr 10, 2023
potiuk added a commit that referenced this issue Apr 10, 2023
@potiuk @uranusjr
Decouple "job runner" from BaseJob ORM model (#30255)
58294c5 
* Decouple "job runner" from BaseJob ORM model

Originally BaseJob ORM model was extended and Polymorphism has been
used to tie different execution logic to different job types. This
has proven to be difficult to handle during AIP-44 implementation
(internal API) because LocalTaskJob, DagProcessorJob and TriggererJob
are all going to not use the ORM BaseJob model, but they should use
BaseJobPydantic instead. In order to make it possible, we introduce
a new type of object BaseJobRunner and make BaseJob use the runners
instead.

This way, the BaseJobRunners are used for the logic of each of the
job, where single, non-polimorphic BaseJob is used to keep the
records in the database - as a follow up it will allow to completely
decouple the job database operations and move it to internal_api
component when db-lesss mode is enabled.

Closes: #30294

Co-authored-by: Tzu-ping Chung <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@potiuk potiuk

Labels
AIP-44 Airflow Internal API
Projects
No open projects
AIP-44 - Internal API
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Decouple "job runner" from BaseJob ORM model potiuk/airflow
1 participant
@potiuk