AIP-44 Remove job_runner from BaseJob as a final step of decoupling #30325
Assignees
Labels
AIP-44
Airflow Internal API
Comments
potiuk
changed the title
potiuk
added a commit
to potiuk/airflow
that referenced
this issue
Apr 11, 2023
This is the final step of decoupling of the job runner from ORM based BaseJob. After this change, finally we rich the state that the BaseJob is just a state of the Job being run, but all the logic is kept in separate "JobRunner" entity which just keeps the reference to the job. Also it makes sure that job in each runner is defined as appropriate for each job type: * SchedulerJobRunner, BackfillJobRunner can only use BaseJob * DagProcessorJobRunner, TriggererJobRunner and especially the LocalTaskJobRunner can keep both BaseJob and it's Pydantic BaseJobPydantic representation - for AIP-44 usage. The highlights of this change: * Job does not have job_runner reference any more * Job is a mandatory parameter when creating each JobRunner * run_job method takes as parameter the job (i.e. where the state of the job is called) and executor_callable - i.e. the method to run when the job gets executed * heartbeat callback is also passed a generic callable in order to execute the post-heartbeat operation of each of the job type * there is no more need to specify job_type when you create BaseJob, the job gets its type by a simply creating a runner with the job This is the final stage of refactoring that was split into reviewable stages: apache#30255 -> apache#30302 -> apache#30308 -> this PR. Closes: apache#30325
github-project-automation
bot
moved this from 🏗 In progress
to ✅ Done
in AIP-44 - Internal API
potiuk
added a commit
that referenced
this issue
Apr 11, 2023
This is the final step of decoupling of the job runner from ORM based BaseJob. After this change, finally we rich the state that the BaseJob is just a state of the Job being run, but all the logic is kept in separate "JobRunner" entity which just keeps the reference to the job. Also it makes sure that job in each runner is defined as appropriate for each job type: * SchedulerJobRunner, BackfillJobRunner can only use BaseJob * DagProcessorJobRunner, TriggererJobRunner and especially the LocalTaskJobRunner can keep both BaseJob and it's Pydantic BaseJobPydantic representation - for AIP-44 usage. The highlights of this change: * Job does not have job_runner reference any more * Job is a mandatory parameter when creating each JobRunner * run_job method takes as parameter the job (i.e. where the state of the job is called) and executor_callable - i.e. the method to run when the job gets executed * heartbeat callback is also passed a generic callable in order to execute the post-heartbeat operation of each of the job type * there is no more need to specify job_type when you create BaseJob, the job gets its type by a simply creating a runner with the job This is the final stage of refactoring that was split into reviewable stages: #30255 -> #30302 -> #30308 -> this PR. Closes: #30325
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As discussed in #30255 (comment) and #30255 (comment) - I think eventually (after #30308 especially, we should be able to get rid of the job_runner back-reference to the job_runner in the BaseJob - this will simplify the relation.
It will be easier however to do it in those steps we have now (each of those changes is relatively small (in terms of base code) and separately reviewable - and we can make sure Airlfow continues to work after each of those steps which makes it easier to implement and merge those changes faster):
#30255 -> #30302 -> #30308 -> this one
CC: @uranusjr @mhenc
The text was updated successfully, but these errors were encountered: