Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature / Runtime child jobs, parallel and sequential job groups #465

Merged
merged 14 commits into from
Oct 31, 2024
Merged

Feature / Runtime child jobs, parallel and sequential job groups #465

merged 14 commits into from
Oct 31, 2024

Conversation

martin-traverse
Copy link
Contributor

  • Adds the concept of a child job to the runtime execution engine
  • Job groups added to the metadata model for defining child jobs
  • Parallel and sequential job groups are supported
  • Nested job groups are supported
  • Works with dev mode translation
  • Includes an example job sequence for import -> process -> export
  • Only available in the runtime, platform support will come later

Martin Traverse added 14 commits October 29, 2024 10:58
Suggested metadata for handling sequential and parallel job groups
d64a42e
Make graph builder hold stateful information on job resources, namesp…
c9e013d 
…ace etc
Use EJobValidation for errors in the graph builder
a1722ca
Make the graph builder try to continue after it hits an error
ae0693d
Add graph definitions, functions and builder logic to express child jobs
bf6ec54
Engine updates to handle processing for child jobs
a4be613
Update dev mode translator to handle job recursion (resources are sta…
762290d 
…teful, job def is stateless)
Update code that depends on the new dev mode translator
c53bf1f
Create an example import -> process -> export job group
0e26256
Fix one warning in the graph module
56a290a
Add some extra paths to git ignore for example data
179252e
Add the import -> process -> export group job to the CI job for examp…
4601905 
…le models
Minor fixes and tidy-ups
7a7fa49
Remove an unneeded code change
f892a07
@martin-traverse martin-traverse merged commit 0cc91e9 into finos:main Oct 31, 2024
41 checks passed
@martin-traverse martin-traverse deleted the feature/job_groups branch October 31, 2024 08:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@martin-traverse