Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Various Task Manage Speedups #373

Merged
merged 10 commits into from
Mar 25, 2020
Merged

Various Task Manage Speedups #373

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
51d5c34
Pipeline: do not add children of tasks already added
nh13 Mar 18, 2020
a472372
Pipeline: solve stack overflow in addChildren
nh13 Mar 18, 2020
789fdfb
Task: find cycles in a collection of tasks all at once.
nh13 Mar 18, 2020
88a8ddd
small performance optimizations on dense dags
nh13 Mar 18, 2020
708284a
TaskManager: exponential reduction in sleep time if we can do anything,
nh13 Mar 18, 2020
5d0206d
No need to check for cycles before executing a task, as it was checked
nh13 Mar 18, 2020
3e95914
Warn if a single step in execution takes longer than 30 seconds
nh13 Mar 18, 2020
1e24919
Optimize GraphNode
nh13 Mar 18, 2020
c3dc024
update based on review
nh13 Mar 18, 2020
e0a268c
ResourceSet bug when subsetting resources and some performance optimi…
nh13 Mar 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions core/src/main/scala/dagr/core/tasksystem/Pipeline.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,10 @@ abstract class Pipeline(val outputDirectory: Option[Path] = None,

/** Recursively navigates dependencies, starting from the supplied task, and add all children to this.tasks. */
private def addChildren(task : Task) : Unit = {
tasks ++= task.tasksDependingOnThisTask
task.tasksDependingOnThisTask.foreach(addChildren)
task.tasksDependingOnThisTask.filterNot(tasks.contains).foreach { child =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than doing a filterNot here, I think you could write this as:

task.tasksDependingOnThisTask.foreach { child =>
  if (tasks.add(child)) addChildren(child)
}

The bonus being that for tasks that have not been previously added you're not checking the set twice.

tasks += child
addChildren(child)
}
}

/** True if we this pipeline is tracking this direct ancestor task, false otherwise. */
Expand Down