Skip to content

Commit

Permalink
feat(core): improved task graph concurrency
Browse files Browse the repository at this point in the history
This commit adds batch-based concurrency to the task graph.

When TaskGraph's process method is called, the requested tasks are
grouped into batches that share one or more keys (including
dependencies). These batches are then queued.

In each iteration of the task graph's main loop, any pending batches
that share no keys with currently in-progress batches are added to the
graph.

This enables e.g. hot reload tasks to be run concurrently with build and
test tasks for their underlying modules, which was one of the primary
motivators behind this change.

Also replaced the concurrencyLimit option for TaskGraph's processTasks
method with an unlimitedConcurrency option. This is useful e.g. when
resolving providers.

Task nodes from batches with unlimited concurrency are processed
regardless of normally available task graph concurrency.
  • Loading branch information
thsig committed Feb 10, 2020
1 parent 6c34af4 commit 1a2f69a
Show file tree
Hide file tree
Showing 8 changed files with 621 additions and 168 deletions.
4 changes: 4 additions & 0 deletions garden-service/src/events.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ export type Events = {
configRemoved: {
path: string
}
internalError: {
timestamp: Date
error: Error
}
projectConfigChanged: {}
moduleConfigChanged: {
names: string[]
Expand Down
4 changes: 1 addition & 3 deletions garden-service/src/garden.ts
Original file line number Diff line number Diff line change
Expand Up @@ -506,9 +506,7 @@ export class Garden {
})

// Process as many providers in parallel as possible
const taskResults = await this.processTasks(tasks, {
concurrencyLimit: tasks.length,
})
const taskResults = await this.processTasks(tasks, { unlimitedConcurrency: true })

const failed = Object.values(taskResults).filter((r) => r && r.error)

Expand Down
Loading

0 comments on commit 1a2f69a

Please sign in to comment.