Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: worker pool that can execute tasks on all workers #32120

Merged
merged 13 commits into from
Jun 29, 2021

Conversation

pieh
Copy link
Contributor

@pieh pieh commented Jun 25, 2021

Description

This swaps jest-worker with our own worker pool that will be adding features we need for PQR. Initially it has feature parity with jest-worker (for the features we do use) + ability to execute certain "tasks" on all workers (needed for bootstrapping workers for PQR - jest-worker setup was too limiting for us - we need greater control).

In future PRs there will be added support for messaging between main and workers.

Perf

Not hugely impactful for us, but I wanted to make sure worker pool performance wouldn't regress I used https://github.com/facebook/jest/tree/master/packages/jest-worker/src/__performance_tests__ and adjusted it a bit (https://gist.github.com/pieh/9c3f6dbdd3acafa5a8811418eca4d356) and compared gatsby-worker against [email protected] (latest as of writing of this comment - note that we currently use older version).

Task that does nothing 100000 times
➜  __performance_tests__ git:(pqr/allow-execute-fn-on-all-workers) ✗ node --expose-gc test.js empty 100000                          
---------------------------------------------------------------------------
jest-worker: { globalTime: 1998.6634929180145, processingTime: 1963.4644238948822 }
gatsby-worker: { globalTime: 1699.7716178894043, processingTime: 1680.174674987793 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2131.742655992508, processingTime: 2099.079083919525 }
gatsby-worker: { globalTime: 1667.6132881641388, processingTime: 1639.6259500980377 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2059.533210992813, processingTime: 2031.5315339565277 }
gatsby-worker: { globalTime: 1617.8644468784332, processingTime: 1593.7371819019318 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 1988.8901131153107, processingTime: 1961.373533964157 }
gatsby-worker: { globalTime: 1653.4446740150452, processingTime: 1631.8277361392975 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2129.582818031311, processingTime: 2103.122942209244 }
gatsby-worker: { globalTime: 1700.000137090683, processingTime: 1678.7925159931183 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2058.744973897934, processingTime: 2031.553738117218 }
gatsby-worker: { globalTime: 1638.2301177978516, processingTime: 1613.9939517974854 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 1980.671779870987, processingTime: 1953.9795258045197 }
gatsby-worker: { globalTime: 1631.1066999435425, processingTime: 1608.66081905365 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 1998.1074059009552, processingTime: 1973.3088948726654 }
gatsby-worker: { globalTime: 1648.0316200256348, processingTime: 1622.8740689754486 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2051.6042828559875, processingTime: 2021.7381830215454 }
gatsby-worker: { globalTime: 1632.7712378501892, processingTime: 1612.0683648586273 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2149.560436964035, processingTime: 2121.764895915985 }
gatsby-worker: { globalTime: 1654.3145129680634, processingTime: 1633.308058977127 }
---------------------------------------------------------------------------
total gatsby-worker: { gWGT: 16543.148352622986, gWPT: 16315.063322782516 }
total jest-worker: { jWGT: 20547.101170539856, jWPT: 20260.91675567627 }
---------------------------------------------------------------------------
global time: gatsby-worker over 100000 calls was faster by % 19.486704156874847
processing time: gatsby-worker over 100000 calls was faster by % 19.47519690484039
Task that calculate PI 10000 times in workers
➜  __performance_tests__ git:(pqr/allow-execute-fn-on-all-workers) ✗ node --expose-gc test.js loadTest 10000                        
---------------------------------------------------------------------------
jest-worker: { globalTime: 961.0926420688629, processingTime: 922.5083830356598 }
gatsby-worker: { globalTime: 777.6850469112396, processingTime: 758.6562309265137 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 854.1666779518127, processingTime: 825.3946840763092 }
gatsby-worker: { globalTime: 695.6478898525238, processingTime: 676.7463660240173 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 729.9372169971466, processingTime: 707.9941308498383 }
gatsby-worker: { globalTime: 662.4068939685822, processingTime: 642.3461120128632 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 726.789785861969, processingTime: 701.6343488693237 }
gatsby-worker: { globalTime: 890.4017338752747, processingTime: 865.6464939117432 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 723.6333639621735, processingTime: 699.9333298206329 }
gatsby-worker: { globalTime: 651.1934118270874, processingTime: 630.4632089138031 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 818.4343338012695, processingTime: 795.8016359806061 }
gatsby-worker: { globalTime: 669.1646339893341, processingTime: 645.9727358818054 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 712.160159111023, processingTime: 686.5256531238556 }
gatsby-worker: { globalTime: 654.4234750270844, processingTime: 635.5800518989563 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 757.0051820278168, processingTime: 731.649176120758 }
gatsby-worker: { globalTime: 676.8136041164398, processingTime: 653.058874130249 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 731.8604440689087, processingTime: 706.2469630241394 }
gatsby-worker: { globalTime: 742.6872208118439, processingTime: 724.3149909973145 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 716.2216629981995, processingTime: 693.820986032486 }
gatsby-worker: { globalTime: 731.365779876709, processingTime: 708.1000800132751 }
---------------------------------------------------------------------------
total gatsby-worker: { gWGT: 7151.789690256119, gWPT: 6940.885144710541 }
total jest-worker: { jWGT: 7731.301468849182, jWPT: 7471.509290933609 }
---------------------------------------------------------------------------
global time: gatsby-worker over 10000 calls was faster by % 7.495656209087455
processing time: gatsby-worker over 10000 calls was faster by % 7.10196729417121

With example benchmarks gatsby-worker seems a bit faster (~7% when calculating PI 10000 times and ~19% when doing no-op task 100000 times), but realistically we won't see impact of it in real life as our task are really heavy and we usually will have much less tasks than 10000 (so any worker pool overhead is dwarfed by actual tasks).

Documentation

In new gatsby-worker utility package README.md

Related Issues

[ch33219]

@pieh pieh added the topic: core Relates to Gatsby's core (e.g. page loading, reporter, state machine) label Jun 25, 2021
@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jun 25, 2021
@pieh pieh removed the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jun 25, 2021
@pieh pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from 845a318 to 4d4272a Compare June 25, 2021 15:13
@pieh pieh marked this pull request as ready for review June 25, 2021 15:31
@pieh pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from 4d4272a to bf4c08b Compare June 28, 2021 09:24
@pieh pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from bf4c08b to ad42d34 Compare June 28, 2021 09:28
@pieh pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from ad42d34 to 6aff32a Compare June 28, 2021 11:10
Comment on lines +131 to +133
expect(returnValue).toBeArrayOfSize(numWorkers)
// turns sync function into async
expect(returnValue).toSatisfyAll(isPromise)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat!

packages/gatsby-worker/src/child.ts Outdated Show resolved Hide resolved
packages/gatsby-worker/src/child.ts Outdated Show resolved Hide resolved
{
"name": "gatsby-worker",
"description": "Utility to create worker pools",
"version": "0.0.0-next.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll want to start this at 1.0.0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Early on I would prefer to keep it <1 as we might quickly introduce breaking changes as we start using worker pool and discover new needs or APIs (that's what <1 is for). 1 is when API becomes actually stable enough

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious how this will play with lerna minor bump - I think it should do 0.1.0-next.0 for the next branch cut? If so, this sounds good to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/gatsbyjs/gatsby/commits/master/packages/gatsby-plugin-graphql-config/package.json is example of 0.X.Y package we have and it does bump minor for each release. We want gatsby-worker to be actual dep only of gatsby core package - if we will need it in gatsby-cli for structured logs, we will import it from gatsby/internal there. Messaging (but this will only need messaging and not queueing up tasks)

packages/gatsby-worker/src/index.ts Show resolved Hide resolved
packages/gatsby-worker/src/index.ts Outdated Show resolved Hide resolved
packages/gatsby-worker/src/index.ts Outdated Show resolved Hide resolved
packages/gatsby-worker/src/task-queue.ts Show resolved Hide resolved
packages/gatsby-worker/tsconfig.json Outdated Show resolved Hide resolved
@pieh pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from 16aab93 to e48e671 Compare June 29, 2021 09:47
Copy link
Contributor

@vladar vladar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we must also replace this usage of JEST_WORKER_ID:

process.env.FORCE_TEST_DATABASE_ID ?? process.env.JEST_WORKER_ID

Comment on lines +45 to +46
// if node is not the head then it will have .prev
taskNode.prev!.next = taskNode.next
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also re-assign taskNode.next.prev? Is it possible that it will retain the task through this link?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are correct - damn basic data structures ;)

Copy link
Contributor Author

@pieh pieh Jun 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in ca74834

@pieh pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from c1ff0b5 to 2bf3fac Compare June 29, 2021 09:59
@pieh pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from 14741b9 to ca74834 Compare June 29, 2021 10:24
@pieh
Copy link
Contributor Author

pieh commented Jun 29, 2021

I think we must also replace this usage of JEST_WORKER_ID:

process.env.FORCE_TEST_DATABASE_ID ?? process.env.JEST_WORKER_ID

Similar as #32120 (comment) - we actually use JEST_WORKER_ID only in test scenarios - and there we need it so parallel running tests don't try to use same db, so JEST_WORKER_ID controlling it is correct (even tho quite confusing indeed - I will add some inline comment about usage of it)

Copy link
Contributor

@LekoArts LekoArts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Contributor

@vladar vladar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really great stuff 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: core Relates to Gatsby's core (e.g. page loading, reporter, state machine)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants