[kbn/optimizer] allocate workers less aggressively

[spalger]

In an attempt to avoid the optimizer overwhelming computers by launching tons of workers this update to the optimizer modifies two things:

1. The worker assignment algorithm will attempt to limit the number of modules assigned to each worker, possibly creating more worker assignments than the max number of workers
2. If there are more worker assignments created than the maximum number of workers then only N workers will be run at a time, as soon as a worker is no longer active (complete in standard mode, idle in watch mode) another worker will be started.

To experiment with these parameter you can use the --workers and --worker-size flags to the node scripts/build_kibana_platform_plugins CLI. Check --help for more info.

To customize these values with yarn start you can specify the KBN_OPTIMIZER_MAX_ACTIVE_WORKERS and KBN_OPTIMIZER_SOFT_MAX_MODULES_PER_WORKER environment variables.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[spalger]

Would love to collect feedback from people about how this runs on their machines before seeking review for merging. Try using different values for KBN_OPTIMIZER_MAX_ACTIVE_WORKERS and KBN_OPTIMIZER_SOFT_MAX_MODULES_PER_WORKER when you run yarn start

cc: @andrew-goldstein @lizozom @timroes @mistic

[spalger]

@elasticmachine merge upstream

[spalger]

@elasticmachine merge upstream

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 7ca8195
• Pipeline Steps (look for red circles / failed steps)
• Interpreting CI Failures

Failed CI Steps

• Build X-Pack Kibana
• Build OSS/Default Kibana
• Execute kibana-intake

Build metrics

‼️ metrics were not reported for [#67735@7ca8195]

History

• 💚 Build #50793 succeeded c57b627
• 💚 Build #50617 succeeded ee12c7c
• 💔 Build #50507 failed 6a4b0ab
• 💔 Build #50498 failed ebe81df

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[spalger]

I haven't seen any evidence that this makes the optimizer more efficient, though we might revisit it again later.