Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[UA] Tight worker loop can cause high CPU usage #60950

Merged
Next Next commit
Addded worker padding to save some CPU
  • Loading branch information
jloleysens committed Mar 23, 2020
commit 865bbc7991c1858767291bd6fe28d0ccea3b3628
13 changes: 9 additions & 4 deletions x-pack/plugins/upgrade_assistant/server/lib/reindexing/worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ const POLL_INTERVAL = 30000;
// If no nodes have been able to update this index in 2 minutes (due to missing credentials), set to paused.
const PAUSE_WINDOW = POLL_INTERVAL * 4;

const WORKER_PADDING_MS = 500;

/**
* A singleton worker that will coordinate two polling loops:
* (1) A longer loop that polls for reindex operations that are in progress. If any are found, loop (2) is started.
Expand Down Expand Up @@ -111,6 +113,9 @@ export class ReindexWorker {

// Push each operation through the state machine and refresh.
await Promise.all(this.inProgressOps.map(this.processNextStep));
// TODO: This tight loop needs something to relax potentially high CPU demands so this padding is added.
// This scheduler should probably be revisited or removed in future.
await new Promise(res => setTimeout(res, WORKER_PADDING_MS));
jloleysens marked this conversation as resolved.
Show resolved Hide resolved
await this.refresh();
}
} finally {
Expand Down Expand Up @@ -173,20 +178,20 @@ export class ReindexWorker {
}
};

private processNextStep = async (reindexOp: ReindexSavedObject) => {
private processNextStep = async (reindexOp: ReindexSavedObject): void => {
const credential = this.credentialStore.get(reindexOp);

if (!credential) {
// Set to paused state if the job hasn't been updated in PAUSE_WINDOW.
// This indicates that no Kibana nodes currently have credentials to update this job.
const now = moment();
const updatedAt = moment(reindexOp.updated_at);
if (updatedAt < now.subtract(PAUSE_WINDOW)) {
return this.reindexService.pauseReindexOperation(reindexOp.attributes.indexName);
await this.reindexService.pauseReindexOperation(reindexOp.attributes.indexName);
return;
} else {
// If it has been updated recently, we assume another node has the necessary credentials,
// and this becomes a noop.
return reindexOp;
return;
}
}

Expand Down