fix: suppress noisy log spam when scaling runners #537

alecthomas · 2023-11-02T04:09:34Z

Also added --recreate flag to ftl serve.

Fixes #534

Before:

info:reconcileRunners: Removing runner: R00000000000000000000002000
warn:runner0: Stream handler failed, retrying in 100ms: context canceled
warn:runner0: Stream handler failed, retrying in 100ms: context canceled
error:reconcileRunners: Error starting runner: context canceled: context canceled
error:controller0: Streaming RPC failed: invalid_argument: protocol error: incomplete envelope: stream error: stream ID 3; CANCEL: /xyz.block.ftl.v1.ControllerService/RegisterRunner

After:

info:reconcileRunners: Removing runner: R00000000000000000000002000
error:controller0: Streaming RPC failed: invalid_argument: protocol error: incomplete envelope: stream error: stream ID 9; CANCEL: /xyz.block.ftl.v1.ControllerService/StreamDeploymentLogs

There's still one error, but it seems more difficult to fix this one, and it seems more intermittent too.

Fixes #534

wesbillman · 2023-11-02T16:57:41Z

cmd/ftl/cmd_serve.go

-	} else {
-		err = exec.Command(ctx, logger.GetLevel(), ".", "ftl-initdb", dsnFlag).Run()
-	}
+	_, err = databasetesting.CreateForDevel(ctx, dsn, recreate)


Thank you! :)

fix: suppress irrelevant log spam when scaling runners

f56377b

Fixes #534

alecthomas requested review from dhanji and wesbillman as code owners November 2, 2023 04:09

alecthomas changed the title ~~fix: suppress irrelevant log spam when scaling runners~~ fix: suppress noisy log spam when scaling runners Nov 2, 2023

alecthomas merged commit 6584788 into main Nov 2, 2023
8 checks passed

alecthomas deleted the aat/quieter-scaling branch November 2, 2023 04:21

wesbillman reviewed Nov 2, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: suppress noisy log spam when scaling runners #537

fix: suppress noisy log spam when scaling runners #537

alecthomas commented Nov 2, 2023 •

edited

Loading

wesbillman Nov 2, 2023

fix: suppress noisy log spam when scaling runners #537

fix: suppress noisy log spam when scaling runners #537

Conversation

alecthomas commented Nov 2, 2023 • edited Loading

wesbillman Nov 2, 2023

Choose a reason for hiding this comment

alecthomas commented Nov 2, 2023 •

edited

Loading