Controller thinks old runners are still active

[matt2e]

Slack discussion: https://sq-tbd.slack.com/archives/C04P86U1B4N/p1720578460763129?thread_ts=1720578445.888969&cid=C04P86U1B4N

Panic should be fixed here because of this issue: #2045
But the underlying reason we are getting an error is not

[matt2e]

Possibly related panic:

ftl-runner
panic: sync: negative WaitGroup counter
ftl-runner
{"level":"error","attributes":{"module":"kcc","scope":"kcc","deployment":"dpl-kcc-11u5wsvaxpo4g8es"},"time":"2024-07-10T00:22:29.665804531Z"}
ftl-runner
goroutine 10 [running]:
ftl-runner
sync.(*WaitGroup).Add(0x126d180?, 0xc0001e65c0?)
ftl-runner
/home/runner/.cache/hermit/pkg/go-1.22.2/src/sync/waitgroup.go:62 +0xd8
ftl-runner
sync.(*WaitGroup).Done(...)
ftl-runner
/home/runner/.cache/hermit/pkg/go-1.22.2/src/sync/waitgroup.go:87
ftl-runner
github.com/TBD54566975/ftl/internal/modulecontext.NewDynamicContext.func2({0x126d180, 0xc0001e65c0})
ftl-runner
/home/runner/go/pkg/mod/github.com/!t!b!d54566975/[email protected]/internal/modulecontext/module_context.go:242 +0x8f
ftl-runner
github.com/TBD54566975/ftl/internal/rpc.RetryStreamingServerStream[...]({0x127c930, 0xc0002278b0}, {0x0, 0x0, 0x0, 0x0, 0x0}, 0xc0001e6080, 0xc000011950, 0xc000286680, ...)
ftl-runner
/home/runner/go/pkg/mod/github.com/!t!b!d54566975/[email protected]/internal/rpc/rpc.go:252 +0x262
ftl-runner
created by github.com/TBD54566975/ftl/internal/modulecontext.grpcModuleContextSupplier.Subscribe in goroutine 1
ftl-runner
/home/runner/go/pkg/mod/github.com/!t!b!d54566975/[email protected]/internal/modulecontext/module_context.go:206 +0x1c5

[matt2e]

Which should have been fixed here? a8bc627

[matt2e]

Closing this as these logs all point to things which have now been fixed. Will re-open if it comes back