Cronjob panic in production during module deployment

[matt2e]

Stacktrace

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x13a8145]
goroutine 22591138 [running]:
github.com/TBD54566975/ftl/backend/controller.(*Service).executeAsyncCalls(0xc000861a00, {0x1ccca78, 0xc00055aa20})
/src/backend/controller/controller.go:1407 +0x985
github.com/TBD54566975/ftl/backend/controller.(*Service).AsyncCallWasAdded.func1()
/src/backend/controller/controller.go:1388 +0x2d
created by github.com/TBD54566975/ftl/backend/controller.(*Service).AsyncCallWasAdded in goroutine 22591109
/src/backend/controller/controller.go:1387 +0x74

This resulted in logs like this one: #2250
Possibly a follow on issue

[matt2e]

After the first occurence, I restarted the runners and it seemed to fix the logs about runner state.

It happened again two hours later though, but has not happened since.

Second stack trace:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x13a8145]
goroutine 356867 [running]:
github.com/TBD54566975/ftl/backend/controller.(*Service).executeAsyncCalls(0xc0005ce000, {0x1ccca78, 0xc000b0e510})
/src/backend/controller/controller.go:1407 +0x985
github.com/TBD54566975/ftl/backend/controller/scheduledtask.(*Scheduler).run.func2()
/src/backend/controller/scheduledtask/scheduledtask.go:148 +0x55
created by github.com/TBD54566975/ftl/backend/controller/scheduledtask.(*Scheduler).run in goroutine 41
/src/backend/controller/scheduledtask/scheduledtask.go:147 +0xfdb

[matt2e]

This looks to be already solved in main.
The issue was this:

call, leaseCtx, err := s.dal.AcquireAsyncCall(ctx)
if errors.Is(err, dalerrs.ErrNotFound) {
	...
} else if err != nil {
	<observability code that accessed call variable>
}