Surface "could not get secrets" (and others) as errors

[alecthomas]

Change the retry code for pulling ModuleContext such that it differentiates between retryable errors and fatal errors. The easiest solution is probably to use gRPC status codes to differentiate.

debug:controller0: Streaming RPC failed: internal: could not get secrets: did_web_portable_did: no keychain entry for "did_web_portable_did": not found: /xyz.block.ftl.v1.VerbService/GetModuleContext

Currently they are at debug level and the plugin just fails to start with no message.

[jonathanj-square]

Based on our conversations there appears to be 2 goals for resolving this issue:

• provide error visibility
• terminate the runner if the error is deemed fatal (to avoid log spam)

What I am seeing is that when the server is successfully terminated the controller will attempt to respawn the server as the controller aims to reach the target replica count. Naturally this complicates the task of reaching the second goal without impacting deployment resilience.