Actor deactivate errors causing app to crash

[MrMint]

Expected Behavior

The lack of an activated actor to deactivate should not cause the app to crash. I would also expect any errors in the onDeactivateInteral call to not cause the app to crash.

js-sdk/src/actors/runtime/ActorManager.ts

Line 85 in 5cefcf1

await actor.onDeactivateInternal();

Actual Behavior

We are seeing a situation where our actor services go into a crash backoff loop which is fueled by errors during deactivate. It would appear that daprd attempts to call deactivate for an actor that does not exist in the service, which results in the service crashing due to this being thrown:

js-sdk/src/actors/runtime/ActorManager.ts

Lines 76 to 81 in 5cefcf1

	throw new Error(
	JSON.stringify({
	error: "ACTOR_NOT_ACTIVATED",
	errorMsg: `The actor ${actorId.getId()} was not activated`,
	}),
	);

Once it has crashed, k8s will go into a back-off restart on the container, which is basically endless as any further deactivate calls will also fail given the app has crashed/restarted and lost all the actor references.

Steps to Reproduce the Problem

Send a DELETE request for an actor with an ID that does not exist. This will crash the app.

daprd logs showing the described actual from it's pov:

{"app_id":"actors","level":"debug","msg":"Deactivated actor 'redact||redact'","scope":"dapr.runtime.actor","type":"log","ver":"1.14.1"}
{"app_id":"actors","level":"debug","msg":"Deactivated actor 'redact||redact'","scope":"dapr.runtime.actor","type":"log","ver":"1.14.1"}
{"app_id":"actors","level":"error","msg":"Failed to deactivate actor redact||redact: Delete \"http://127.0.0.1:3000/actors/redact/redact\": EOF","scope":"dapr.runtime.actor","type":"log","ver":"1.14.1"}
{"app_id":"actors","level":"error","msg":"Error performing request: Get \"http://127.0.0.1:3000/healthz\": dial tcp 127.0.0.1:3000: connect: connection refused","scope":"actorshealth","type":"log","ver":"1.14.1"}
{"app_id":"actors","level":"error","msg":"Failed to deactivate actor redact||redact: Delete \"http://127.0.0.1:3000/actors/redact/redact\": dial tcp 127.0.0.1:3000: connect: connection refused","scope":"dapr.runtime.actor","type":"log","ver":"1.14.1"}
{"app_id":"actors","level":"error","msg":"Failed to deactivate actor redact||redact: Delete \"http://127.0.0.1:3000/actors/redact/redact\": dial tcp 127.0.0.1:3000: connect: connection refused","scope":"dapr.runtime.actor","type":"log","ver":"1.14.1"}
{"app_id":"actors","level":"error","msg":"Failed to deactivate actor redact||redact: Delete \"http://127.0.0.1:3000/actors/redact/redact\": dial tcp 127.0.0.1:3000: connect: connection refused","scope":"dapr.runtime.actor","type":"log","ver":"1.14.1"}

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[dapr-bot]

This issue has been automatically closed because it has not had activity in the last 67 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.