[GCS] Make Gcs-based actor scheduler's bookkeeping consistent

[Chong-Li]

Why are these changes needed?

In the current version of gcs-based actor scheduler, there is an inconsistency: the destroyed actor forgets
to return resources back to gcs_resource_manager. See #16580 (comment) for details @iycheng . This PR fixes this error.

Note, this is just a temporary workaround. When we take the multi-threaded Java workers into account (in the next PR), the whole GcsActorWorkerAssignment handling will be changed.

Related issue number

#16580

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[fishbone]

@Chong-Li could you add one unit test to make sure lease - release is working correctly?

[fishbone]

Btw, if we don't have resources temporarily, will it be rescheduled later when there is enough resources?

[Chong-Li]

Btw, if we don't have resources temporarily, will it be rescheduled later when there is enough resources?

For the current version, SchedulePendingActors() only reacts to NodeAdd or ReportResourceUsage. I was planning to
make it more complete in the next PR with job distribution bookkeeping enabled. But to make the current version self-contained, I just (temporarily) updated the release_resources lambda function. It may be not efficient because of over reaction.

[fishbone]

LG! Thanks for the fixing! I'm OK with this one. But before merging, could you rebase to trunk? Recently we got some implicit merge conflict which break the trunk and it just got fixed.

[rkooo567]

tests:test_client_reconnect seems to be flaky in the master