Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shard lease did not release held leases cleanly when downed #7379

Closed
Arkatufus opened this issue Nov 6, 2024 · 1 comment · Fixed by #7383
Closed

Shard lease did not release held leases cleanly when downed #7379

Arkatufus opened this issue Nov 6, 2024 · 1 comment · Fixed by #7383
Labels
akka-cluster-sharding akka-coordination Akka.Coordination package updates and materials potential bug
Milestone

Comments

@Arkatufus
Copy link
Contributor

Version Information
Version of Akka.NET? 1.5.30
Which Akka.NET Modules? Akka.Sharding, Akka.Coordination

Describe the bug
Shard lease did not release held leases cleanly when downed, causing the same shard to fail to acquire the same lease when it is restarted.

This happens both in the Kubernetes and Azure Lease implementation, leaving a strong impression that Akka.Sharding Shard did not release the Lease it held properly when downed.

Kubernetes logs:
image

Azure logs:
image

The difference between k8s and Az is that, using k8s lease, the problem apparently heals itself quite fast while it lingers in Az.

Difference between k8s and az defaults:

akka.coordination.lease.kubernetes.lease-operation-timeout = 5s
akka.coordination.lease.azure.lease-operation-timeout = 15s
akka.coordination.lease.kubernetes.api-service-request-timeout = 2s
akka.coordination.lease.azure.api-service-request-timeout = 6s
@Aaronontheweb Aaronontheweb added akka-cluster-sharding akka-coordination Akka.Coordination package updates and materials labels Nov 6, 2024
@Aaronontheweb Aaronontheweb added this to the 1.5.31 milestone Nov 6, 2024
@Arkatufus
Copy link
Contributor Author

During termination, lease release operation was not being awaited properly, so there is a chance that a lease might not be released properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
akka-cluster-sharding akka-coordination Akka.Coordination package updates and materials potential bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants