fix(CSI-224,WEKAPP-417375): race condition on multiple volume deletion in parallel

[sergeyberezansky]

TL;DR

Refactored garbage collection initiation and handling in the WekaFS package for better traceability, logging, and performance.

What changed?

• Introduced OpenTelemetry tracing to GC functions.
• Improved logging for GC initiation and directory deletion processes.
• Replaced synchronous volume purge with move to trash followed by async GC initiation.
• Enhanced directory deletion using worker pool mechanism to limit concurrent deletions.
• Updated variable names and function names for better clarity.

How to test?

• Run unit tests available for the wekafs package.
• Perform integration testing by deploying in staging environment and monitor logs and traces for GC operations.
• Create dozens of PVCs and load them with thousands of files. Delete all PVCs at once, follow up on garbage collection process

Why make this change?

• Improved traceability and debugging with OpenTelemetry spans and enriched logging.
• Increased performance during directory deletions using a worker pool mechanism.
• Cleaned up code for better readability and maintainability.

[sergeyberezansky]

• fix(CSI-224,WEKAPP-417375): race condition on multiple volume deletion in parallel #286 : 3 dependent PRs (#289 , #292 , #294 ) 👈
• fix(CSI-234): leak of wekafs mounts due to bad IsMounted() check #293
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @sergeyberezansky and the rest of your teammates on Graphite

[graphite-app]

Graphite Automations

"Request reviewers once CI passes" took an action on this PR • (07/23/24)

1 reviewer was added to this PR based on Sergey Berezansky's automation.

[sergeyberezansky]

Merge activity

• Jul 31, 6:18 AM EDT: @sergeyberezansky started a stack merge that includes this pull request via Graphite.
• Jul 31, 6:18 AM EDT: @sergeyberezansky merged this pull request with Graphite.