Goroutine leak in expiration manager

[viveklak]

Describe the bug
Recently upgraded to 0.11.x vault from 0.9.x. Noticed a steep/monotonically increasing rise in goroutine counts for the process. Hitting the vault process with a SIGQUIT and looking at the goroutines, the bloat seems to be almost entirely in github.com/hashicorp/vault/vault.expireLeaseStrategyRevoke.func1

This grows for the lifetime of the process or until vault gets sealed.

The issue seems to be here:

func expireLeaseStrategyRevoke(ctx context.Context, m *ExpirationManager, le *leaseEntry) {
...
		go func() {
			select {
			case <-ctx.Done():
			case <-m.quitCh:
				cancel()
                         // Leaked for the lifetime of the process
			}
		}()

FWIW there isn't a concern for perf impact here, the goroutines don't do anything interesting. However, the goroutine count is something we value as a metric for overall health of vault and the bloat may hide other genuine problems. Ideally this should be easy to fix/avoid.

To Reproduce
Steps to reproduce the behavior:

1. Run vault 0.10.2 or higher
2. Create a bunch of leases
3. See runtime.num_goroutines climb
4. kill -3 to get goroutine dump (also kills process)
5. cat log | grep goroutine -A 1 | grep -v goroutine | grep -v "\-\-" | sed 's|(.*$||g' | sort | uniq -c
6. Notice github.com/hashicorp/vault/vault.expireLeaseStrategyRevoke.func1 dominating the list

Expected behavior
Expect the goroutine count to stay bounded

Environment:

• Vault Server Version (retrieve with vault status): 0.11.2
• Vault CLI Version (retrieve with vault version): 0.11.2
• Server Operating System/Architecture: ubuntu 18.x

[viveklak]

Added a potential fix in #5506

[briankassouf]

fixed by #5506