client stalls when accessing a data dir that is already in use

[jazzyfresh]

Overview

When you run two clients with the same data_dir, the second one will indefinitely stall rather than failing.

Behavior

Before: Client setup stalls

After: Client setup fails

Reproduction

Usually, two clients with the same config will fail on port conflict first, but you can get into this stalled state in two ways

• Have clients use different port, but same data dir
• Have a Consul service stanza (this gets it into a weird state to be documented in another ticket)

Implementation

• Configure boltdb to use a 5 second timeout when accessing the data dir for the Nomad state store
• Special case error message for bolt.ErrTimeout, suggest that another client may be running on the same data_dir

Todo

• I'm not sure how to improve the error message to indicate that the client tried to access a data dir that was already in use
• submit e2e tests in separate pull request

[schmichael]

https://godoc.org/github.com/boltdb/bolt#Options

Unfortunately this doesn't sound like it works on Windows. Still a step in the right direction!

[nickethier]

It looks like the client exits from that error here:

nomad/command/agent/command.go

Lines 601 to 604 in 4ae2cd5

	if err := c.setupAgent(config, logger, logOutput, inmem); err != nil {
	logGate.Flush()
	return 1
	}

Perhaps after that logGate.Flush() we log an error so it shows up at the end. Or even better we just move the error thats already logged up to this block so its logged after the flush.

[notnoop]

@nickethier That is an excellent suggestion - I've hit this before.

This is a good incremental improvement for sure. We can follow up with using a nomad process file lock using a cross-platform library (e.g. https://github.com/gofrs/flock) early on in initialization process. This can handle windows and ensures that we never even attempt to start consul goroutines in the first place.

As for improving the error message, I'd suggest dropping the "failed to create database: " prefix, and special casing bolt.ErrTimeout error with something more descriptive (e.g. "timed out while openning database; is there another nomad process running?")`

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.