NRG (2.11): Add ability to move cluster traffic into asset accounts

[neilalexander]

This PR adds the ability to move NRG traffic out of the system account and into the asset accounts. Particularly in heavily multi-tenanted systems, this can help cases where head-of-line blocking may happen in the system account due to large amounts of replication traffic.

This needs to be enabled on a per-account basis on each participating server. Servers advertise their capability to support account NRG to each other. If a Raft group detects a peer coming up without support, they will revert back to using the system account automatically.

Requires nats-io/jwt#223 and then go.mod updating.

Signed-off-by: Neil Twigg [email protected]

[ripienaar]

Can we also expose this in $SYS.REQ.SERVER.PING.VARZ? Be nice if nats server info could easily show this.

Also in JSZ.

Might also be worth to ask @mprimi to get into audit a check that all machines have this enabled and alert if not symetrical

[mprimi]

@ripienaar what's a good way to track requested/suggested checks and other improvements for audit?
Should i open an issue in that project?

@neilalexander I'm not familiar with this feature or the subsystems around it, however just from your description i have a comment/concern:

enabled on all servers participating in the cluster.
You cannot know if you know all members of the cluster, there may be some unknown unknowns.

How about instead having the the cluster meta-leader is turning this on or off based on the same (ephemeral, possibly outdated) view?

[ripienaar]

@mprimi you are welcome to open them here if you wish, add a label (or ask me) so you can find them easily

[ripienaar]

IF a new server joins a cluster that is in this mode and actively all happy but that new node is not configured in this manner what happens?

Can we make that node disable JetStream with a error in the logs?

Is there any concerns with Leafnodes?

[derekcollison]

I will try to take a look at this next week when I am back.. Will need to take my time.

Do we have a plan for visibility into $SYS via nats traffic when they is opted in?

[derekcollison]

@neilalexander let's sync this week on this one and get it merged. Agree we should be able to place in any account.

[wallyqs]

got a panic in one of the runs:

=== RUN   TestJetStreamClusterSnapshotBeforePurgeAndCatchup
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe90e12]
goroutine 67601 [running]:
github.com/nats-io/nats-server/v2/server.(*raft).recreateInternalSubsLocked(0xc00bfff808)
	/home/travis/build/nats-io/nats-server/server/raft.go:553 +0x92
github.com/nats-io/nats-server/v2/server.(*raft).RecreateInternalSubs(0xc00bfff808)
	/home/travis/build/nats-io/nats-server/server/raft.go:548 +0x89
github.com/nats-io/nats-server/v2/server.(*Server).updateNRGAccountStatus(0xc001944008)
	/home/travis/build/nats-io/nats-server/server/events.go:1725 +0x2f5
github.com/nats-io/nats-server/v2/server.(*Server).remoteServerUpdate(0xc001944008, 0x0?, 0xc00d5a4138?, 0xc001b1c000?, {0xc00082de40?, 0x446685?}, {0xc009036868?, 0xc009036830?}, {0x0, 0x0, ...}, ...)
	/home/travis/build/nats-io/nats-server/server/events.go:1654 +0x736
github.com/nats-io/nats-server/v2/server.(*Server).internalReceiveLoop(0xc001944008, 0xc009036800)
	/home/travis/build/nats-io/nats-server/server/events.go:436 +0x464
created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 67583
	/home/travis/build/nats-io/nats-server/server/server.go:1785 +0x945
FAIL	github.com/nats-io/nats-server/v2/server	373.110s

[neilalexander]

Panic fixed BTW.

[derekcollison]

We are very close. Just a general question when we see a change, do we want to redo internal state of existing node or should we delete old node and create a new one with same name?

[neilalexander]

For now have disabled being able to select a third account, so now limited to system and owner.

[derekcollison]

LGTM - one nit but not a blocker.

[derekcollison]

LGTM

Can pick up as needed in separate PR.

[aricart]

I have a PR for the validation of the JWT, if this is the case, the JWT can be validated after you decode. See nats-io/jwt#224