storage: fix decommissioning of absent node

[tbg]

When a node is decommissioned in absentia, the gateway node may not be
aware of its most recent liveness entry. Before this bug fix, the
gateway would fail the decommissioning process. Instead, it now falls
back to reading the liveness record from the KV store.

Fixes #17995.

Release note (bug fix): Decommissioning a node that has already been
terminated now works in all cases. Success previously depended on
whether the gateway node "remembered" the absent decommissionee.

[cockroach-teamcity]

This change is 

[nvanbenschoten]

Review status: 0 of 2 files reviewed at latest revision, all discussions resolved, some commit checks failed.

---

pkg/storage/node_liveness.go, line 276 at r1 (raw file):

		// results in unintentional no-ops (due to the Gossip lag); this could be
		// observed by users in principle, too.
		dummyLiveness := &Liveness{NodeID: nodeID} // always catches condition failed

Why don't we create a new method to retrieve the liveness instead of this obscure approach of sending a cput and hoping for the condition failed error? I don't think we need the retry logic in updateLiveness if we're only performing a read.

---

Comments from Reviewable

[bdarnell]

---

Review status: 0 of 2 files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

---

Comments from Reviewable

[tbg]

TFTRs!

---

Review status: 0 of 2 files reviewed at latest revision, 1 unresolved discussion.

---

pkg/storage/node_liveness.go, line 276 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Why don't we create a new method to retrieve the liveness instead of this obscure approach of sending a cput and hoping for the condition failed error? I don't think we need the retry logic in updateLiveness if we're only performing a read.

Good point, done.

---

Comments from Reviewable

[nvanbenschoten]

---

Review status: 0 of 2 files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

---

pkg/storage/node_liveness.go, line 276 at r2 (raw file):

		// results in unintentional no-ops (due to the Gossip lag); this could be
		// observed by users in principle, too.
		oldLiveness := &Liveness{}

Nit: put oldLiveness on the stack instead of dereferencing everywhere below.

---

Comments from Reviewable

[tbg]

Review status: 0 of 2 files reviewed at latest revision, 1 unresolved discussion, all commit checks successful.

---

pkg/storage/node_liveness.go, line 276 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Nit: put oldLiveness on the stack instead of dereferencing everywhere below.

Done.

---

Comments from Reviewable