Follower rejecting leader's pre-vote

[k-jingyang]

Hello, thanks for the library! We were trying out raft v1.7.0, and found that the pre-vote implementation has a bug where a follower rejects the pre-vote req from node X even when when the follower thinks that X is the leader.

This is the offending line. https://github.com/hashicorp/raft/blob/main/raft.go#L1752

// Check if we have an existing leader [who's not the candidate] and also
var candidate ServerAddress // ISSUE: this is not set at all
candidateID := ServerID(req.ID)

// if the Servers list is empty that mean the cluster is very likely trying to bootstrap,
// Grant the vote
if len(r.configurations.latest.Servers) > 0 && !inConfiguration(r.configurations.latest, candidateID) {
	r.logger.Warn("rejecting pre-vote request since node is not in configuration",
		"from", candidate)
	return
}

if leaderAddr, leaderID := r.LeaderWithID(); leaderAddr != "" && leaderAddr != candidate { // ISSUE: so this will always return true as long as the follower has a leader
	r.logger.Warn("rejecting pre-vote request since we have a leader",
		"from", candidate,
		"leader", leaderAddr,
		"leader-id", string(leaderID))
	return
}

[banks]

@k-jingyang great catch and thanks for reporting this and creating a PR.

I'm curious how you found this case. Did you observe this bug in a production or test settting? Or was this something you spotted in the code?

I agree we should probably fix that code as it's at least not doing what was intended, just trying to understand the scenarios where it would have an impact.

[k-jingyang]

Hey @banks, we observed this bug in our test cluster of 3 nodes. For us, it happened on re-deployment, and it has only happened once so far.

There's an ongoing discussion in the PR regarding what actually happened and the scenarios.