node/manager: Remove ipset config from previous node state

Found by code inspection from
cilium#23208 (comment),
thanks to Joe.

Fix this discrepancy so that we can potentially backport this fix if
needed to older branches. The surrounding logic will get refactored in
the aforementioned PR.

This affects users that are running with the following:
  - --tunnel=disabled (native routing)
  - --enable-bpf-masquerade=false
  - --enable-ipv{4,6}-masquerade=true

Fixes: 49cb220 ("iptables: Don't masquerade traffic to cluster nodes")
Suggested-by: Joe Stringer <[email protected]>
Signed-off-by: Chris Tarazi <[email protected]>

pkg/node/manager/manager.go

@@ -484,6 +484,9 @@ func (m *manager) NodeUpdated(n nodeTypes.Node) {
 		// Delete the old node IP addresses if they have changed in this node.
 		var oldNodeIPAddrs []string
 		for _, address := range oldNode.IPAddresses {
+			if option.Config.NodeIpsetNeeded() && address.Type == addressing.NodeInternalIP {
+				iptables.RemoveFromNodeIpset(address.IP)
+			}
 			if skipIPCache(address) {
 				continue
 			}