Failed to wait for master change - key /calico/openstack/v2/no-region/neutron_election: WatchTimedOut()

[mariomk777]

Hi,
we have a cluster:

• OpenStack: Rocky
  • Controllers: 5
  • Compute nodes: ~700
  • Instances: ~5000
  • Ports: ~5500
  • Security group lists: ~800
• Calico: 3.15
• Etcd: 3.3.12
• Kernel: 5.4

Issues:

• on the controllers we often get (several times for an hour):

2021-01-25 08:08:12.081 31272 WARNING networking_calico.plugins.ml2.drivers.calico.election [-] Failed to wait for master change - key /calico/openstack/v2/no-region/neutron_election: WatchTimedOut(): None: etcd3gw.exceptions.WatchTimedOut
2021-01-25 08:08:12.081 31258 WARNING networking_calico.plugins.ml2.drivers.calico.election [-] Failed to wait for master change - key /calico/openstack/v2/no-region/neutron_election: WatchTimedOut():  None: etcd3gw.exceptions.WatchTimedOut                                                                                                  
2021-01-25 08:08:12.082 31245 WARNING networking_calico.plugins.ml2.drivers.calico.election [-] Failed to wait for master change - key /calico/openstack/v2/no-region/neutron_election: WatchTimedOut(): None: etcd3gw.exceptions.WatchTimedOut                                                                                                  
2021-01-25 08:08:12.082 31258 INFO networking_calico.plugins.ml2.drivers.calico.election [-] Retrying leader election in 1.7 seconds
2021-01-25 08:08:12.082 31272 INFO networking_calico.plugins.ml2.drivers.calico.election [-] Retrying leader election in 1.4 seconds
2021-01-25 08:08:12.083 31245 INFO networking_calico.plugins.ml2.drivers.calico.election [-] Retrying leader election in 1.6 seconds

If I understand this correctly, the elected master should refresh the master key every 10s. The non-master nodes watch the master key with the timeout 2*10s and then raise WatchTimedOut exception. When I watch the master key directly, I see changes between 10s and 30s. There is no etcd compaction running. Is this the expected behavior?

We have additional clusters, exactly the same setup and configuration, only a smaller number of compute nodes (~250) and we see no problem. Does the number of compute nodes matter? Because only the elected master does all the work (read and update states ...).

• changing (API call) security group on a neutron port is very slow (>30s). No debug yet, but again - a small cluster behaves much better

Thanks