StaleDataError under load

[sabaini]

Hi,

under load (e.g. heat-driven deployments) I'm getting faults when allocating ports.

I can see stack traces like the below, and subsequent timeouts plugging in VIF interfaces.

Is it possible we are seeing some kind of race condition here?

Thank you,
peter.

2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 427, in _call_on_drivers
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/gbpservice/neutron/plugins/ml2plus/drivers/apic_aim/mechanism_driver.py", line 2317, in update_port_precommit
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers segment=context.original_bottom_bound_segment, remove=True)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/gbpservice/neutron/plugins/ml2plus/drivers/apic_aim/mechanism_driver.py", line 4170, in _update_static_path
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers **{'old_path' if remove else 'new_path': hlink})
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/gbpservice/neutron/plugins/ml2plus/drivers/apic_aim/mechanism_driver.py", line 4128, in _update_static_path_for_network
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers self.aim.update(aim_ctx, epg, static_paths=epg.static_paths)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/aim/common/utils.py", line 57, in wrapper
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers return method(*args, **kwargs)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/aim/aim_manager.py", line 220, in update
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers return self.get(context, resource)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/aim/aim_manager.py", line 284, in get
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers for_update=for_update)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/aim/aim_manager.py", line 489, in _query_db_obj
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers objs = self._query_db(store, cls, for_update=for_update, **id_attr)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/aim/aim_manager.py", line 474, in _query_db
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers **kwargs) if db_cls else None)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/aim/aim_store.py", line 379, in query
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers **filters).all()
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2703, in all
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers return list(self)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2854, in iter
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers self.session._autoflush()
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1397, in _autoflush
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers self.flush()
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers self._flush(objects)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers transaction.rollback(_capture_exception=True)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in exit
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers compat.reraise(exc_type, exc_value, exc_tb)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers flush_context.execute()
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers rec.execute(self)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers uow
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers mapper, table, update)
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers (table.description, len(records), rows))
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers StaleDataError: UPDATE statement on table 'aim_endpoint_groups' expected to update 1 row(s); 0 were matched.
2021-07-01 11:19:29.068 150291 ERROR neutron.plugins.ml2.managers