Add index on extras_cachedvalue.object_id for better update performance with many interfaces

[arjanhak]

NetBox version

v3.7.0

Feature type

Data model extension

Proposed functionality

Add an index to the object_id column on the extras_cachedvalue table.

This was suggested before in #11775 but for inserting new interfaces, and that was solved in #11785 by skipping the clearing of the cache on inserts, but that logic does of course not apply to updating interfaces.
Then there was an implementation of indexes for all generic foreign keys in #11790, but that did not include this object_id as it is something that can refer to multiple tables and is therefore not a formal foreign key.

Use case

When there are many interfaces (we are currently working with 300k+, times 7 attributes gives 2.2 million rows in the extras_cachedvalue table), updating (many of) them takes a lot of time. Looking at the PostgreSQL stats, this is the query that takes a lot of time:

DELETE FROM "extras_cachedvalue" WHERE ("extras_cachedvalue"."object_id" = 268011 AND "extras_cachedvalue"."object_type_id" = 38)

where in this instance 38 is the ID of dcim.interface, and that object_id is just an example.

Adding the suggested index made a big improvement in the time it takes to process the updates.

Database changes

CREATE INDEX extras_cachedvalue_object_id_TBD ON extras_cachedvalue USING btree(object_id);

External dependencies

No response

[jeremystretch]

Then there was an implementation of indexes for all generic foreign keys in #11790, but that did not include this object_id as it is something that can refer to multiple tables and is therefore not a formal foreign key.

That's what "generic" foreign key means in Django vernacular: a foreign key that can refer to multiple object types. This should have been implemented under #11790 as you point out, and more recently under #14436 for v3.7. I suspect it was missed because CachedValue is using our custom RestrictedGenericForeignKey field instead of the normal GenericForeignKey, so it escaped both audits.

CREATE INDEX extras_cachedvalue_object_id_TBD ON extras_cachedvalue USING btree(object_id);

We should create an index for both the object type and object ID fields, as we've done for other GFKs, because both are needed to query a record by a related object. This is done under the model's Meta class:

class Meta:
    indexes = (
        models.Index(fields=('object_type', 'object_id')),
    )

Thanks for the report!