Custom Script initialisation via API requests full execution history of that script for no obvious reason

[pgnuta]

Deployment Type

Self-hosted

NetBox Version

v4.0.10

Python Version

3.12

Steps to Reproduce

1. Have 300,000+ execution history records for one particular custom script (2000+ executions per hour).
2. Have an architecture that includes a database server not hosted on the same host as the netbox process.
3. Initialise a new execution of that custom script via an API call.
4. Observe that the API call HTTP response takes 20+ seconds to return.

Expected Behavior

The script execution job should be added to the background queues list faster than it is.

Observed Behavior

When the API call is first received by netbox it performs an SQL query requesting the entire execution history for that script:

2024-08-31 23:23:46.694 NZST [23469] netbox_test LOG: statement: SELECT "core_job"."id", "core_job"."object_type_id", "core_job"."object_id", "core_job"."name", "core_job"."created", "core_job"."scheduled", "core_job"."interval", "core_job"."started", "core_job"."completed", "core_job"."user_id", "core_job"."status", "core_job"."data", "core_job"."error", "core_job"."job_id" FROM "core_job" WHERE ("core_job"."object_type_id" = 69 AND "core_job"."object_id" IN (2)) ORDER BY "core_job"."created" DESC

This query takes a long time to transfer the volume of data in the table due to the size of the execution history and it waits for this to finish before the job is inserted into the table:

2024-08-31 23:24:06.992 NZST [23469] netbox_test LOG: statement: INSERT INTO "core_job" ("object_type_id", "object_id", "name", "created", "scheduled", "interval", "started", "completed", "user_id", "status", "data", "error", "job_id") VALUES (69, 2, 'SyncLibrenms', '2024-08-31 11:24:06.972273+00:00'::timestamptz, NULL, NULL, NULL, NULL, 1, 'pending', NULL, '', '7667384535e94785b75acb525133e6c5'::uuid) RETURNING "core_job"."id"

As far as i can tell there is no reason to perform this query as it is not performed when initialising a custom script execution through the frontend.

[pgnuta]

Just a followup:

• Yes i know that 300k+ executions in the history table is a lot, retention has been shortened from 90 days to 5 days as a workaround while we resolve this.
• The issue is that when a lot of these scripts are fired off by external systems hitting the API at random they consume a uwsgi/gunicorn worker for 20+ seconds and the whole application grinds to a halt.

[arthanson]

@pgnuta can you please try the PR #17373 (Note: On NetBox 4.1.0) please let me know if that resolves the issue for you.

[arthanson]

@pgnuta ?

[pgnuta]

@arthanson. I have tested the PR and it has resolved the issue, thanks.