Pre-generate the OpenAPI schema

[candlerb]

NetBox version

v2.11.3

Feature type

Change to existing functionality

Proposed functionality

A feature like "collectstatic" which builds the openapi schema once in ./upgrade.sh, then serves it as a static file.

Currently it's served dynamically and is very slow, taking consistently 7-8 seconds on my system:

# time curl -sS localhost:8001/api/docs/?format=openapi | wc
      0   78785  799797

real	0m7.805s
user	0m0.016s
sys	0m0.005s

Alternatively, cache it in redis (it will take nearly a megabyte of RAM, but worth it)

Use case

ansible netbox inventory

The ansible netbox inventory fetches the whole openapi schema each time it runs. When combined with the actual data queries it does for devices, vms etc, this gives around a 10-second lead time to any playbook which is hitting the netbox API.

Whilst this inventory module has a feature to cache the results of the inventory (if you configure a suitable cache plugin), I want to get the "real-time" up-to-the-minute data from Netbox. The plugin can't be configured to cache the openapi schema only, whilst still making data queries to Netbox.

pynetbox

Any code which uses pynetbox, and calls nb.openapi(), will also benefit.

import pynetbox
nb = pynetbox.api(API_URL, token=API_TOKEN)
nb.openapi()

Database changes

None

External dependencies

None

[mraerino]

This really got worse on 2.11 - my webserver even times out while generating the spec.

Is there a console command you can run to get the openapi json?

[jeremystretch]

This really got worse on 2.11

I haven't noticed any difference from v2.10, and I can't recall any changes we've made that would have a significant impact on it.

[sdktr]

While serving a static version of the spec seems harmless since it's version bound anyway, I'm not sure that it makes sense to request the complete schema each time the plugin starts? No matter how fast netbox spits out the file, is that a roundtrip we can avoid?

[candlerb]

There are two ways to bite the cherry.

1. At the server end: pre-generate the spec and serve it statically
2. At the client end: read the spec once and cache it

Whilst both are possible, (1) gives the maximum benefit:

• It only has to be implemented once
• It benefits all clients, even dumb ones
• Clients don't have to worry about running with a stale copy of the spec when Netbox changes
• TBH, transferring 1MB over the network is cheap these days

Ideally, Netbox would pregenerate the spec and give an ETag - then smart clients can check and don't have to refetch unless it has changed.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

[jeremystretch]

I don't think we can pre-generate the full API spec as part of the release process, given that it contains some values unique to the installation (i.e. the base API URL). We also can't generate this at the time of installation because such values may change as the configuration is modified (e.g. if BASE_PATH is changed).

What we can do is specify a cache timeout for the rendered page. This isn't perfect, but it should at least greatly reduce load times for bringing up the API docs. I'm going to try a 24-hour timeout for the rendered specs.

[candlerb]

We also can't generate this at the time of installation because such values may change as the configuration is modified (e.g. if BASE_PATH is changed).

I think it would be reasonable to require the user to re-run some command like manage.py genapi if the BASE_PATH is changed (but to have it run automatically on ./update.sh, like collectstatic is)

[jeremystretch]

It would be very difficult to convey that to the user and ensure that it's undertaken consistently. IMO it's not worth the additional overhead for such a minor impact.

[jeremystretch]

IMO this is a reasonable optimization that avoids introducing any additional overhead. Just wanted to make some progress on this as no one else has volunteered to own this.

I'm happy to consider further improvements, though it's worth noting that we're somewhat limited by the capabilities of the drf-yasg package. Any suggestions for improvements to the actual spec rendering should be made upstream.

[candlerb]

Given that the solution turns out to be just a case of adding cache_timeout=86400, that seems like a very reasonable cost/benefit tradeoff :-)