Displaying the page of a prefix with many child prefixes (each having a child range) runs into gateway timeout

[reduzent]

Deployment Type

Self-hosted

NetBox Version

v3.6.6

Python Version

3.11

Steps to Reproduce

The page /ipam/prefixes/1/ on our Netbox instance takes more than 20mins to display and runs into a timeout when not increasing timeout values.

My test system is a Debian 12 VM (VMware VM) with netbox 3.6.6. The system has two CPU cores and 4 GB Memory. Initial state is an empty database.

I can trigger the 504 Gateway Time-out (nginx error) when I create a prefix with 1000 child prefixes where each child prefix has one child range.

This is how I create the objects in nbshell:

mainpref = Prefix(prefix='10.128.0.0/11', status='active')
mainpref.full_clean()
mainpref.save()

mainpref = Prefix.objects.filter(prefix='10.128.0.0/11')[0]

prefsize = 256
prefcount = 1000

for i in range(0, prefcount*prefsize, prefsize):
    newpref = Prefix(prefix=mainpref.prefix[i].format() + '/24', status='active')
    newpref.full_clean()
    newpref.save()

for cpref in mainpref.get_child_prefixes():
    newrange = IPRange(start_address=cpref.prefix[100].format() + '/24' , end_address=cpref.prefix[250].format() + '/24' , status='active')
    newrange.full_clean()
    newrange.save()

If you apply this to a new installation, this URL triggers the timeout:

/ipam/prefixes/1/

Or click the result of this search:

/ipam/prefixes/?q=10.128.0.0%2F11

There is a related #13471 where it appears that we can't do much about it but hope that this going to be addressed in Netbox itself.

Expected Behavior

The page should be rendered within reasonable time.

Observed Behavior

We get a 504 when trying to visit the particular page.

[reduzent]

This is a duplicate of #13880 which was closed due missing information. Missing information is now hopefully provided.

[jeremystretch]

The delay stems from the call to get_available_ips() for the parent prefix. It's actually happening twice: first with the call to return the number of available IPs, and again to return the first available IP. We need to optimize the logic for calculating available IPs when accounting for ranges.

[Smixi]

Hello,
I made a small PR for this if you want to look at it. I just investigated specifically on the mentionned function, but I guess it can be used anywhere where netaddr.ipset are used.

This is related to how adding and removing items in the sets works. It's seem its way faster to make a list of network items then transform it as a ipset rather than adding them one by one. I think this is because it make internal loops everytime you add items in it.

The page loads with the given script in ~6s on my computer.

[arthanson]

Assigning to you @Smixi as I'm working on other bugfixes right now.

[Smixi]

Note: Maybe we can also add cached property into prefix as it is made inside IPRange ? It will solve the second point mentionned here :

The delay stems from the call to get_available_ips() for the parent prefix. It's actually happening twice: first with the call to return the number of available IPs, and again to return the first available IP.

Or create another issue ?

[reduzent]

Nice. Thanks for fixing!