Rack list contains duplicate entries in paginated API responses

[henyxia]

Environment

• Python version: 3.7
• NetBox version: 2.6.10

Steps to Reproduce

1. GET /api/dcim/racks/ and iterate over all elements

Expected Behavior

List all racks without duplicates.

Expected Behavior

Some duplicated racks.

Context

We have around 4500 racks and we have only one duplicate.

Script used to reproduce the isse

#!/usr/bin/perl

use strict;
use warnings;

use JSON;

my $next = 'http://127.0.0.1:32769/api/dcim/racks/';
my $response;
my $res;

do {
    $response = `curl -s '$next'`;
    $res = decode_json($response);
    $next = $res->{next};
    foreach my $e (@{$res->{results}}) {
        print "$e->{id}\n";
    }
} while ($next);

And then run dup.pl | sort | uniq -d

Links

This might be a regression, or at least link to: #2938

[jeremystretch]

First, this is going to be very difficult to reproduce without any example data. Can you see if you can narrow it down to a specific set of conditions or naming pattern?

Second, if this is related to NaturalOrderingManager, it will be be rendered moot by #3799 pretty soon, so it may not be worth digging too far into it.

[jeremystretch]

@henyxia can you try commenting out this line under class Rack in dcim/models.py and see if you can still reproduce this issue?

    objects = NaturalOrderingManager()

[henyxia]

Oy!
So, commenting the line (and restarting gunicorn) does not change anything.
I still got the same amount of duplicates in both (with and without the comment) cases.
I added even more racks and it seems to be even more duplicates.
Right now, I'm trying to isolate the replicate the issue with some random data for the issue to be more reproductible :)

[henyxia]

So, right now I am now able to reproduce it on a bare new install.
Steps:

1. Spawn a new netbox instance (latest version)
2. Create a new site
3. Create 8k racks in this site

for i in {1..8000}; do curl -H "Content-Type: application/json" -H "Accept: application/json" -d '{"name":"rack1", "site":1}' -X POST -H 'Authorization: Token  0123456789abcdef0123456789abcdef01234567' http://127.0.0.1:32769/api/dcim/racks/; done

And then, multiple duplicates are returned with dup.pl | sort | uniq -d

[henyxia]

I'm still searching on my side but I think the bug is now easily reproducible (through the steps described before).
Do you think anything need to be added to help?

[hSaria]

I can confirm that this is happening when there are racks with the same name. You don't even need to go crazy with the numbers; 50 racks with 10 on the paging limit can recreate this issue.

The lower the page limit, the more often you'll see duplicates. I'm also running NetBox directly using manage.py runserver to eliminate as many variables as possible.

This happens with or without brief; I just added it to spare you from the long output.

>>> import json, requests
>>> 
>>> headers = {'Authorization': 'Token ef1764f45a5dd423e7bb6ba8502aabb9838b0c36', 'Content-type': 'application/json'}
>>> 
>>> for i in range(50):
...     resp = requests.post('http://netbox/api/dcim/racks/', data='{"name": "rack1", "site": 1}', headers=headers)
... 
>>> url = 'http://netbox/api/dcim/racks/?limit=10&brief=true'
>>> results = []
>>> 
>>> while url:
...     data = json.loads(requests.get(url, headers=headers).content)
...     url = data['next']
...     results += data['results']
... 
>>> for index, result in enumerate(results):
...     if results.count(result) > 1:
...         print('duplicate', result['id'], 'at', index)
... 
duplicate 50 at 9
duplicate 50 at 19
>>> results[9]
{'id': 50, 'url': 'http://netbox/api/dcim/racks/50/', 'name': 'rack1', 'display_name': 'rack1', 'device_count': None}
>>> results[19]
{'id': 50, 'url': 'http://netbox/api/dcim/racks/50/', 'name': 'rack1', 'display_name': 'rack1', 'device_count': None}

[hSaria]

I haven't gone around testing it, but this should be affecting other models too as ordering is not guaranteed if left unspecified as warned about in Django's docs.

[jeremystretch]

@hSaria Yeah, I think what's happening is that the ordering of Rack objects for which the set (site, group, name) is identical is non-deterministic. As you point out, this is likely affecting other models with non-unique ordering fields (though that doesn't seem to come up much). We'll need to comb through the models and identify any similar cases.

[hSaria]

I've tested the same thing on devices by leaving the name empty and it does also return duplicates. I suppose it's rare that people create into duplicate objects.

I'll go through the models to try to figure out which ones can be dup-ed with the current order, but the eyes of someone more experienced would definitely prove worthy.

[jeremystretch]

FYI the apparent solution here is to simply append id to the list of fields by which each affected model is ordered.

[hSaria]

Found and confirmed (tested) the following:

• dcim.Device
• extras.ImageAttachment
• ipam.VRF
• ipam.Prefix
• ipam.IPAddress
• ipam.VLANGroup (not sure if you're supposed to be able to create a duplicate global group)
• ipam.VLAN

I'll work on a PR for this. I don't mind if someone else wants to.

[jeremystretch]

Hold off on the PR for now. This will require introducing new migrations and I'd rather avoid having to refactor the significant number of migrations pending for v2.7.

[jeremystretch]

This will be fixed in the v2.7 release.