Migrate from using uint32 to uint64 for identifiers

[bdlamprecht]

Issue type

[X] Feature request

• NetBox version: 2.2.2

Description

According to the conversation in the NetworkToCode (Slack Channel) on October 19th, NetBox currently uses uint32 for all of its id values. This provides up to a maximum of 4,294,967,295 unique numbers.

While this quantity appears quite large at first glance, I can foresee this becoming a limiting factor when there are multiple devices with unique interfaces and the possibility of each interface consuming several IP addresses (ipv6).

Also, each new ip-address uses an id that never gets reused (even when that ip-address is deleted) and when scaled out against several VRFs multiplied by each device in the DCIM, that only further exposes this limitation.

By transitioning now to use uint64, this future headache (if not addressed now) will allow for 18,446,744,073,709,551,615 unique id numbers which allows for a much larger quantity of values that should not be exhausted any time soon (not going to say "never").

[darkstar]

I think this is not as urgent as you make it seem.

If you create 10 new objects every second, for 24/7, you will hit the limit somewhere around 2030.

If we assume that each object in the database takes up about 250 bytes (just a rough guess), you will hit the limit around the time your database exceeds 1TB in size.

I guess we can safely assume that both of these scenarios are still a while off, even for use-cases with lots and lots of tenants and devices :)

[RyanBreaker]

@darkstar /r/theydidthemath

[bdlamprecht]

Yes, you are correct, it is not as "urgent" as in I need it by tomorrow.

That being said, although I sense the implied sarcasm, but in my circumstances, my customer currently has over 10,000 networking devices which I will eventually need to import. Each of those devices has several VRFs and in each of those VRFs is at least 25 ip-addresses (ipv4 & ipv6). Similar math came be done for interfaces.

When you account for the changing environment of this customer and that an id is never re-used once it has been "created" (even upon deletion), this makes your comment less than helpful.

So claiming that if I only added 10 objects every second I would be good until 2030 is a little naive.
I appreciate you doing the math, but your comment came of as somewhat arrogant.

[RyanBreaker]

We're just having some fun discussion, no arrogance intended, apologies if it came off as such as tones don't always come through text the best.

By that sort of use case I can understand the request for this and would agree on a "why not" basis in the first place, no harm in it anyway. I'll look into how this can be changed, and pinging @jeremystretch on if he knows right away of how to do so or if he has any additional input.

[darkstar]

Sorry if I missed your sense of humo(u)r. There was no sarcasm intended, I just wanted to show that most people don't realize how big a number 4.3 billion is (...and, continuing with the maths, even in your case you could import a couple hundred of those customers into NetBox, or have that one customer change his whole infrastructure/IP range a couple of hundred times ;) ).

But I see your point, and I agree with Ryan on the "why not" basis. It's just an ID field anyway...

[RyanBreaker]

Looking at this actually, the id types that the application sets up in the database are of Postgres type INTEGER, a 4-byte signed type, but it also uses a custom sequencer which appears to reuse unused IDs instead of auto-incrementing.

All of this appears to go into the inner-workings of the Postgresql driver the application uses with how it sets up the tables in the first place.

[lampwins]

Just to chime in; not to throw any fuel on the fire here...

2^32 is a really big number. Also keep in mind the ID's are per table. Meaning you roughly get 2^32 of each kind of object in NetBox, not 2^32 objects in total.

Just a trivial example to show that I really think this is a non issue (for now): taking an example line card from a Nexus 9K which supports 208,000 IPv4 entries, times 10,000 devices, equals 2.08e9 which is less than half of 2^32.

At that scale I question if it is even worth keeping track of information in this manor. If you do manage to get all that data in netbox at some point, I would really like to know how it performs. I have been wanting to so some stress testing in a variety of scenarios for a while but I simple don't have data at that scale.

[jeremystretch]

My primary concern with the 32-bit primary key size relates to inventory items. Many devices can have dozens or hundreds of individual components. I have a Juniper EX9200, for instance, with 220 discrete inventory items (most of which are optics).

This is a concern because the most efficient way to update a device's inventory data is to wipe out all of its existing items and create a new object for each detected item. (Yes, there are more intelligent approaches, but they involve somewhat rather complex logic regarding parent relationships and would result in much longer run times.) So, if you're updating inventory data for 5,000 devices, with an average of say 50 items per device, each run increments the PK value by around 250,000. This will exhaust the 32-bit PK space after around 17,000 runs.

Granted, these are fairly large numbers, and 17 thousand runs allows for 46 years of daily inventory runs, but it's still a bit too tangible a number for comfort. Whether it makes sense to expand the field size for all objects or just InventoryItems I don't know.

[jeremystretch]

Casually commenting three years later...

Looks like DEFAULT_AUTO_FIELD is coming in Django 3.2, which should help with this.