API Requests Via Token WITH "Allowed IPs" Set Broken After Upgrade to 3.6.5

[bnerickson]

NetBox version

v3.6.5

Python version

3.9

Steps to Reproduce

Unless otherwise noted, leave everything else default:

1. Create a permission with "can view" selected and select every object type.
2. Create a user with a password and the newly created permission assigned.
3. Create an API token assigned to the new user with "Allowed IPs" set to ::/0,0.0.0.0/0 to allow all.
4. Perform an API get request using the new token (should fail).
5. Remove the "Allowed IPs" setting on the API token and retry the request (should succeed).

Expected Behavior

Setting AllowedIPs should not throw a 500 Internal Server Error.

Observed Behavior

500: Internal Server Error is raised. Setting debug logging for django and netbox yield the following:

2023-11-10 18:01:57,367 netbox.config DEBUG: No previous configuration found in database; proceeding with default values
2023-11-10 18:01:57,368 netbox.config DEBUG: Initialized configuration
2023-11-10 18:01:57,402 netbox.config DEBUG: Cleared configuration
2023-11-10 18:01:57,402 django.request ERROR: Internal Server Error: /api/ipam/vlans/

[tomryder-inspirenet]

Observed a similar problem here, with IPv6 connections being denied (HTTP 403) for tokens with an ACL. I strongly suspect #14235 fixing #14085 is the culprit; it looks like it needs to be adjusted to handle parsing out the port for IPv6 connections.