Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNS Zone Webhook Returns Empty List #89

Closed
ben-ihelputech opened this issue Nov 9, 2023 · 10 comments
Closed

DNS Zone Webhook Returns Empty List #89

ben-ihelputech opened this issue Nov 9, 2023 · 10 comments
Assignees
Labels
bug Something isn't working

Comments

@ben-ihelputech
Copy link

Versions
NetBox Version: 3.6.4
NetBox DNS Version: 0.20.2
Python Version: 3.10.12

Describe the bug
When creating a new zone, the webhook returns an empty list and does not display the nameservers. This is an issue because I cannot update a specific nameserver with the new zone data.
I also noticed that it does not return the correct nameserver when I update the zone.

To Reproduce

  1. Create a webhook. Ex:
{
    "id": 3,
    "url": "https://netbox.infra.example.net/api/extras/webhooks/3/",
    "display": "DNS Webhook",
    "content_types": [
  	  "netbox_dns.nameserver",
  	  "netbox_dns.zone",
  	  "netbox_dns.record",
  	  "netbox_dns.view"
    ],
    "name": "DNS Webhook",
    "type_create": true,
    "type_update": true,
    "type_delete": true,
    "type_job_start": false,
    "type_job_end": false,
    "payload_url": "http://10.30.23.69:18000",
    "enabled": true,
    "http_method": "POST",
    "http_content_type": "application/json",
    "additional_headers": "",
    "body_template": "",
    "secret": "",
    "conditions": null,
    "ssl_verification": false,
    "ca_file_path": null,
    "custom_fields": {},
    "tags": [],
    "created": "2023-11-09T20:32:14.599971Z",
    "last_updated": "2023-11-09T20:35:08.783193Z"
}
  1. I used nc -l 18000 on my local machine to listen to all requests coming from the webhook.
  2. Create two nameservers, ns1.example.com and ns2.example.com
  3. Create a zone, example.com with ns1.example.com and ns2.example.com selected in the WebUI as the nameservers.
  4. The json result does not show any nameservers:
{
    "event": "created",
    "timestamp": "2023-11-09 20:46:15.821455+00:00",
    "model": "zone",
    "username": "ben",
    "request_id": "61039297-0d40-486e-893a-040a21493d60",
    "data": {
        "id": 11,
        "url": "/api/plugins/netbox-dns/zones/11/",
        "name": "example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "Internal",
            "name": "Internal"
        },
        "display": "[Internal] example.com",
        "nameservers": [],
        "status": "active",
        "description": "test zone",
        "tags": [],
        "created": "2023-11-09T20:46:15.692991Z",
        "last_updated": "2023-11-09T20:46:15.693009Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 4,
            "url": "/api/plugins/netbox-dns/nameservers/4/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "admin.example.com",
        "soa_serial": 1699562776,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "active": null,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": null,
        "postchange": {
            "created": "2023-11-09T20:46:15.692Z",
            "last_updated": "2023-11-09T20:46:15.693Z",
            "view": 1,
            "name": "example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 4,
            "soa_rname": "admin.example.com",
            "soa_serial": 1699562776,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "test zone",
            "arpa_network": null,
            "tenant": null,
            "nameservers": [],
            "custom_fields": {},
            "tags": []
        }
    }
}
  1. I deleted ns2.example.com from the zone in the webUI, but the update then returned both Nameservers:
{
    "event": "updated",
    "timestamp": "2023-11-09 20:55:56.702573+00:00",
    "model": "zone",
    "username": "ben",
    "request_id": "bd59306d-1c19-465d-abba-edc5ee433ec7",
    "data": {
        "id": 11,
        "url": "/api/plugins/netbox-dns/zones/11/",
        "name": "example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "Internal",
            "name": "Internal"
        },
        "display": "[Internal] example.com",
        "nameservers": [
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns1.example.com",
                "name": "ns1.example.com"
            },
            {
                "id": 5,
                "url": "/api/plugins/netbox-dns/nameservers/5/",
                "display": "ns2.example.com",
                "name": "ns2.example.com"
            }
        ],
        "status": "active",
        "description": "test zone",
        "tags": [],
        "created": "2023-11-09T20:46:15.692991Z",
        "last_updated": "2023-11-09T20:55:56.602596Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 4,
            "url": "/api/plugins/netbox-dns/nameservers/4/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "admin.example.com",
        "soa_serial": 1699562776,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "active": true,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": {
            "created": "2023-11-09T20:46:15.692Z",
            "last_updated": "2023-11-09T20:46:15.787Z",
            "view": 1,
            "name": "example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 4,
            "soa_rname": "admin.example.com",
            "soa_serial": 1699562776,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "test zone",
            "arpa_network": null,
            "tenant": null,
            "nameservers": [
                4,
                5
            ],
            "custom_fields": {},
            "tags": []
        },
        "postchange": {
            "created": "2023-11-09T20:46:15.692Z",
            "last_updated": "2023-11-09T20:55:56.602Z",
            "view": 1,
            "name": "example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 4,
            "soa_rname": "admin.example.com",
            "soa_serial": 1699562776,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "test zone",
            "arpa_network": null,
            "tenant": null,
            "nameservers": [
                4,
                5
            ],
            "custom_fields": {},
            "tags": []
        }
    }
}
  1. I readded ns2.example.com to the zone, and it only showed ns1.example.com:
{
    "event": "updated",
    "timestamp": "2023-11-09 20:56:06.296369+00:00",
    "model": "zone",
    "username": "ben",
    "request_id": "dc454d86-731b-4be2-b59a-b4388679528d",
    "data": {
        "id": 11,
        "url": "/api/plugins/netbox-dns/zones/11/",
        "name": "example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "Internal",
            "name": "Internal"
        },
        "display": "[Internal] example.com",
        "nameservers": [
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns1.example.com",
                "name": "ns1.example.com"
            }
        ],
        "status": "active",
        "description": "test zone",
        "tags": [],
        "created": "2023-11-09T20:46:15.692991Z",
        "last_updated": "2023-11-09T20:56:06.175544Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 4,
            "url": "/api/plugins/netbox-dns/nameservers/4/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "admin.example.com",
        "soa_serial": 1699563357,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "active": true,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": {
            "created": "2023-11-09T20:46:15.692Z",
            "last_updated": "2023-11-09T20:55:56.666Z",
            "view": 1,
            "name": "example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 4,
            "soa_rname": "admin.example.com",
            "soa_serial": 1699563357,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "test zone",
            "arpa_network": null,
            "tenant": null,
            "nameservers": [
                4
            ],
            "custom_fields": {},
            "tags": []
        },
        "postchange": {
            "created": "2023-11-09T20:46:15.692Z",
            "last_updated": "2023-11-09T20:56:06.175Z",
            "view": 1,
            "name": "example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 4,
            "soa_rname": "admin.example.com",
            "soa_serial": 1699563357,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "test zone",
            "arpa_network": null,
            "tenant": null,
            "nameservers": [
                4
            ],
            "custom_fields": {},
            "tags": []
        }
    }
}
@peteeckel
Copy link
Owner

Hi @ben-ihelputech, thanks for this report.

I know that there are issues with the data that are passed by webhooks and already made some (unfortunately not very successful) attempts at fixing them. They are caused by a combination of the inner workings of NetBox with regard to webhooks and the internal mechanisms of NetBox DNS when Zone objects are crated or updated - they just don't mix very well.

Another issue is that in some cases the webhook is triggered twice for a single operation, with different data provided.

There is, however, an open issue netbox-community/netbox#14132 with the main NetBox project that aims at redesigning the whole webhook mechanism to a more general event based architecture, and I decided to put off all further attempts at fixing the problems with webhook data until that has been done. In the ideal case the problem will simply go away, in the less ideal (but more probable) case any solution that I'll find now will be broken again once that issue has been worked on.

I know that's not really helpful to you in the short term, but it really doesn't make much sense to tweak that code before the NetBox core team finalizes the new architecture.

@peteeckel peteeckel added the bug Something isn't working label Nov 9, 2023
@ben-ihelputech
Copy link
Author

That makes sense. I wasn't sure if it was a bug with the plugin or upstream, so I filed it here first. I'll keep an eye on that issue. Thanks!

@peteeckel peteeckel self-assigned this Nov 9, 2023
@peteeckel
Copy link
Owner

Hi @ben-ihelputech, a little update on this.

Meanwhile the first beta release of NetBox 3.7 became available.

The bad news is that the bug does not disappear by itself, so it really is an issue in NetBox DNS that needs to be addressed. The good news is that it is compatible with 3.7, i.e. it occurs in exactly the same way as before. That's good news because it means that it is very likely that it can be fixed without breaking compatibility with NetBox 3.5 and 3.6.

I did some more investigation in what causes the problem, and it seems to be closely related with changes to the many-to-many relationship between Zone and Nameserver objects. After fixing another redundancy in the code (which needs to have some tests assigned, but so far it doesn't seem to cause any regressions) I am seeing the following behaviour:

  • If the nameservers list for a zone is changed, an additional update webhook is triggered after the create or update webhook for the creation of or change to a zone. The first one does not have the correct nameservers, the second one does.
  • If the nameservers list is not changed, all is well and the webhook gets the correct data for the changed or created zone.

So now the challenge is to find out what causes the second (early) invocation of the webhook code. So far I am fairly sure that it isn't the call to django.models.db.save() - added some debugging code to Django and it's called exactly once. My hope is that as soon as I find out what causes that extraneous event I can modify the code so it doesn't happen anymore.

@peteeckel
Copy link
Owner

peteeckel commented Dec 6, 2023

Another update: I think I found the cause of the problem. Have a look at netbox-community/netbox#14450.

The issue only arises with models that have a many-to-many relationship (such as zones). Essentially, two events are queued when the m2m relationship changes, and since they are not queued immediately after each other NetBox fails to see that the second one is an update to the first one. So you get two events, the first of which does not take the m2m change into account.

Even worse, when you have a webhook for a create operation and none for update, that webhook never sees the correct data.

I created a patch for NetBox that can fix the problem, provided it doesn't have any negative implications on NetBox itself, and is accepted by the core team. If they don't accept it at all, I'm afraid it will be difficult to find a workaround in NetBox DNS.

@peteeckel
Copy link
Owner

Update: Some good news - the issue has been accepted and a PR fixing it is on its way. So we're hopefully on path to finally closing this.

@peteeckel
Copy link
Owner

Before the NetBox patch:

[1] Fri, 01 Mar 2024 18:45:54 GMT 127.0.0.1 "POST / HTTP/1.1" 200 -
Host: localhost:9000
Accept-Encoding: identity
User-Agent: python-urllib3/2.1.0
Content-Type: application/json
Content-Length: 3058

{
    "event": "updated",
    "timestamp": "2024-03-01T18:45:54.873803+00:00",
    "model": "zone",
    "username": "admin",
    "request_id": "817400ab-bacd-4ff0-aaa6-36d03eb3d761",
    "data": {
        "id": 9,
        "url": "/api/plugins/netbox-dns/zones/9/",
        "name": "zone10.example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "internal",
            "name": "internal"
        },
        "display": "[internal] zone10.example.com",
        "nameservers": [
            {
                "id": 2,
                "url": "/api/plugins/netbox-dns/nameservers/2/",
                "display": "ns2.example.com",
                "name": "ns2.example.com"
            },
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns4.example.com",
                "name": "ns4.example.com"
            },
            {
                "id": 5,
                "url": "/api/plugins/netbox-dns/nameservers/5/",
                "display": "ns5.example.com",
                "name": "ns5.example.com"
            }
        ],
        "status": "active",
        "description": "",
        "tags": [
            {
                "id": 1,
                "url": "/api/extras/tags/1/",
                "display": "tag1",
                "name": "tag1",
                "slug": "tag1",
                "color": "ff0000"
            }
        ],
        "created": "2024-01-31T14:51:24.505695Z",
        "last_updated": "2024-03-01T18:45:54.606811Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/nameservers/1/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "hostmaster.example.com",
        "soa_serial": 1709318733,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "rfc2317_prefix": null,
        "rfc2317_parent_managed": false,
        "rfc2317_parent_zone": null,
        "rfc2317_child_zones": [],
        "registrar": null,
        "registry_domain_id": null,
        "registrant": null,
        "tech_c": null,
        "admin_c": null,
        "billing_c": null,
        "active": true,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": {
            "created": "2024-01-31T14:51:24.505Z",
            "view": 1,
            "name": "zone10.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709318733,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        },
        "postchange": {
            "created": "2024-01-31T14:51:24.505Z",
            "last_updated": "2024-03-01T18:45:54.606Z",
            "view": 1,
            "name": "zone10.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709318733,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        }
    }
}
Completed request #1
------------
[2] Fri, 01 Mar 2024 18:45:54 GMT 127.0.0.1 "POST / HTTP/1.1" 200 -
Host: localhost:9000
Accept-Encoding: identity
User-Agent: python-urllib3/2.1.0
Content-Type: application/json
Content-Length: 2523

{
    "event": "updated",
    "timestamp": "2024-03-01T18:45:54.884969+00:00",
    "model": "zone",
    "username": "admin",
    "request_id": "817400ab-bacd-4ff0-aaa6-36d03eb3d761",
    "data": {
        "id": 9,
        "url": "/api/plugins/netbox-dns/zones/9/",
        "name": "zone10.example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "internal",
            "name": "internal"
        },
        "display": "[internal] zone10.example.com",
        "nameservers": [
            {
                "id": 1,
                "url": "/api/plugins/netbox-dns/nameservers/1/",
                "display": "ns1.example.com",
                "name": "ns1.example.com"
            },
            {
                "id": 2,
                "url": "/api/plugins/netbox-dns/nameservers/2/",
                "display": "ns2.example.com",
                "name": "ns2.example.com"
            },
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns4.example.com",
                "name": "ns4.example.com"
            },
            {
                "id": 5,
                "url": "/api/plugins/netbox-dns/nameservers/5/",
                "display": "ns5.example.com",
                "name": "ns5.example.com"
            }
        ],
        "status": "active",
        "description": "",
        "tags": [
            {
                "id": 1,
                "url": "/api/extras/tags/1/",
                "display": "tag1",
                "name": "tag1",
                "slug": "tag1",
                "color": "ff0000"
            }
        ],
        "created": "2024-01-31T14:51:24.505695Z",
        "last_updated": "2024-03-01T18:45:54.680853Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/nameservers/1/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "hostmaster.example.com",
        "soa_serial": 1709318755,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "rfc2317_prefix": null,
        "rfc2317_parent_managed": false,
        "rfc2317_parent_zone": null,
        "rfc2317_child_zones": [],
        "registrar": null,
        "registry_domain_id": null,
        "registrant": null,
        "tech_c": null,
        "admin_c": null,
        "billing_c": null,
        "active": null,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": null,
        "postchange": {
            "created": "2024-01-31T14:51:24.505Z",
            "last_updated": "2024-03-01T18:45:54.680Z",
            "view": 1,
            "name": "zone10.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709318755,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                1,
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        }
    }
}
Completed request #2
------------
[3] Fri, 01 Mar 2024 18:45:55 GMT 127.0.0.1 "POST / HTTP/1.1" 200 -
Host: localhost:9000
Accept-Encoding: identity
User-Agent: python-urllib3/2.1.0
Content-Type: application/json
Content-Length: 2523

{
    "event": "updated",
    "timestamp": "2024-03-01T18:45:54.889053+00:00",
    "model": "zone",
    "username": "admin",
    "request_id": "817400ab-bacd-4ff0-aaa6-36d03eb3d761",
    "data": {
        "id": 9,
        "url": "/api/plugins/netbox-dns/zones/9/",
        "name": "zone10.example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "internal",
            "name": "internal"
        },
        "display": "[internal] zone10.example.com",
        "nameservers": [
            {
                "id": 1,
                "url": "/api/plugins/netbox-dns/nameservers/1/",
                "display": "ns1.example.com",
                "name": "ns1.example.com"
            },
            {
                "id": 2,
                "url": "/api/plugins/netbox-dns/nameservers/2/",
                "display": "ns2.example.com",
                "name": "ns2.example.com"
            },
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns4.example.com",
                "name": "ns4.example.com"
            },
            {
                "id": 5,
                "url": "/api/plugins/netbox-dns/nameservers/5/",
                "display": "ns5.example.com",
                "name": "ns5.example.com"
            }
        ],
        "status": "active",
        "description": "",
        "tags": [
            {
                "id": 1,
                "url": "/api/extras/tags/1/",
                "display": "tag1",
                "name": "tag1",
                "slug": "tag1",
                "color": "ff0000"
            }
        ],
        "created": "2024-01-31T14:51:24.505695Z",
        "last_updated": "2024-03-01T18:45:54.727447Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/nameservers/1/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "hostmaster.example.com",
        "soa_serial": 1709318755,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "rfc2317_prefix": null,
        "rfc2317_parent_managed": false,
        "rfc2317_parent_zone": null,
        "rfc2317_child_zones": [],
        "registrar": null,
        "registry_domain_id": null,
        "registrant": null,
        "tech_c": null,
        "admin_c": null,
        "billing_c": null,
        "active": null,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": null,
        "postchange": {
            "created": "2024-01-31T14:51:24.505Z",
            "last_updated": "2024-03-01T18:45:54.727Z",
            "view": 1,
            "name": "zone10.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709318755,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                1,
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        }
    }
}
Completed request #3
------------
[4] Fri, 01 Mar 2024 18:45:55 GMT 127.0.0.1 "POST / HTTP/1.1" 200 -
Host: localhost:9000
Accept-Encoding: identity
User-Agent: python-urllib3/2.1.0
Content-Type: application/json
Content-Length: 2523

{
    "event": "updated",
    "timestamp": "2024-03-01T18:45:54.893500+00:00",
    "model": "zone",
    "username": "admin",
    "request_id": "817400ab-bacd-4ff0-aaa6-36d03eb3d761",
    "data": {
        "id": 9,
        "url": "/api/plugins/netbox-dns/zones/9/",
        "name": "zone10.example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "internal",
            "name": "internal"
        },
        "display": "[internal] zone10.example.com",
        "nameservers": [
            {
                "id": 1,
                "url": "/api/plugins/netbox-dns/nameservers/1/",
                "display": "ns1.example.com",
                "name": "ns1.example.com"
            },
            {
                "id": 2,
                "url": "/api/plugins/netbox-dns/nameservers/2/",
                "display": "ns2.example.com",
                "name": "ns2.example.com"
            },
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns4.example.com",
                "name": "ns4.example.com"
            },
            {
                "id": 5,
                "url": "/api/plugins/netbox-dns/nameservers/5/",
                "display": "ns5.example.com",
                "name": "ns5.example.com"
            }
        ],
        "status": "active",
        "description": "",
        "tags": [
            {
                "id": 1,
                "url": "/api/extras/tags/1/",
                "display": "tag1",
                "name": "tag1",
                "slug": "tag1",
                "color": "ff0000"
            }
        ],
        "created": "2024-01-31T14:51:24.505695Z",
        "last_updated": "2024-03-01T18:45:54.773895Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/nameservers/1/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "hostmaster.example.com",
        "soa_serial": 1709318755,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "rfc2317_prefix": null,
        "rfc2317_parent_managed": false,
        "rfc2317_parent_zone": null,
        "rfc2317_child_zones": [],
        "registrar": null,
        "registry_domain_id": null,
        "registrant": null,
        "tech_c": null,
        "admin_c": null,
        "billing_c": null,
        "active": null,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": null,
        "postchange": {
            "created": "2024-01-31T14:51:24.505Z",
            "last_updated": "2024-03-01T18:45:54.773Z",
            "view": 1,
            "name": "zone10.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709318755,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                1,
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        }
    }
}
Completed request #4
------------
[5] Fri, 01 Mar 2024 18:45:55 GMT 127.0.0.1 "POST / HTTP/1.1" 200 -
Host: localhost:9000
Accept-Encoding: identity
User-Agent: python-urllib3/2.1.0
Content-Type: application/json
Content-Length: 2523

{
    "event": "updated",
    "timestamp": "2024-03-01T18:45:54.897911+00:00",
    "model": "zone",
    "username": "admin",
    "request_id": "817400ab-bacd-4ff0-aaa6-36d03eb3d761",
    "data": {
        "id": 9,
        "url": "/api/plugins/netbox-dns/zones/9/",
        "name": "zone10.example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "internal",
            "name": "internal"
        },
        "display": "[internal] zone10.example.com",
        "nameservers": [
            {
                "id": 1,
                "url": "/api/plugins/netbox-dns/nameservers/1/",
                "display": "ns1.example.com",
                "name": "ns1.example.com"
            },
            {
                "id": 2,
                "url": "/api/plugins/netbox-dns/nameservers/2/",
                "display": "ns2.example.com",
                "name": "ns2.example.com"
            },
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns4.example.com",
                "name": "ns4.example.com"
            },
            {
                "id": 5,
                "url": "/api/plugins/netbox-dns/nameservers/5/",
                "display": "ns5.example.com",
                "name": "ns5.example.com"
            }
        ],
        "status": "active",
        "description": "",
        "tags": [
            {
                "id": 1,
                "url": "/api/extras/tags/1/",
                "display": "tag1",
                "name": "tag1",
                "slug": "tag1",
                "color": "ff0000"
            }
        ],
        "created": "2024-01-31T14:51:24.505695Z",
        "last_updated": "2024-03-01T18:45:54.818815Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/nameservers/1/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "hostmaster.example.com",
        "soa_serial": 1709318755,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "rfc2317_prefix": null,
        "rfc2317_parent_managed": false,
        "rfc2317_parent_zone": null,
        "rfc2317_child_zones": [],
        "registrar": null,
        "registry_domain_id": null,
        "registrant": null,
        "tech_c": null,
        "admin_c": null,
        "billing_c": null,
        "active": true,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": null,
        "postchange": {
            "created": "2024-01-31T14:51:24.505Z",
            "last_updated": "2024-03-01T18:45:54.818Z",
            "view": 1,
            "name": "zone10.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709318755,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                1,
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        }
    }
}
Completed request #5
------------

After:

[1] Fri, 01 Mar 2024 18:28:18 GMT 127.0.0.1 "POST / HTTP/1.1" 200 -
Host: localhost:9000
Accept-Encoding: identity
User-Agent: python-urllib3/2.1.0
Content-Type: application/json
Content-Length: 3178

{
    "event": "updated",
    "timestamp": "2024-03-01T18:28:18.161019+00:00",
    "model": "zone",
    "username": "admin",
    "request_id": "f4f7c2f5-dea6-4482-9cb5-fe4c04b86184",
    "data": {
        "id": 9,
        "url": "/api/plugins/netbox-dns/zones/9/",
        "name": "zone10.example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "internal",
            "name": "internal"
        },
        "display": "[internal] zone10.example.com",
        "nameservers": [
            {
                "id": 1,
                "url": "/api/plugins/netbox-dns/nameservers/1/",
                "display": "ns1.example.com",
                "name": "ns1.example.com"
            },
            {
                "id": 2,
                "url": "/api/plugins/netbox-dns/nameservers/2/",
                "display": "ns2.example.com",
                "name": "ns2.example.com"
            },
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns4.example.com",
                "name": "ns4.example.com"
            },
            {
                "id": 5,
                "url": "/api/plugins/netbox-dns/nameservers/5/",
                "display": "ns5.example.com",
                "name": "ns5.example.com"
            }
        ],
        "status": "active",
        "description": "",
        "tags": [
            {
                "id": 1,
                "url": "/api/extras/tags/1/",
                "display": "tag1",
                "name": "tag1",
                "slug": "tag1",
                "color": "ff0000"
            }
        ],
        "created": "2024-01-31T14:51:24.505695Z",
        "last_updated": "2024-03-01T18:28:18.102837Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/nameservers/1/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "hostmaster.example.com",
        "soa_serial": 1709317699,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "rfc2317_prefix": null,
        "rfc2317_parent_managed": false,
        "rfc2317_parent_zone": null,
        "rfc2317_child_zones": [],
        "registrar": null,
        "registry_domain_id": null,
        "registrant": null,
        "tech_c": null,
        "admin_c": null,
        "billing_c": null,
        "active": true,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": {
            "created": "2024-01-31T14:51:24.505Z",
            "view": 1,
            "name": "zone10.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709315450,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        },
        "postchange": {
            "created": "2024-01-31T14:51:24.505Z",
            "last_updated": "2024-03-01T18:28:18.102Z",
            "view": 1,
            "name": "zone10.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709317699,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                1,
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        }
    }
}
Completed request #1
------------

@peteeckel
Copy link
Owner

I'll try to get the number of save() operations down as well. There are definitely too many of them, which negatively affects performance as well as causing trouble with the existing NetBox code.

There's no way to get them to zero without either sacrificing functionality or creating a horribly convoluted construction. I tried for a couple of hours ... at the end of the day the need to automatically generate the SOA serial number makes a second save() inevitable, and that's enough to trigger the mess shown above (first webhook has incorrect instance data, second one an empty prechange snapshot.

@peteeckel
Copy link
Owner

peteeckel commented Mar 1, 2024

Ah yes: Of course it works with zone creation as well:

[1] Fri, 01 Mar 2024 19:02:20 GMT 127.0.0.1 "POST / HTTP/1.1" 200 -
Host: localhost:9000
Accept-Encoding: identity
User-Agent: python-urllib3/2.1.0
Content-Type: application/json
Content-Length: 2525

{
    "event": "created",
    "timestamp": "2024-03-01T19:02:20.025333+00:00",
    "model": "zone",
    "username": "admin",
    "request_id": "14f24892-a8e0-4d33-85f5-56f8ee3f0b96",
    "data": {
        "id": 22,
        "url": "/api/plugins/netbox-dns/zones/22/",
        "name": "zone12.example.com",
        "view": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/views/1/",
            "display": "internal",
            "name": "internal"
        },
        "display": "[internal] zone12.example.com",
        "nameservers": [
            {
                "id": 1,
                "url": "/api/plugins/netbox-dns/nameservers/1/",
                "display": "ns1.example.com",
                "name": "ns1.example.com"
            },
            {
                "id": 2,
                "url": "/api/plugins/netbox-dns/nameservers/2/",
                "display": "ns2.example.com",
                "name": "ns2.example.com"
            },
            {
                "id": 4,
                "url": "/api/plugins/netbox-dns/nameservers/4/",
                "display": "ns4.example.com",
                "name": "ns4.example.com"
            },
            {
                "id": 5,
                "url": "/api/plugins/netbox-dns/nameservers/5/",
                "display": "ns5.example.com",
                "name": "ns5.example.com"
            }
        ],
        "status": "active",
        "description": "",
        "tags": [
            {
                "id": 1,
                "url": "/api/extras/tags/1/",
                "display": "tag1",
                "name": "tag1",
                "slug": "tag1",
                "color": "ff0000"
            }
        ],
        "created": "2024-03-01T19:02:19.720510Z",
        "last_updated": "2024-03-01T19:02:19.967254Z",
        "default_ttl": 86400,
        "soa_ttl": 86400,
        "soa_mname": {
            "id": 1,
            "url": "/api/plugins/netbox-dns/nameservers/1/",
            "display": "ns1.example.com",
            "name": "ns1.example.com"
        },
        "soa_rname": "hostmaster.example.com",
        "soa_serial": 1709319740,
        "soa_serial_auto": true,
        "soa_refresh": 172800,
        "soa_retry": 7200,
        "soa_expire": 2592000,
        "soa_minimum": 3600,
        "rfc2317_prefix": null,
        "rfc2317_parent_managed": false,
        "rfc2317_parent_zone": null,
        "rfc2317_child_zones": [],
        "registrar": null,
        "registry_domain_id": null,
        "registrant": null,
        "tech_c": null,
        "admin_c": null,
        "billing_c": null,
        "active": null,
        "custom_fields": {},
        "tenant": null
    },
    "snapshots": {
        "prechange": null,
        "postchange": {
            "created": "2024-03-01T19:02:19.720Z",
            "last_updated": "2024-03-01T19:02:19.967Z",
            "view": 1,
            "name": "zone12.example.com",
            "status": "active",
            "default_ttl": 86400,
            "soa_ttl": 86400,
            "soa_mname": 1,
            "soa_rname": "hostmaster.example.com",
            "soa_serial": 1709319740,
            "soa_refresh": 172800,
            "soa_retry": 7200,
            "soa_expire": 2592000,
            "soa_minimum": 3600,
            "soa_serial_auto": true,
            "description": "",
            "arpa_network": null,
            "tenant": null,
            "registrar": null,
            "registry_domain_id": null,
            "registrant": null,
            "admin_c": null,
            "tech_c": null,
            "billing_c": null,
            "rfc2317_prefix": null,
            "rfc2317_parent_managed": false,
            "rfc2317_parent_zone": null,
            "nameservers": [
                1,
                2,
                4,
                5
            ],
            "custom_fields": {},
            "tags": [
                "tag1"
            ]
        }
    }
}
Completed request #1
------------

@peteeckel
Copy link
Owner

Good news: There is a new PR by @jeremystretch that will fix this issue: netbox-community/netbox#16366

I hope it will be merged before the next NetBox release. I already prepared a set of tests tests for NetBox DNS that detect the issues (as well as the tags problem reported by @Toxma in #277), and they succeed with the new NetBox PR in place.

@peteeckel
Copy link
Owner

It's fixed!

netbox-community/netbox#15194

Thanks so much, @jeremystretch!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants