Heads up: IPAM DNSsync is coming, IPAM Coupling will be removed in version 1.1

[peteeckel]

Breaking change ahead

In version 1.1.0, planned for later this autumn, the new and much more capable IPAM DNSsync feature will replace IPAM Coupling.

While still not finished, the new feature is on course for a release together with NetBox 4.1, planned for late 2024. The new feature is going to automate the creation of address records for IPAM IPAddress objects in a new way, doing away with the need to manually assign a zone and name to an IP address. The functionality is based on IPAM Prefix objects being linked to NetBox DNS View objects:

• One or more prefixes can be assigned to one or more views in NetBox DNS (this is a many-to-many-relationship)
• When an IP address that is contained in a prefix is created or updated, NetBox DNS looks for DNS zones in the linked views that match the IP addresses dns_name field
• An address record per zone is created with the name provided in dns_name so that the FQDN of the record matches dns_name
• Changes to the zone, view, prefix or IP address or in the assignment between them will be reflected automatically
• The assignment between prefixes and views is inheritable. A sub-prefix without a view will inherit the views from its next parent prefix with views, or it can be assigned to its own view if necessary.

This makes it possible to create multiple address records in different views and in more than one zone, which is necessary for split zone setups and many other purposes. It is also much simpler to have an address record (and the corresponding pointer record) created for IP addresses, which with IPAM coupling requires a manual assignment per address. In the ideal case, linking IP addresses to DNS records will be completely transparent.

On the technical side, the current custom fields ipaddress_dns_record_name and ipaddress_dns_zone_id are no longer necessary. There will, however, be some optional custom fields that can be used to control the records created for an IP address, such as ipaddress_dns_record_ttl and ipaddress_dns_record_disable_ptr (with the same functionality as for IPAM Coupling) and a new field ipaddress_dns_disabled to disable creation of a DNS record for an IP address altogether. To disable the feature altogether, just not mapping any prefix to views does the trick.

Basic functionality is already working, though some polish is still needed and a release in autumn is targeted.

There is one downside: IPAM DNSsync and IPAM Coupling (the current experimental feature) cannot coexist. I tried, but the effort of making both work alternatively is not in a reasonable ratio to the usefulness of having both present at the same time. That means that the release of IPAM DNSsync will terminate the functionality of IPAM Coupling irrevocably. It does not make much sens to provide a migration path either, since the ways both work are too different - however, I'm confident that the much lower amount of maintaining DNS address records for IPAM IP addresses with IPAM DNSsync will make the transition rather painless.

Any comments, suggestions or questions are welcome!

[peteeckel]

Following the release of NetBox 4.1.0-beta1 some hours ago, I just pushed the first beta release NetBox DNS 1.1.0-beta1.

Currently there are some issues with testing on GitHub that I can't really explain (locally all tests work as expected, but the GH workflow fails reproducably for three tests), but in general the new version and especially the DNSsync feature should be stable enouft for first experiments. I'll be thankful for issues, suggestions and comments!

[peteeckel]

I just pushed a new beta release 1.1.0-beta2 to PyPI

Apart from a couple of bugfixes, this release adds the dnssync_conflict_ deactivate configuration option. By default this option is disabled, but when you enable it it will automatically deactivate existing address records (A or AAAA) that were created manually.

This feature aims at making migrating to IPAM DNSsync easier.

[peteeckel]

Somehow the fact that there isn't any feedback is a bit disconcerting ... either nobody has tried the new beta, or I'm better at finding my bugs than you are :-)

Anyway, there is a new and possible final beta release available, 1.1b3 (sorry for the inconsistency in the naming schema, I'm still experimenting with the GitHub workflow and apparently haven't had enough coffee yet).

Apart from some additional checks preventing NetBox DNS from reporting internal server errors in unusual contexts the main feature is the "DNS Views" button in the prefixes detail view which makes it possible to assign views to prefixes, not just the other way around. That proved a major click-saver in my internal testing.

[peteeckel]

A different kind of good news: The breaking change won't be too bad, at least if nothing unforseen happens in terms of API changes it will be possible to run NetBox DNS 1.0.x (with IPAM Coupling) on NetBox 4.1.x, or NetBox DNS 4.1.1 (with IPAM DNSsync) on NetBox 4.0.x.

That provides a much smoother migration path, leaving users of IPAM Coupling time until they want to upgrade to NetBox 4.2, which will likely break compatibility with NetBox DNS 1.0.x completely. So if you rely on IPAM Coupling, you are probably safe to keep it until NetBox 4.2 comes around. At the same time, NetBox DNS 1.2.x will become the current version, which will no longer support NetBox 4.0.x.

[ThomasADavis]

Finally found the time to look at this.. We will keep you posted about any problems we find. It could easily clean up a few things for us.

[kollross]

I've been trying to keep up, but it's hard to get a feel for next steps. Assuming we currently using the coupling feature, is/will there be a documented migration path once 1.1 becomes GA?

I'm willing to test on the beta, but I'm not exactly sure what I need to do to get from A to B.

[peteeckel]

Update: I just released 1.1.0b4, which does not have the "AutoDNS" trademark in it anymore, anywhere.

[netravnen]

Release a 1.1.0b5?

ea5962b committed after 1.1.0b4 seems rather important.
Without this 'fix', the prefix selector when editing the view is unusable (inline search in the field does not function)

git log --oneline --graph --all --since="4 days ago" --max-count=6
* d67900a (origin/feature/ipam-dnssync) Added "Related IP Address" column to managed records table (no sorting)
* 60da38a Compatibility information with coupling or DNSsync
* ea5962b Fixed API widget for Prefix assignment to View objects
* 337f213 Renamed AutoDNS to DNSsync for trademark issues
* 34e3449 Renamed all files from *autodns* to *dnssync*
* 23ecf84 (tag: 1.1.0b4) Prepared release 1.1.0b4

[netravnen]

Very recently learned about the DNSSync feature of netbox-plugin-dns. Did not know it existed.

After installing the branch directly /opt/netbox/venv/bin/pip3 install git+https://github.com/peteeckel/netbox-plugin-dns.git@feature/ipam-dnssync and restarting netbox, netbox-rq, the prefix filter works as described in ea5962b. Am trying setting the beta release up on a cloned instance.

[peteeckel]

Release a 1.1.0b5?

ea5962b committed after 1.1.0b4 seems rather important. Without this 'fix', the prefix selector when editing the view is unusable (inline search in the field does not function)

Hard to argue ... wait a sec.

[peteeckel]

1.1.0b5 with the fixed selector is out.

[ThomasADavis]

Ok, working with it.. my questions are:

• How does one verify this is working? Do the records move to 'Managed' state?
  • we already populate the dns_name field when possible before this.
• What exactly is need for configuration?
  • I see autodns_conflict_deactivate
  • is there any other flags needed?
• does deleting an IP address also delete the dns record?

From what I can see far..

• install plugin
• run migrate/restart netbox
• add to plugins.py autodns_conflict_deactivate with true if you dangling records removed..
• once netbox is up and running, run /opt/netbox/netbox/manage.py setup_dnssync
• go to netbox, dns zones page, find a zone, and attach a prefix. something happens.

One feature request - when viewing the DNS record, can the IP address be clickable to the IPAM page for the IP?

• one might want some logic to insure the IP is actually in the IPAM section before allowing this to happen..

I found also that the drop down for the add a prefix to zone doesn't work very well - if I start typing in a prefix, I'd thought it should at least narrow down what it's showing, but it doesn't.. Don't know if that's a netbox or a plugin problem..

[peteeckel]

Also, we do create all IP's dns_names with FQDN. I wonder if that could be a problem..

No, that's actually necessary to find the matching zones.

[ThomasADavis]

Yea, this a 100% private internal DNS, so we shortened the domain. We use to run a sub-domain, but decided to move off that to make leaking queries easier to find.

[peteeckel]

I won't be able to provide this today, but if you want to be able to test further, please modify the file netbox_dns/utilities/ipam_dnssync.py.

Find the line

        fqdn.split(i)[1].to_text().rstrip(".") for i in range(3, len(fqdn.labels))

and change it to

        fqdn.split(i)[1].to_text().rstrip(".") for i in range(2, len(fqdn.labels))

Then restart NetBox and you should be able to create the names in the TLD.

And be aware that you should never to that with public zones ... :-)

[peteeckel]

The latest commit has the feature.

You can now set dnssync_minimum_zone_labels to 1 to allow your FQDN format (please be aware that this applies globally, i.e. for public zones as well - so you'd probably better not add the .com zone to your NetBox DNS :-)).

[ThomasADavis]

thanks! we will test it out.

[peteeckel]

Important Fix

I just found a big pile of manure that can only be explained by lack of caffeine, beer, sleep or whatever.

The code that determines whether a DNS record for an IP address should be updated or not was totally broken. And by 'totally' I mean it did not work at all.

The good news is that it failed on the safe side, which did not result in any inconsistencies or data errors. The bad news is that it led to some operations (especially ones that take very long anyway) taking up to five times as long.

1.1.0b6 is on its way.

[ThomasADavis]

Ok, getting closer.. Now I have this showing up:

There is at least one active A record for name b59-ups-2 in zone [OMNI] omni and TTL is different (86400).

This is an existing record, and the TTL according to the IPam page is.. "-". What can be done about this? I don't like the idea of hitting every single IP record to set a TTL..

[ThomasADavis]

also added zone_default_ttl to the plugin config, it still happens..

[peteeckel]

1. You have not set up the custom fields for DNSsync yet (manage.py setup_dnssync). They are not strictly necessary if you don't need the functionality, but you still have the old ones from IPAM Coupling, some of which ('Zone', 'Name') are without function. However, this is not the problem here.
2. You seem to have explicit TTLs for the existing records in your zones. NetBox DNS used to set a new record's TTL to the zone default TTL explicitly in the beginning, recent versions set the TTL for new records to None unless specified otherwise.
3. There is obviously a bug in the functionality of dnssync_conflict_deactivate which makes it fail when there is a TTL on the existing record. I'll fix that.

Until that fix is available (should be easy to do, so you can probably expect it in time to continue testing) you can remove the TTLs from your existing address records - this will put the zone's default TTL in function, or deactivate them manually. But as I said, expect a fix shortly.

Update: The fix is in feature/ipam-dnssync now. New Beta due today (there have been some major performance improvements for specific operations as well)

[peteeckel]

By the way - this doesn't get said often enough: I'm really grateful to everyone who decides to spend their time to test, create issues, ask questions and be engaged in any form of activity around this project. Very often questions or specific use cases help discovering errors or finding bugs, or just make me aware that there are use cases I didn't think about.

It's really difficult to do this on my own, even if I have some customer environments where I can test as well - it's always from my own perspective, which isn't necessarily the only or even the most common one. So again: Thanks to you all!

[ThomasADavis]

Yea, I've wiped/reset the db too many times, and have lost track of "did I migrate it?"

This db has been using the previous plugin for ever.. Now I'm starting to wonder if I should prep for the move by stopping the octodns sync, delete all the 'a' records since we have set a FQDN dns_name in all IPs, and then migrate it all, then re-enable the octodns sync.. This will also clean out of a bunch of cruft any way..

[peteeckel]

A clean-slate approach wouldn't be a bad start for going productive, but anyway as long as you are willing to test I'll gladly provide fixed versions when you find something.

If you want to go on - do you need a new beta release or are you OK with using the latest version of the feature branch? There have been some fixes over the last few hours, particularly the one for the "conflict resolution with non-matching TTL" issue you discovered yesterday, but also some performance-related stuff.

[ThomasADavis]

Oh, we can keep on breaking stuff. This is a dev system.. so it's for breaking things. 😁 I think it's easier to keep track of what I'm testing by doing beta releases then using the feature head. This should help you with catching some of the weirder stuff..

[peteeckel]

Thanks, I really appreciate it! Currently I'm in a NetBox related conf call, but as soon as that's finished I'll push 1.1b7 for you.

[peteeckel]

... and done. Just released 1.1.0b7 to PyPI.

[ThomasADavis]

Just a quick note, that last release seems to fix all the problems I have other than the self-inflicted ones. 😁

[peteeckel]

Hi @ThomasADavis, thanks for the news, that's good to hear! And thank you very much for the input, that was really helpful - especially since you found some weird edge cases I didn't think of (and some not-so-weird oversights by yours truly :-)).

Currently I'm doing some fine polish on main and rebase them on the feature branch. I expect NetBox 4.1 in the next few days, and will release 1.1.0 GA shortly thereafter.

[peteeckel]

NetBox DNS 1.1.0 has just been released.

[phlpr]

Hi,
We have just completed our migration to Netbox 4.1.3 and noticed the changes related to the DNS plugin.

As heavy users of the IPAM-Coupling feature, we now need to transition to DNSSync. While we've successfully completed the migration steps on our development machine, I am having some difficulty fully understanding the overall concept. I have the following question:

In the DNSSync documentation, it states: "IPAM DNSsync no longer overwrites the 'DNS Name' field of IP addresses but uses it to create DNS records. One downside is that IDNs must be formatted in Punycode instead of being entered directly, but the advantage is that the 'DNS Name' field is now properly validated."

If I create a new IP address and enter just the hostname (e.g., "host1"), no record is created, which is expected. However, if I enter a fully qualified domain name (FQDN) with a non-existent zone, such as "host1.failure1.example.com," no record is created either.

Does this mean that, compared to IPAM-Coupling, DNSSync relies on accurate data input by users? Or am I overlooking something important?

Thanks,
Phil

[peteeckel]

If I create a new IP address and enter just the hostname (e.g., "host1"), no record is created, which is expected. However, if I enter a fully qualified domain name (FQDN) with a non-existent zone, such as "host1.failure1.example.com," no record is created either.

This is correct.

Does this mean that, compared to IPAM-Coupling, DNSSync relies on accurate data input by users? Or am I overlooking something important?

You are not missing anything, indeed DNSSync relies on the dns_name being correct inasmuch as it ends on the name of an existing domain.

If you just forgot to create the domain before adding the value in dns_name that's not a problem - when you create a matching zone the names will be populated automatically.

Validating mis-typed names is a different issue. One way to notice that something went wrong is to check whether the "Related DNS Address Records" pane shows up after you enter the DNS name. If the pane doesn't show up DNSSync couldn't find a matching zone, so there's no A record either.

If you used IPAM Coupling before, the DNS names should be correct for all existing records, so there shouldn't be an issue for existing data.

If you want to make sure your entered FQDNs are correct you can also use a custom validator if you want to make sure FQDNs match an existing zone. A code fragment to start with:

from dns import name as dns_name
from netbox_dns.utilities import get_zone

def check_valid_fqdn(ip_address):
    if len(dns_name.from_text(ip_address.dns_name)) > 2:
        return bool(get_zones(ip_address))

The condition checks if the dns_name is a valid FQDN (at least one .), get_zones is an internal function that returns the zones matching the IP address. This is obviously not production-level code, but you'll get the spirit. If you build a custom validator around it you can check dns_name values and reject values that don't match an existing zone.

[phlpr]

Thank you for the code snippet. A custom validator should be quick to implement.

[phlpr]

The custom-validator works fine:

from extras.validators import CustomValidator
from dns import name as plugin_dns_name
from netbox_dns.utilities import get_zones

class IPAddressValidator(CustomValidator):
    """ IPAdress validator; for DNS Name validation """
    def validate(self, instance, request):
        ip_dns = str(instance.dns_name)

        if len(plugin_dns_name.from_text(ip_dns)) > 2:
            b_zone_exist = bool(get_zones(instance))
            s_zone = ip_dns.split('.', 1)[1]
            if not b_zone_exist:
                self.fail(f"Zone '{s_zone}' does not exist in 'Netbox DNS' -> 'Zones'")

[peteeckel]

I could also add DNS records to the IP address tables so you can check correct record assignments without having to go to the detail view for each address ... let me see ...

[peteeckel]

That's better. Generally ManyToManyColumn rendering is hard to get right, but I think for the given purpose that should be OK.

[phlpr]

Wow, this looks perfect! :) Do you already have a beta version available for installation?

[peteeckel]

I'm still working on some stuff, so the release will still be a couple of days away - but if you want to you can check out the repository (the changes are in main) and install it editable using pip -e /path/to/git/clone.

Currently there are no changes to the database model in the queue, so upgrading from that to final is no issue.

[peteeckel]

Or, if you just want the table columns, you can simply take the file `netbox_dns/template_content.py´ out of the repo and replace your version with it - that's all it needs.

[phlpr]

Thank you! I installed it manually, and everything looks good. :)

[peteeckel]

If you need more time: NetBox DNS 1.0.x still has IPAM Coupling and is compatible with NetBox 4.1. This is a temporary situation, though - possibly it will still work with 4.2, but at some point there will be changes in NetBox that break compatibility beyond the point where it can be fixed easily.

[phlpr]

Thanks for the info. However, I believe the best approach is to proceed and transition to DNSSync immediately :)

[peteeckel]

So do I. As useful as Coupling used to be, there were some limitations that made it difficult to live with in larger environments.

[phlpr]

Another question came up during testing: I have several IP addresses with DNS A records in more than one zone. Is it common practice to let DNSSync create the first A record (e.g., "host1.zone1.example.com") and then manually create the second A record (e.g., "host1.zone2.example.com")?

[peteeckel]

Correct. That's where automation would be too complex, among other reasons because one of the advantages of the new mechanism was a simpler interface to the core IPAddress model (i.e. dns_name) instead of complex CF operations. Internally, custom fields are not full members of the fields categoriy, especially in terms of searching and indexing they are rather limited.

So it's one FQDN that is connected to DNS, and any other one will need to be created manually.

[phlpr]

Okay, I believe this will be my last question before the migration. :) I would like to continue using a custom field with the type Object(Zone) at the device level, but in a custom script, I need to retrieve the name of the zone. What is the best way to access the name property of this custom field? Using custom_fields.get('my_zone') only returns the ID of the zone.

[peteeckel]

Never mind, as you have seen recently questions usually lead to improvements :-)

I think I mentioned recently that custom fields are somewhat limited in their possibilities - essentially the custom_field_data stored with an object is a JSON-formatted string and does not really contain any object. If you want to do it by the book you need to retrieve the content type from the custom field's definition in the database, determine the model class of the referenced object from it and then do a query with that model's manager ... pretty cumbersome.

But since you know what class of object your CF is refrring to you can take a shortcut.

from netbox_dns.models import Zone

zone = Zone.objects.get(pk=device.custom_field_data.get("my_zone"))

(and again, this is not production ready :-)).

[phlpr]

I've been a bit busy over the past few days :) However, we successfully completed the migration to DNSSync today. Thank you very much!

[peteeckel]

Awesome, glad to hear it worked out.

If there's anything that you feel could be improved or if you find features that would be useful for you, please don't hesitate to open a bug report or a feature request, or just tell me and the other users about your experience. It's always inspiring to see how others use the plugin in practice!

[phlpr]

During final testing, I discovered that my custom validator fails in certain scenarios. This issue arises because not all prefixes are currently assigned to a view. To address this, I’ve adjusted the validator as shown below, which I believe is the most effective solution for the time being.

from extras.validators import CustomValidator
from dns import name as plugin_dns_name
from netbox_dns.utilities import get_zones
from netbox_dns.models import Zone

class IPAddressValidator(CustomValidator):
    """ IPAdress validator; for DNS Name validation """
    def validate(self, instance, request):
        ip_dns = str(instance.dns_name)

        if len(plugin_dns_name.from_text(ip_dns)) > 2:
            s_zone = ip_dns.split('.', 1)[1]
            # b_zone_exist = bool(get_zones(instance))
            b_zone_exist = len(Zone.objects.filter(name=s_zone)) > 0
            if not b_zone_exist:
                self.fail(f"Zone '{s_zone}' does not exist in 'Netbox DNS' -> 'Zones'")

This step takes the longest time for us due to the size of our environment and the large number of prefixes involved :)

[peteeckel]

The approach has two problems (that may or may not affect you, but in general they are something to observe):

s_zone = ip_dns.split('.', 1)[1]

Stripping the first label off the host name will not necessarily give you the zone. Take, for example he hostname github.com and you see immediately what happens: The record name is @ (the empty label) and does not appear in the FQDN at all, the zone is github.com.

 b_zone_exist = len(Zone.objects.filter(name=s_zone)) > 0

This will only work if you don't have multiple views.

If you want to detect prefixes that don't have a view associated with them you can use

def has_prefix_with_view(ip_address):
    return Prefix.objects.filter(
        vrf=ip_address.vrf,
        prefix__net_contains_or_equals=str(ip_address.address.ip),
        netbox_dns_views__isnull=False
    ).exists()

This is suprisingly complicated because in NetBox Core there actually is no relation between Prefix and IPAddress objects.

If you change your code to

from extras.validators import CustomValidator
from ipam.models import Prefix
from dns import name as plugin_dns_name
from netbox_dns.utilities import get_zones

class IPAddressValidator(CustomValidator):
    """ IPAdress validator; for DNS Name validation """
    def validate(self, instance, request):

        if len(plugin_dns_name.from_text(ip_dns)) <= 2:
            # Not an FQDN
            return

        if not Prefix.objects.filter(
            vrf=ip_address.vrf,
            prefix__net_contains_or_equals=str(ip_address.address.ip),
            netbox_dns_views__isnull=False
        ).exists():
            # No prefix containing the IP address has a DNS View assigned
            return

        if not get_zones(instance):
            # No matching DNS zone in any of the views assigned to the IP address' prefix
            self.fail(f"There is no matching zone for name {instance.dns_name}' in 'Netbox DNS' -> 'Zones'")

both problems can be avoided.

[phlpr]

Apologies for my delayed response; I’ve been occupied with other work outside of NetBox :)

I understand the first issue and your proposed solution. However, I initially had some difficulty following your approach for the second problem. We haven’t yet mapped all prefixes to views, so my initial thought was to simply check for a match in the zone table.

"This will only work if you don't have multiple views."

In my tests, this approach also works even when the prefix exists in multiple views.

However, after further consideration, I believe it might be more effective to prioritize and accelerate the view-to-prefix mapping instead.

[peteeckel]

Apologies for my delayed response; I’ve been occupied with other work outside of NetBox :)

Never mind, same here.

I understand the first issue and your proposed solution. However, I initially had some difficulty following your approach for the second problem. We haven’t yet mapped all prefixes to views, so my initial thought was to simply check for a match in the zone table.

As I said, it depends how your method of proceeding is. Just checking the zone name will tell you that there is some zone matching the host name, but not whether the zone's view is mapped to the prefix your IP address resides in, neither whether it is mapped at all. If you're fine with this it works, but it may leave you with IP addresses with a proper zone name but without assigned address records.

In my tests, this approach also works even when the prefix exists in multiple views.

... with the above limitation.

However, after further consideration, I believe it might be more effective to prioritize and accelerate the view-to-prefix mapping instead.

It may save you some performance issues. If you first assign DNS names to all IP addresses and then map the prefix to a view, all the IP addresses need to be handled at once. If you map first and then assign DNS names it's just one at a time. Depending on how many IP addresses you have per prefix, it may take a while to assign them and you may have to tweak timeout parameters in your web server.

[peteeckel]

For those still relying on IPAM Coupling:

As I said previously I made some effort to keep NetBox DNS 1.0.x (with IPAM Coupling) compatible with NetBox 4.1.x, and vice versa NetBox DNS 1.1.x (with IPAM DNSsync) with NetBox 4.0.x to give users time to to migrate to the new IPAM integration paradigm.

I started adapting NetBox DNS to the upcoming NetBox 4.2.x release train recently, and the changes required for 4.2 compatibility don't seem to make it feasible to extend this transition period, so NetBox DNS 1.2.x will no longer be backward compatible with NetBox 4.1 and lower, and NetBox DNS 1.0.x (and 1.1.x) will no longer run on NetBox 4.2.x.

[poupryc]

Hey, do you have some details about the new features making it impossible?

[peteeckel]

The main problem is that the URLs for Bulk Imports have changed from import to bulk_import. I just tested NetBox 1.1.5 against the NetBox feature branch and it fails badly (as was to be expected given the nature of the change).

netbox-community/netbox#18068

[peteeckel]

Some of the other changes could be postponed until 4.3, but as they provide major improvements I would not necessarily want to go down that path.

Generally, implementing version dependent code just for supporting a deprecated feature that had always explicitly been marked as 'experimental' further beyond its lifetime is nothing I would recommend to anyone. This kind of code tends to introduce weird issues and instabilities.