Integration between NetBox IPAM and Netbox DNS

[pshunter]

Hi,
We switched to Netbox as a truth source some time ago and we used it to manage our DNS by pulling data from the DNSName record in devices. We'd like to use the DNS plugin to be able to really manage our DNS tables and also some exceptions.
However, as it seems there is no link between the devices' DNS Name record and the contents of the DNS table.
This leads us to enter the same data twice in 2 different locations (dns name of a device and IP).
Would there be a possibility to automatically create a DNS record when the DNSName field of a device or VM is filled up ?
Would it also be possible to remove the corresponding record if a device or VM is deleted ?

It would also be nice if the ips from DNS records would be clickable and lead to IPS in the IPAM module/view, and maybe add a link in the other direction too.

Thanks,
J.C.H.

[peteeckel]

Hi @pshunter, if I understand you correctly you want to have a link between the DNS Name field in IPAM and the DNS PTR or address records in NetBox DNS, correct?

Unfortunately this is not really a trivial task and cannot be solved for the general case. IP addresses may he duplicated in different VRFs, and the same is true for DNS records in different views, and the assignment of IP addesses/DNS names to DNS resource records is non-obvious. So far nobody came up with a good canonical solution for the general case so that the mechanism could be fully automated.

When the environment is simple (just one VRF, generally "global", and just one or no views in NetBox DNS) you can use the custom script in examples/custom-scripts/DNS_IPAM_Updater.py, possibly adjusted to your specific environment, as a starting point. A fully automated solution can be built upon the Webhooks mechanism using the NetBox API endpoint for the custom script as a webhook on record/IP Address changes.

When you need any help implementing this, please feel free to come back to me and I'll try to support you. Ideally, we get another useful example for the documentation out of it.

It would also be nice if the ips from DNS records would be clickable and lead to IPS in the IPAM module/view, and maybe add a link in the other direction too.

That's not that simple, for exactly the same reason - ambiguity in the general case. If someone comes up with a bullet-proof way to select the correct related object under all circumstances I'll gladly try to add the functionality ... I would be happy to have it myself.

[pshunter]

Hi, thank you very much for the insight. As you suggest, my case is the "simple" one and I will try to make use of the example script. As per the possibility of connecting a DNS record back to the correct device, maybe the best way is to add a field to the DNS table to store an explicit reference to the concerned device. That way, if records are generated by an updater script from the devices section they could also populate the backref, and otherwise it should be of course possible to set it manually.
My 2 cents,
J.C.H

[zeddD1abl0]

I wasn't sure where to put this, but I definitely didn't want to start another thread on "Why not link to IPAM?", so it's going to live here.

Let me start out by saying, I understand the drive for this to be perfect and having stuff just magically work would be AWESOME. But, I think you might be too concerned with the affect of a VRF on DNS entries. And, I think you might also be worried about trying to save people from their own stupidity.

Let's start with an example. I have the default "Global" VRF, and I have 2 named VRFs; VRF1 and VRF2. If I have the following:

Global - hostname.example.com - 10.0.0.1/24
VRF1 - vrf.example.com - 10.0.0.1/24
VRF2 - magic.example.com - 10.0.0.2/24

And they're all in the same DNS zone, how would I deal with this anyway? They're all in the "EXAMPLE.COM" zone, so they're going to overlap anyway. Even from a network perspective, I don't have a solution for that internally. DNS will have three entries. BUT, I also don't care. I'm not worried that pinging vrf.example.com will give me the same result as pinging magic.example.com, even though I'm in the Global VRF. What would be slightly annoying is that the PTR record might be a bit weird. But, that's not impossible to rectify either. Once the records are created, I figure out which PTR I want, and then I disable the PTR record for the other entries.

OR, what could be done is; allow the ZONE/VIEW to filter entries based on VRF. This way, you could have a zone called "EXMAPLE.COM" that only has entries for the "Global" VRF, and a zone for VRF1 doing the same. Maybe you don't care about DNS for VRF2, you just want to put names against the hosts in there for IaC reasons. And, I just checked, and you don't provide a unique claim on the domain name, only the combination of domain name and view. So, potentially, just allow VRFs to be assigned to VIEWs, and then you could have "External View", "View", "VRF1 View", etc.

What this would mean is:

1. On creation of a zone/view, you're asked for the "Primary" VRF and the "Allowed" VRFs. The primary VRF would default to the "Global" VRF. Allowed VRFs would be a multi-select.

2. When an IP is added to the VRF in the IPAM table, it gets linked to the zone/view created with that VRF either as the "Primary" or in the list of "Allowed" VRFs. PTR records would be for the Primary only (or if you really want to, ranked VRF but I suggest not).

3. When an IP is removed from the VRF in the IPAM table, it gets deleted from the DNS setup too.

This isn't perfect, but it would be workable. If you were to go down this path, I'd also ask that you allow for a few things:

1. Zone filteriing based on the given "DNS Name". If I have "EXAMPLE.COM" and "EXAMPLE.IO", and the hostname is "magic.example.io", it shouldn't go into the "EXAMPLE.COM" domain.

2. Allow and link CNAMEs from one domain to another. If I have "magic.example.io" and I also want it to point to "magic.example.com", I'd love for that pathway to be connected, so I can go from CNAME > A Record > IPAM > Device, and possibly back the other way.

It's not a perfectly fully formed idea, but allowing the assignment of a VRF to a view, and then allowing that view do to the filtering for automatic creation wouldn't be too hard I would imagine. It would require people to think about what they're doing, but Netbox is allowed to be a bit idiosyncratic with some things. Take Primary IPs for example. And too, I'd much prefer something that works automatically 80% of the time, than something I have to do manually 100% of the time.

[peteeckel]

Hi @zeddD1abl0, thanks for your thoughts.

I admit to being a sucker for perfection ;-) ... but that's not the main thought behind trying to do 'the right thing'. The problem is more that when the flaws in a concept are too obvious, too many people will fall into traps and especially with DNS and other central infrastructure the results can be ... well, let's say you don't want that to happen. Data integrity is the main concern.

That's one of the reasons why I always suggest that when you have a well-defined use case, implement it as a custom script or something else and publish it so that others can participate from your partial solution. I will be glad to put it into the examples section, the documentation or wherever ... no problem at all. It's not that I don't want to have a solution, it's that none so far is something I would want to put in for general use.

There is also a new PR #68 by @jean1 which looks really interesting and which could make it into production code because it doesn't make any assumptions about specifics of a particular environment. I think it could be a step in the right direction. We discussed the issue offline for a while, and he's aware of the pitfalls.

Anyway, there is ongoing work towards that issue, and I will look deeper into it as soon as I have more time than I have right now. I really want to find a solution to this issue as well.

[zeddD1abl0]

I can understand the data integrity concern. And it's a legitimate concern if the situation isn't as clear as we all wish it was. I did see PR #68, and that was actually what led me to check the discussions section. I spent a good, long, time trying to come up with a good map of my thoughts around the IPAM <> Netbox DNS linking, so I could post an idea. I figured I wasn't the first, but I also hadn't seen any feature requests along those lines.

I'm sure you've probably got a better grasp on this situation than I do, being that you're actually working with the code, and I'm just working with your plugin. Just wanted to add a different perspective in case it helped. Best of luck with everything. I can't wait to see what PR #68 brings us.

[peteeckel]

IPAM linking

One thing about using dns_name (the 'DNS Name' field from IPAM) as a data source is that while there is some basic validation on its value the validation is by no means complete and does not cover all names allowed by the various RFCs, while allowing stuff that's forbidden for DNS labels.

It does, for example, not allow IDNs written in their UTF-8 representation (which is acceptable, though uncomfortable), but it happily accepts a name like aa--bb.example.com, which is pretty wierd but actually an illegal name, while xn-aa.example.comis legal and - to make confusion complete xn--bb.example.com is not (see RFC5980 for the ultimate horror if you're feeling strong enough). Labels with two dashes in position 3 and 4 are illegal, unless they start with xn and represent a valid punycode - which xn-aa is and xn-bb is not. There are other wierd cases as well.

One could of course validate the value dns_name in the process of creating an A/AAAA record, but consider cases in which there are large numbers of IP addresses with DNS names in the database ... it's no fun at all. That's one of the reasons I recommended updating 'DNS Name' from NetBox DNS data and not vice versa when I wrote the example scripts in the distribution.

That's exactly the problem that @jean1's approach seems to avoid - I'll do some extensive testing later on, but I'm quite hopeful it's a good approach.

CNAMEs

You also mention CNAME records in your initial comment, and I actually have an action item regarding CNAMEs on my list. One of the more common errors leading to DNS malfunction is broken CNAMEs. Either the creator of a CNAME record omits the dot at the end of the value when it's an FQDN, or they specify names that don't exist (or the targets are removed afterwards). So I'm planning to implement CNAME validation in NetBox DNS. Turns out that this is not as simple as it looks, and it can't be done in an exhaustive way. There are a couple of common cases:

The value of the CNAME RR does not end in a dot

That looks like the easy case but isn't.

• If we are dealing with a simple label (i.e., no interior dots) it's fine - the record has to be in the same zone, we can check that just so.
• If it has an interior dot, there are two major options: The target record is in the same zone (defined with an interior dot) or in a child zone. The first case is simple again, the second case has two sub-cases: The target could be in a child zone that is in NetBox DNS, or in a delegated zone managed somewhere else. The first case can be validated, the second one can't, except by doing DNS queries - and trust me, you don't want to open that can of worms. Think bulk imports or updates, for example, or multiple views.

The value of the CNAME RR ends in a dot

• In that case we have an absolute name, and that is essentially the same case as the last sub-sub-case discussed above. Detecting whether the name can be validated at all requires determining whether NetBox DNS maintains the zone the target name is in or not.

So we can validate simple name values (single labels without dot) in any case, and for all other cases we first need to find out whether we have authoritative information about the zone. If we do, we can check whether the name is in the zone or not - the latter would be an error. If we don't, we can't validate the CNAME.

If we can validate the CNAME we can also link to its target value, can't we? The answer is - no, we can't. A CNAME is an alias for a name, not for a record, so it doesn't necessarily have a unique target record. It can (and in fact does) point to all RRs that happen to have the CNAMEs value as their name, as it is a symbolic link to a name, not to a record. Simple example: An IPv4 and an IPv6 record for the same name usually exists in non-legacy environments. At best, we can (and probably will) add a list of all records a CNAME points to in the CNAMEs detail view. But your suggested CNAME linking just doesn't work, as it's not the way CNAMEs are defined.

Some features are really easy to implement. And some just look simple, but aren't ...

[peteeckel]

While working with @jean1 on PR #68 I stumbled across another example why it's difficult to reliably map IPAM IP Address objects to NetBox DNS records. A pretty simple and fairly common scenario, in this case from my own environment:

I have an internal and an external DNS view in order to handle NAT for my IPv4 address space and avoid leakage of internal network information to the Internet. Furthermore I (mainly) use an IPv6 prefix internally and externally, of course without NAT.

So now the problem ist as follows:

• Address records for all internal IPv4 addresses (NATted to an external /24) are in the internal view
• Address records for all external IPv4 addresses, most of them in the external /24 but some located elsewhere as well, are in the external view
• Address records for IPv6 addresses fall in two categories: The ones used in the internal communication only are in the internal view, the ones used in both internal and external communication are in both views. From an IP address standpoint, however, they are the same address, no separate VRFs whatsoever. This is where the 1/1 map from IP Address to Record fails.

This is one case where #68 will not help - each IP address can only be linked to one address record, so you need to choose the address in one of the views. Works for IPv4, would work for IPv6 (if there were no IPv4, i.e. an ideal world :-)), but it does not work for the dual-stack case.

Any thoughts or ideas?

[zeddD1abl0]

So, I got this message and my mind immediately started wandering through this situation you describe. I haven't looked over the code, and I haven't tried using the new patch. I'm running the old version of the plugin, and I haven't tried to review the code in any way. In this situation, I may be WAY off base with the way you're attempting to set this up, but here is a suggested option:

Set the IP Prefix up with a multi-select field for the "View" that the IP Prefix should be associated with. Then, when a new IP is added to the Prefix, update it in the DNS plugin.

There are a few reasons to take this approach. Firstly, this allows you to update multiple views with the same IP. This is useful in situations where you have specific DNS servers for, say development, but you also have a generalised DNS server for internal networking, or some such "Multiple DNS servers with specific requirements". Secondly, this means that your issue subsides somewhat. You can have 19 different views, and all of them will get the DNS update. Finally, you can do it at whatever level of IP Prefix you want. Have a 172.16.0.0/12 prefix that is split into /16's, that are then split into /22's, then split into /24's, you could assign a /22, 2 /16's, and 22 /24's to one View.

What problems does this have? It requires the user to specify the View. It might also require some very involved code on the plugin side. It could be very easily broken if the structure of the IP Prefix changes. This might be hard to understand for a user coming into it. I'm sure there are more problems, but I'm only thinking of problems now, and I haven't thought my way through the whole process.

I don't know if there's a specific need to have a 1:1 mapping of IP to DNS entry, especially when it's possible to have multiple DNS Views contain the same, or similar, information. You could put the multi-select field on the IP Address rather than the IP Prefix, but I think the IP Prefix is more logical and lazier from a user perspective.

[peteeckel]

Hm ... the idea has its merits. The only drawback I can see is that prefixes are a) optional and b) hierarchical, i.e. an IP address can have multiple prefixes or none at all. On the other hand, hierarchical is good - take the most specific one that is linked to a view and you'll be fine.

One issue might be if someone doesn't or want prefixes, in which case your suggestion just won't work. But that's a minor issue IMHO - after all, it doesn't hurt to use them.

The other issue is more relevant - the 'DNS Name' field isn't really suitable for generic DNS names, as pointed out above. You can use it in many cases, but there are legitimate names it won't accept and there are illegitimate names it does. The second case can be handled, albeit in an ugly way, by NetBox DNS' name validation, but the first one is a real restriction, though it won't happen too often (I hope that at least, there are users in Asian countries who might disagree - I just don't know).

But, to be fair - #68 has an issue with the first case as well, as you can create name/zone combinations that are legal in DNS but won't be accepted in the 'DNS Name' field.

[zeddD1abl0]

Is it worth raising an issue with Netbox? Their whole DNS validator is:

DNSValidator = RegexValidator(
    regex=r'^([0-9A-Za-z_-]+|\*)(\.[0-9A-Za-z_-]+)*\.?$',
    message='Only alphanumeric characters, asterisks, hyphens, periods, and underscores are allowed in DNS names',
    code='invalid'
)

Which is just a regex.

Mind you, I also have no idea what is an invalid DNS entry. I recall "a__a" being bad. But past that, I do barely anything past "putsomeletters.example.com" for A records.

[peteeckel]

Mind you, I also have no idea what is an invalid DNS entry. I recall "a__a" being bad. But past that, I do barely anything past "putsomeletters.example.com" for A records.

It's complicated :-)

I'm not sure I got it completely right, as there is a ton of rules dispersed over multiple RFCs, and some of the rules contradict each other. To make it worse, there are certain implementors (e.g. Microsoft) ignoring some of them in their products, while others (e.g. ISC) enforce them and don't accept the ones the first group generates by default. The validation code in NetBox DNS tries to 'do the right thing', but it's not a pretty sight.

On the other hand, NetBox' regex is really a bit on the simple side. NetBox DNS does regex checking as well:

[...]
LABEL = r"[a-z0-9][a-z0-9-]*(?<!-)"
UNDERSCORE_LABEL = r"[a-z0-9][a-z0-9-_]*(?<![-_])"
LEADING_UNDERSCORE_LABEL = r"[a-z0-9_][a-z0-9-]*(?<!-)"


def has_invalid_double_dash(name):
    return bool(re.findall(r"\b(?!xn)..--", name, re.IGNORECASE))


def validate_fqdn(name):
    if get_plugin_config("netbox_dns", "tolerate_underscores_in_hostnames"):
        regex = rf"^(\*|{UNDERSCORE_LABEL})(\.{UNDERSCORE_LABEL})+\.?$"
    else:
        regex = rf"^(\*|{LABEL})(\.{LABEL})+\.?$"

    if not re.match(regex, name, flags=re.IGNORECASE) or has_invalid_double_dash(name):
        raise ValidationError(f"Not a valid fully qualified DNS host name")


def validate_extended_hostname(name, tolerate_leading_underscores=False):
    if tolerate_leading_underscores:
        regex = rf"^([*@]|(\*\.)?{LEADING_UNDERSCORE_LABEL}(\.{LEADING_UNDERSCORE_LABEL})*\.?)$"
    elif get_plugin_config("netbox_dns", "tolerate_underscores_in_hostnames"):
        regex = rf"^([*@]|(\*\.)?{UNDERSCORE_LABEL}(\.{UNDERSCORE_LABEL})*\.?)$"
    else:
        regex = rf"^([*@]|(\*\.)?{LABEL}(\.{LABEL})*\.?)$"

    if not re.match(regex, name, flags=re.IGNORECASE) or has_invalid_double_dash(name):
        raise ValidationError(f"Not a valid DNS host name")


def validate_domain_name(name):
    if name == "." and get_plugin_config("netbox_dns", "enable_root_zones"):
        return

    if get_plugin_config("netbox_dns", "tolerate_underscores_in_hostnames"):
        regex = rf"^{UNDERSCORE_LABEL}(\.{UNDERSCORE_LABEL})*\.?$"
    else:
        regex = rf"^{LABEL}(\.{LABEL})*\.?$"

    if not re.match(regex, name, flags=re.IGNORECASE) or has_invalid_double_dash(name):
        raise ValidationError(f"Not a valid DNS domain name")

after running all names through validation with dnspython. You get the picture ...

[peteeckel]

Mind you, I also have no idea what is an invalid DNS entry. I recall "a__a" being bad. But past that, I do barely anything past "putsomeletters.example.com" for A records.

By the way, a__a is either totally illegal because it contains underscores which are not accepted by, e.g. BIND, or it's fine if you accept underscores as legal. a--a is fine as well, but aa--a is not, while xn--a is (but, just to confuse you, xn--b is not :-)). Now go figure out the rules ...

Oh, I forgot ... even BIND does accept some underscores ... _xmpp._tcp.example.com. is fine for a SRV record, for example.

[zeddD1abl0]

a--a is fine as well, but aa--a is not, while xn--a is (but, just to confuse you, xn--b is not :-)). Now go figure out the rules ...

This is the thing I remember. Weird "double-dash but not but also" rules are awesome. I leave it to people more capable than me... you. :)

[peteeckel]

Thanks for your trust in my capabilities ... sometimes I'm not so sure myself.

But anyway, you now see the problem with using IPAM dns_name as a data source for NetBox DNS, and that's pretty much the reason why @jean1 did not do it - at least that's what I think was his reason. A while ago I created two custom scripts, one for updating NetBox DNS with data from IPAM and one for the other direction. As it turned out, the direction using IPAM as the data source did not work that well.

That said, we will have to do some error handling, and still we won't be able to create all legal DNS names that way. So I guess the only way to do it properly is to ask the NetBox team to fix their regex and use dnspython for additional validation. I'll open an issue as soon as I have some spare time to properly work it out.

[peteeckel]

For all who haven't noticed yet: NetBox DNS 0.20.0 has @jean1's experimental feature for linking IP addresses to DNS address records now. I played around with it over the weekend and yesterday, and I think it's very useful for many use cases and pushes NetBox DNS a major step forward.

Please try it out and write about your experiences. It doesn't have to stay experimental for too long :-)

[kollross]

We had some discussion whether it would be a good idea to retain existing address records or not, essentially converting them to managed address records when an IP address with a matching name and zone is created. I wanted the behaviour of the plugin to be conservative and not modify existing records that had been entered manually, so the outcome was that if you already have an A or AAAA record with a given name, the IPAM Coupling feature will add another one with the same name, this time as a managed record.

I did not see that as the case. Unless I did something wrong, the only records that showed up in the managed records were @ and NS records from all the zones, I had configured, no A records. Unless I misunderstanding what you are saying.

I suppose that this is what bites you. The issue is not dual-stacking but you already have an address record of the same type (A or AAAA) with the same name, so when NetBox DNS tries to create another one with the same name (and type) it fails. Can you confirm that? If so, please set that record to inactive (or delete it, which is eventually what you will want to do) and retry. In my short test that works.

I was able to work around the V6 issue by not setting the DNS Host fields while applying the v6 address to a Device.

[peteeckel]

The conflicting A/AAAA record is not a managed record but a normal one, since you entered it manually at some point. I can reproduce exactly that constellation. You can't manually define managed records, that's their whole point.

[kollross]

I don't think I was hitting the managed records issue. Some form of operations that I was doing was hitting the unique requirement. I was adding a V6 address to an existing device. Though I can't reproduce what I was doing earlier.

[peteeckel]

I just found an error in the uniqueness check, but that shouldn't have been the cause of your problem - unless your IPv6 record and your IPv4 record have the same value, which is hard to accomplish :-)

Please test with 0.20.1 anyway once I'm done releasing it.

[kollross]

Is there a way to "re-run" the conversion from what's in the unmanaged records to managed records? I looked and it seemed to correctly create all the PTR records for all of the A records that previously existed, but none of the A records were copied into the managed records area. All the "A" records that previously existed were also set to the DNS Host field on the IP address.

[peteeckel]

There is no conversion. Unmanaged records stay unmanaged, managed records are added independently. If you have unmanaged records that clash with managed ones, set them to 'inactive' and you're fine.

Existing records should not be touched, at least unless there is an error in the new code I didn't find yet.

Did you find unmanaged A or AAAA records that were converted by the new functionality?

Another information: PTR records are created per address record, there is a 1:1 relationship between them.

[kollross]

I started off with a couple thousand unmanaged records. All of those unmanaged A records had the corresponding FDQN set has the DNS Hostname field for the IP.

After upgrading the plugin, I ended up with a a hundred or so managed records, all where NS and @ records. No A or AAAA records existed in the managed records section.

Based on what you said before "so the outcome was that if you already have an A or AAAA record with a given name, the IPAM Coupling feature will add another one with the same name, this time as a managed record." it sounded like there should have been a copy of those A/AAAA records created in the managed records section, but this doesn't appear to have happened.

[peteeckel]

Ah. That was a misunderstanding.

The IPAM coupling feature leaves the address records as they are. If you create an IP address that would result in an address record with the same name, that record is created and the existing one is left intact. Unless, as you noticed, you have the enforce_unique_records setting enabled, which results in the exception you've seen if it would create a duplicate record (managed or unmanaged doesn't matter for that decision).

The issue I mentioned earlier is that you actually see the exception in the NetBox GUI. You shouldn't, that kind of error should be caught in the clean() function and displayed in the input form as usual. I'm working on that, but it's not that easy as I need to rewrite part of the central code of the coupling function. That, however, doesn't have anything to do with the problem you are seeing, and which is very probably a result of the enforce_unique_records setting clashing with automatic address record generation.

[kollross]

OK so to confirm, there is no way to convert existing unmanaged records to managed records, nor is there anyways to convert the existing DNS Hostname field on the IP address to a managed record.

[peteeckel]

As far as the current implementation is concerned, both statements are correct.

For reasons explained a couple of times, IPAM 'DNS Name' is currently unsuitable as a source for generating DNS records.

Given the fact that this functionality is currently labeled 'experimental', it does not (or at least should not, unless something goes wrong) alter or remove any existing DNS records.

There still are some rough edges, especially with error handling. Once that's sorted out the feature will no longer be experimental, but it may take some iterations of fixes.

[kollross]

Understood I just wanted clarification. I'll just need to think of good way (probably via API) to convert the 3000 records in the "unmanaged records" section to the new managed section.

[peteeckel]

So I think I now see your use case.

First things first: If you run on production data (which I assume you do), have a backup of the database. Then, make another one, just in case.

There are several ways to achieve what you want, but all of them depend on the structure of the data you have and the integrity and consistency of your IPAM data vs. NetBox DNS data. From what you wrote, my understanding of your use case is as follows:

• You have a couple of thousand IPAddress objects in IPAM, which have DNS Names associated with them
• You also have DNS address records in NetBox DNS, which are (more or less) consistent with the DNS information in IPAM
• You have the enforce_unique_records setting enabled in NetBox DNS
• You want to unify the data using the new IPAM Coupling feature
• You don't have multiple views in NetBox DNS
• You don't have too many manually created DNS address records (A and/or AAAA) in NetBox DNS

You can use the API for this, but I would go for a custom script in this case. You can start with the example script DNSRecordUpdater in the examples directory, but you'll need to make some adjustments:

• Loop over all IPAddress objects in the ipam.IPAddress model (that's unchanged from the example)
• Extend the record_filter list (line 135) with managed=False
• Instead of creating a new address record (line 150), set the custom fields ipaddress_dns_record_name and ipaddress_dns_zone_id by splitting your dns_name in name and domain and querying for the correct zone id for the domain part
• If you find an existing unmanaged address record in NetBox DNS, set it to inactive instead of updating it (line 160ff.).

That should get you started. IMHO it's much easier (and definitely faster) than using the API, but your mileage may vary.

And have that backup handy :-)

[peteeckel]

Current State of Affairs and Perspective

As you may have noticed, we made some substantial progress in this matter, mainly thanks to this month's contribution from @jean1, which is a large step forward.

I just released version 0.20.2 based on Jean's work, which improves on the validation of DNS data entered and straightens some edges, and for some use cases this will already be a good solution.

But please be aware that this feature is still marked experimental, which means it can go away at any time (not likely at all) or change substantially (very likely). There will probably be migration paths, but my capacities are limited and so support for experimental features is based on best effort. At the very least, have a backup (I know this is boring).

Implemented Functionality

With the new experimental 'IPAM Coupling' feature it is possible to assign a zone and a name to any IP address, which will then result in an address record with the name provided will be created in the selected zone. That record is coupled to the IP address in a way that changes in the IP Address, the record name or the zone are reflected in the address record. Vice versa, the address record cannot be edited (it is implemented as a managed record).

Deleting the IP address will also delete the address record. Vice versa, deleting the DNS zone will remove the assignment from the IP address to the address record and clear the 'DNS name' field in the IP address object.

Uncovered Requirements

While this will cover many, especially simple, use cases, there are still some requirements that can not be covered with this concept. Most of them occur in more complex environments, i.e. when views and VRFs are involved, and some of them are related to large numbers of objects to be handled.

• The use case mentioned above: You have IPv4 addresses with NAT and global IPv6 addresses in your network, and use a split zone DNS with two views to cover the NAT case. The 'internal' view maps host names to internal IPv4 addresses (post-NAT), the 'external' view does the same with external IPv4 addresses, and both views have the IPv6 addresses (or a subset thereof). This would require to be able to assign multiple DNS zones (in this example, one in view 'internal' and one in view 'external') to the same IP address.
• Automatic import of data: The implementation using custom fields limits bulk import data for objects like coupled zones to their IP. I did not find a way to use the zone name for the 'Zone' custom field yet, which makes importing large amounts of data more difficult than desirable.
• The custom field mechanism itself is, though it is really neat for assigning data to core objects like IP addresses, limited in some ways, starting with the necessity to define them and the havoc that will result when someone deletes one of the coupling custom fields (did I mention backups?).
• It would be helpful to be able to assign DNS record name and zone from existing data - that is, 'DNS name'. But as was discussed a couple of times that alone won't do the trick, think VRFs and Views, and has severe limitations because of the way that field is validated.
• to be continued ...

Next Steps

During integration of the current functionality and following the comment from @zeddD1abl0 I got some ideas how we can improve on the remaining issues, and maybe solve them in a more extensive way.

• The mechanisms implemented by @jean1 provide us with options to improve on the validation of the IPAM dns_name field. Using the post_clean handler in Jean's middleware, we can validate the data entered there much better, making them safer to use as an input for NetBox DNS. We can't resolve the issue that dns_name does not allow, for example, IDNs, but we can make validation of the data entered sufficiently strict that it can be used for NetBox DNS. So dns_name just got promoted to 'candidate for replacing the IMAP IPAddress custom fields'. That's a major step forward.
• @zeddD1abl0 suggested using IPAM Prefixes for mapping IPAddress to NetBox DNS Views. I never thought of that before, and I haven't thought it through yet, but it might be a brilliant idea. Assigning Prefixes to DNS Views solves the issue with mixed VRF/Views for IPv4 and IPv6 I described above, and it might solve most other issues as well together with the option of mapping multiple views to one prefix.
• @jean1's mechanisms in the IPAM Coupling middleware can easily be adjusted to that model, and doesn't need large adjustments to work with the new idea as well.

The idea that is currently forming is as follows:

• Extend the View model in NetBox DNS with a many-to-many relation to ipam.Prefix. That would make it possible to assign multiple prefixes to each view. The challenge is that Views are optional, so probably the best path forward is to first make the default View (currently None) explicit without breaking compatibility. I'll have to think about that, but it would remove some other ugliness in the code as well.
• Remove the custom fields introduced by @jean1, replacing them with data entered via 'DNS Name'. That makes it necessary to properly validate the 'DNS Name' first, but we now have the mechanisms required for that in place. That way, we can get the record name and zone from dns_name (it's not a simple split, but we can do it pretty easily using mechanisms already present in dnspython and NetBox DNS). This involves lookups in all the views associated with a given prefix and finding all name/zone combinations that apply. Observing hierarchies of Prefixes, default settings if no prefix matches etc. can be implemented as well.
• Essentially that's it - we now have a list of zones that we need to insert the name to. The rest has already been done in the current code.

Any thoughts?

[zeddD1abl0]

Thank you for the call-out. I had a thought about the issue with names vs RFCs, and I think the better solution might be to just import the entries that fit the restrictions and then have a list of "unimportable" entries with links to their IPAM entry and a short reason why they couldn't be imported. Sort of like how the zones currently report bad NS entries, and other things. It might be useful to implement a warnings/errors page.

One thing I did think about; how do you deal with entries that are across multiple zones. Like the scenario:

10.0.0.1 - host.example.com
10.0.0.2 - host.example.io

I had thought initially that the IPAM couping would be simply "This prefix is connected to this zone, any hostnames that end in the zone, will be added to the zone (nearest match to hostname possible)." but it appears that this is not the way the Coupling currently functions. I don't know how you'd handle this in the future, and it's unclear if there's a major issue here, but it was a thought I had.

Otherwise, I'm very curious to see how this evolves. I had a quick dream about being able to do site-specific DNS with NAT, etc. But that's a very complex, almost silly, idea currently. For the moment though, the current implementation works as expected, and works well.

[peteeckel]

@zeddD1abl0, don't mind the way the mapping is working right now, that's a different pair of shoes - manual vs. automatic assignment, that is. Currently we define a name and select a zone, which is fine but may become tedious for large data sets. The solution at the moment is external automation tools, but my idea is to have NetBox DNS do the work.

I had thought initially that the IPAM couping would be simply "This prefix is connected to this zone, any hostnames that end in the zone, will be added to the zone (nearest match to hostname possible)."

My interpretation is a little bit different: We map prefixes to views (by maintaining a list of connected prefixes per View object, "no view" becoming a view of its own).

Now when you assign an IP address and add a dns_name, we can try to find the domain within the view that provides the longest match with dns_name, which is now validated against the NetBox DNS naming rules, which means it will be valid depending on settings like tolerate_underscores_in_hostnames etc., so NetBox DNS will accept it.

Let's say in your example the prefix 10.0.0.0/24 is connected to the view 'internal' and the zones example.com and example.io are defined in that view, that query would return ('host', Zone<[internal] example.com>) for the first IP address and ('host', Zone<[internal] example.io>) for the second one. Now we can create the two A records and link them to their respective IP address.

One thing comes to my mind right now: We then need a setting 'Do not assign records for IPAM addresses' on the zones, otherwise anyone using the permit_root_zone feature will be in for a lot of fun :-) ... or we don't allow multi-label host names to be added to zones by IPAM Coupling at all. Otherwise host.lab.example.io would be added as ('host.lab', Zone<[internal] example.io>) (minor issue) and any arbitrary host would be added as ('host.sub.domain.tld', Zone<[internal] .>) (major mess-up :-)).

Afterthought: A minimum label count for the domain name (default 2) should do as well.

[zeddD1abl0]

Now when you assign an IP address and add a dns_name, we can try to find the domain within the view that provides the longest match with dns_name, which is now validated against the NetBox DNS naming rules, which means it will be valid depending on settings like tolerate_underscores_in_hostnames etc., so NetBox DNS will accept it.

That's exactly what I'd want. I have some thoughts around "Same domain but two different views" stuff, but I need to get it straight in my mind (and get a heap of other stuff done first0 and then I'll come back with a discussion point.

[peteeckel]

Technically, the same zone within two different views is not the same zone, it's two zones with the same name, but different views. [edited out the word 'domain', which can be misleading]