Configure HTTP-based ARP information fetching from Palo Alto PAN-OS firewalls using management profiles

NOTES.rst

@@ -20,6 +20,37 @@ to be able to upgrade to Python 3.11:
 * :mod:`django-crispy-forms`
 * :mod:`crispy-forms-foundation`
 
+
+NAV 5.12
+========
+Deprecation warnings
+--------------------
+.. warning:: The ``[paloaltoarp]`` section of :file:`ipdevpoll.conf`, used for
+             configuring HTTP-based ARP fetching from Palo Alto firewalls, is
+             deprecated and will be ignored in NAV 5.12 and future versions.
+             HTTP-based ARP fetching from Palo Alto
+             firewalls *must* now be configured using management profiles,
+             analogous to configuration of SNMP-based fetching.  :ref:`See below
+             for more details<5.12-new-http-rest-api-management-profile-type>`.
+
+.. _5.12-new-http-rest-api-management-profile-type:
+New way to configure fetching of Palo Alto firewall ARP cache data
+------------------------------------------------------------------
+.. NOTE:: See
+          :ref:`management profile reference documentation<http-rest-api-management-profile>`
+          for instructions on how to reconfigure your Palo Alto firewall
+          devices in NAV 5.12 to enable support for fetching of their
+          ARP information.
+
+Starting with NAV 5.12, a new ``HTTP API`` management profile type has been
+added to NAV for configuring HTTP API specific parameters used in fetching of
+ARP information from Palo Alto firewalls running PAN-OS. Currently, this
+management profile type is only used to configure Palo Alto firewall devices. If
+support for other devices that similarly can be managed using a HTTP API is
+added to NAV in future releases, you can expect to be able to configure API
+parameters for these devices by using management profiles as well.
+
+
 NAV 5.11
 ========
 

changelog.d/3147.changed.md

@@ -0,0 +1,3 @@
+The ipdevpoll plugin to fetch ARP cache data from a netbox's Palo Alto firewall
+API is now configured through a new management profile type assigned to that
+netbox.

doc/reference/ipdevpoll.rst

@@ -106,35 +106,6 @@ Section [linkstate]
   The value ``any`` will generate alerts for all link state changes, but
   **this is not recommended** for performance reasons.
 
-Section [paloaltoarp]
----------------------
-
-This section configures the Palo Alto ARP plugin.  Palo Alto firewalls do
-support SNMP.  They do not, however, support fetching ARP cache data using
-SNMP.  This plugin enables fetching ARP records from Palo Alto firewalls using
-their built-in REST API.
-
-Currently, there is no management profile type for this type of REST APIs, so
-credentials to access a Palo Alto firewall's API must be configured in this
-section.
-
-If you have a Palo Alto firewall named ``example-fw.example.org``, with an IP
-address of ``10.0.42.42`` and a secret API token of
-``762e87e0ec051a1c5211a08dd48e7a93720eee63``, you can configure this in this
-section by adding::
-
-  example-fw.example.org = 762e87e0ec051a1c5211a08dd48e7a93720eee63
-
-Or, alternatively::
-
-  10.0.42.42 = 762e87e0ec051a1c5211a08dd48e7a93720eee63
-
-
-.. warning:: The Palo Alto ARP plugin does not currently verify TLS
-             certificates when accessing a Palo Alto API.  This will be changed
-             at a later date, but if it worries you, you should not use the
-             plugin yet.
-
 
 Job sections
 ------------

doc/reference/management-profiles.rst

@@ -86,7 +86,38 @@ Use keys
 Alternate port
   If access to the switch is not on the default port (22, in the case of the
   JunOS driver), put the alternate port here.
-  
+
 
 .. _`NAPALM`: https://napalm.readthedocs.io/en/latest/
 .. _`NETCONF`: https://en.wikipedia.org/wiki/NETCONF
+
+.. _http-rest-api-management-profile:
+HTTP APIs
+--------------
+As of NAV 5.12, HTTP API profiles are used to configure access to
+services of the following devices.
+
+`Palo Alto PAN-OS firewalls`_
+  A HTTP API profile is needed for NAV to access the firewall's ARP information.
+
+.. warning:: The Palo Alto ARP implementation in NAV does not currently verify TLS
+             certificates when accessing a Palo Alto API.  This will be changed
+             at a later date, but if it worries you, you should not configure
+             any netboxes to use the Palo Alto Arp service yet.
+
+.. image:: http-rest-api-profile-example.png
+
+If you have a Palo Alto firewall running on a netbox managed by NAV,
+with a secret API key of ``762e87e0ec051a1c5211a08dd48e7a93720eee63``,
+you can configure NAV to fetch ARP information from this firewall by
+creating a new management profile with
+
+* protocol set to ``HTTP API``,
+
+* API key set to ``762e87e0ec051a1c5211a08dd48e7a93720eee63``,
+
+* service set to ``Palo Alto ARP``,
+
+and then add this management profile to the netbox.
+
+.. _`Palo Alto PAN-OS firewalls`: https://docs.paloaltonetworks.com/pan-os/11-0/pan-os-panorama-api/pan-os-xml-api-request-types/configuration-api/get-active-configuration/use-xpath-to-get-arp-information

python/nav/ipdevpoll/plugins/paloaltoarp.py

@@ -15,14 +15,12 @@
 # License along with NAV. If not, see <http://www.gnu.org/licenses/>.
 #
 
-"""ipdevpoll plugin for fetching arp mappings from Palo Alto firewalls
-
-Add [paloaltoarp] section to ipdevpoll.conf
-add hostname = key to [paloaltoarp] section
-for example:
-[paloaltoarp]
-10.0.0.0 = abcdefghijklmnopqrstuvwxyz1234567890
+"""
+ipdevpoll plugin for fetching arp mappings from Palo Alto firewalls
 
+Configure a netbox to work with this plugin by assigning it a
+HTTP_API management profile with service set to "Palo Alto ARP"
+in seedDB.
 """
 
 import xml.etree.ElementTree as ET
@@ -35,68 +33,67 @@
 from twisted.web.http_headers import Headers
 
 from nav import buildconf
+from nav.ipdevpoll import db
 from nav.ipdevpoll.plugins.arp import Arp
+from nav.models.manage import Netbox, ManagementProfile, NetboxProfile
 
 
 class PaloaltoArp(Arp):
-    configured_devices: dict[str, str] = {}
-
-    @classmethod
-    def on_plugin_load(cls):
-        """Loads the list of PaloAlto access keys from ipdevpoll.conf into the plugin
-        class instance, so that `can_handle` will be able to answer which devices
-        this plugin can run for.
-        """
-        from nav.ipdevpoll.config import ipdevpoll_conf
-
-        cls._logger.debug("loading paloaltoarp configuration")
-        if 'paloaltoarp' not in ipdevpoll_conf:
-            cls._logger.debug("PaloaltoArp config section NOT found")
-            return
-        cls._logger.debug("PaloaltoArp config section found")
-        cls.configured_devices = dict(ipdevpoll_conf['paloaltoarp'])
-
     @classmethod
+    @defer.inlineCallbacks
     def can_handle(cls, netbox):
         """Return True if this plugin can handle the given netbox."""
-        return (
-            netbox.sysname in cls.configured_devices
-            or str(netbox.ip) in cls.configured_devices
-        )
+        has_configurations = yield cls._has_paloalto_configurations(netbox)
+        returnValue(has_configurations)
 
     @defer.inlineCallbacks
     def handle(self):
         """Handle plugin business, return a deferred."""
-
-        api_key = self.configured_devices.get(
-            str(self.netbox.ip), self.configured_devices.get(self.netbox.sysname, "")
-        )
         self._logger.debug("Collecting IP/MAC mappings for Paloalto device")
 
-        mappings = yield self._get_paloalto_arp_mappings(self.netbox.ip, api_key)
-        if mappings is None:
-            self._logger.info("No mappings found for Paloalto device")
-            returnValue(None)
+        configurations = yield self._get_paloalto_configurations(self.netbox)
+        for configuration in configurations:
+            mappings = yield self._get_paloalto_arp_mappings(
+                self.netbox.ip, configuration["api_key"]
+            )
+            if mappings:
+                yield self._process_data(mappings)
+                break
 
-        yield self._process_data(mappings)
+    @staticmethod
+    @db.synchronous_db_access
+    def _has_paloalto_configurations(netbox: Netbox):
+        """
+        Make a database request to check if the netbox has any management
+        profile that configures access to Palo Alto ARP data via HTTP
+        """
+        queryset = _paloalto_profile_queryset(netbox)
+        return queryset.exists()
 
-        returnValue(None)
+    @staticmethod
+    @db.synchronous_db_access
+    def _get_paloalto_configurations(netbox: Netbox):
+        """
+        Make a database request that fetches all management profiles of
+        the netbox that configures access to Palo Alto ARP data via HTTP
+        """
+        queryset = _paloalto_profile_queryset(netbox)
+        return list(queryset)
 
     @defer.inlineCallbacks
-    def _get_paloalto_arp_mappings(self, address: str, key: str):
-        """Get mappings from Paloalto device"""
-
+    def _get_paloalto_arp_mappings(self, address: IP, key: str):
+        """
+        Make a HTTP request for ARP data from Paloalto device with the given
+        ip-address, using the given api-key. Returns a formatted list of ARP
+        mappings for use in NAV.
+        """
         arptable = yield self._do_request(address, key)
-        if arptable is None:
-            returnValue(None)
-
-        # process arpdata into an array of mappings
-        mappings = parse_arp(arptable.decode('utf-8'))
+        mappings = _parse_arp(arptable) if arptable else []
         returnValue(mappings)
 
     @defer.inlineCallbacks
-    def _do_request(self, address: str, key: str):
-        """Make request to Paloalto device"""
+    def _do_request(self, address: IP, key: str):
+        """Make an HTTP request to a Palo Alto device"""
 
         class SslPolicy(client.BrowserLikePolicyForHTTPS):
             def creatorForNetloc(self, hostname, port):
@@ -127,16 +124,15 @@ def creatorForNetloc(self, hostname, port):
         returnValue(response)
 
 
-def parse_arp(arp):
+def _parse_arp(arpbytes: bytes) -> list[tuple[str, IP, str]]:
     """
     Create mappings from arp table
     xml.etree.ElementTree is considered insecure: https://docs.python.org/3/library/xml.html#xml-vulnerabilities
     However, since we are not parsing untrusted data, this should not be a problem.
     """
-
     arps = []
 
-    root = ET.fromstring(arp)
+    root = ET.fromstring(arpbytes.decode("utf-8"))
     entries = root.find("result").find("entries")
     for entry in entries:
         status = entry.find("status").text
@@ -147,3 +143,18 @@ def parse_arp(arp):
                 arps.append(('ifindex', IP(ip), mac))
 
     return arps
+
+
+def _paloalto_profile_queryset(netbox: Netbox):
+    """
+    Creates a Django queryset which when iterated yields JSON dictionaries
+    representing configurations for accessing Palo Alto ARP data of the given
+    netbox via HTTP. The keys in these dictionaries are the attribute-names of
+    the :py:class:`~nav.web.seeddb.page.management_profile.forms.HttpRestForm`
+    Django form.
+    """
+    return NetboxProfile.objects.filter(
+        netbox_id=netbox.id,
+        profile__protocol=ManagementProfile.PROTOCOL_HTTP_API,
+        profile__configuration__contains={"service": "Palo Alto ARP"},
+    ).values_list("profile__configuration", flat=True)

python/nav/models/manage.py

@@ -131,10 +131,12 @@ class ManagementProfile(models.Model):
     PROTOCOL_SNMP = 1
     PROTOCOL_NAPALM = 2
     PROTOCOL_SNMPV3 = 3
+    PROTOCOL_HTTP_API = 4
     PROTOCOL_CHOICES = [
         (PROTOCOL_SNMP, "SNMP"),
         (PROTOCOL_NAPALM, "NAPALM"),
         (PROTOCOL_SNMPV3, "SNMPv3"),
+        (PROTOCOL_HTTP_API, "HTTP API"),
     ]
     if settings.DEBUG:
         PROTOCOL_CHOICES.insert(0, (PROTOCOL_DEBUG, 'debug'))

python/nav/web/seeddb/page/management_profile/forms.py

@@ -62,6 +62,28 @@ def _post_clean(self):
                 cfg[field] = self.cleaned_data.get(field)
 
 
+class HttpApiForm(ProtocolSpecificMixIn, forms.ModelForm):
+    PROTOCOL = ManagementProfile.PROTOCOL_HTTP_API
+    PROTOCOL_NAME = PROTOCOL_CHOICES.get(PROTOCOL)
+
+    class Meta(object):
+        model = ManagementProfile
+        configuration_fields = ['api_key', 'service']
+        fields = []
+
+    api_key = forms.CharField(
+        label="API key",
+        help_text="Key/token to authenticate to the service",
+        required=True,
+    )
+
+    service = forms.ChoiceField(
+        choices=(("Palo Alto ARP", "Palo Alto ARP"),),
+        help_text="",
+        required=True,
+    )
+
+
 class DebugForm(ProtocolSpecificMixIn, forms.ModelForm):
     PROTOCOL = ManagementProfile.PROTOCOL_DEBUG
     PROTOCOL_NAME = PROTOCOL_CHOICES.get(PROTOCOL)