Merge branch 'master' into snmp-version

.github/workflows/dockercompose.yml

@@ -31,3 +31,17 @@ jobs:
         if: failure()
         run: |
           docker-compose logs
+
+      - name: Send GitHub Action trigger data to Slack workflow
+        id: slack
+        if: failure()
+        uses: slackapi/[email protected]
+        with:
+          payload: |
+            {
+              "title": "Building NAV's Docker Compose dev environment failed",
+              "github_ref_name": "${{ github.ref_name }}",
+              "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

AUTHORS.rst

@@ -24,6 +24,10 @@ Currently active contributors/maintainers
   Joined in 2021. Currently hacking away at open issues and learning the NAV
   codebase.
 
+* Ilona Podliashanyk <ilona.podliashanyk at sikt.no>
+  Joined in 2022, working mainly as a front-end developer. Currently hacking
+  away at open issues and learning the NAV codebase.
+
 Other contributors and previous maintainers
 -------------------------------------------
 * Vidar Faltinsen <vidar.faltinsen at sikt.no>
@@ -40,6 +44,10 @@ Other contributors and previous maintainers
   Active from 2017, until he left Uninett in 2019. An experienced Python
   developer who, among other things, rewrote the ipdevpoll multiprocess mode.
 
+* Ragnhild Bodsberg
+  Contributed various bugfixes to NAV as an intern at Sikt, during the summer
+  of 2022.
+
 * Philipp Petermann <philipp.petermann at unibas.ch>
   Contributed support for enabling CDP when configuring Cisco Voice VLANs in
   PortAdmin.

CHANGELOG.md

@@ -11,19 +11,73 @@ found in the [HISTORY](HISTORY) file.
 
 ## Fixed
 
-- Metric values of 0.0 are evaluated correctly by threshold rules [#2447](https://github.com/Uninett/nav/issues/2447)
+- Merge two fixes from the 5.4.x stable series that never actually made it into the 5.5 series:
+  - Metric values of 0.0 are evaluated correctly by threshold rules [#2447](https://github.com/Uninett/nav/issues/2447)
+  - Validate maintenance calendar input form to avoid e-mail spam from bots scanning for vulnerabilities [#2420](https://github.com/Uninett/nav/issues/2420)
+
+
+## [5.5.2] - 2022-11-10
+
+### Fixed
+
+- Fix serious collection breakdown in ipdevpoll by re-generating a valid Python representation of CISCO-ENHANCED-MEMPOOL-MIB ([#2494](https://github.com/Uninett/nav/issues/2494), [#2495](https://github.com/Uninett/nav/pull/2495))
+- Fix broken trap processing in snmptrapd ([#2497](https://github.com/Uninett/nav/issues/2497), [#2498](https://github.com/Uninett/nav/pull/2498))
+
+
+## [5.5.1] - 2022-11-09
+
+### Fixed
+
+- Delete and ignore module devices with fake serial number `BUILTIN`, as reported by Juniper equipment, in order to avoid spamming with `device[SFH]wUpgrade` alerts ([#2491](https://github.com/Uninett/nav/issues/2491), [#2492](https://github.com/Uninett/nav/pull/2492))
+
+## [5.5.0] - 2022-11-04
+
+### Changed
+
+- Bump `lxml` from 4.6.5 to 4.9.1 in /tests ([#2443](https://github.com/Uninett/nav/pull/2443))
+- Links and documented references to the NAV mailing lists have changed to the `lister.sikt.no` domain.
+
+### Added
+
+- Add link to #nav irc channel on Libera.Chat to README file ([#2475](https://github.com/Uninett/nav/pull/2475))
+- Add `mac_addresses` attribute to `/netbox/` API endpoint ([#2487](https://github.com/Uninett/nav/pull/2487), [#2490](https://github.com/Uninett/nav/pull/2490))
+- Add ability to filter by alert severity in the status tool ([#2467](https://github.com/Uninett/nav/pull/2467))
+- Support for fetching ARP cache entries from all Arista VRF instances ([#2262](https://github.com/Uninett/nav/issues/2262), [#2454](https://github.com/Uninett/nav/pull/2454)))
+- Link aggregation information added to NAV API ([#1765](https://github.com/Uninett/nav/issues/1765), [#2440](https://github.com/Uninett/nav/pull/2440))
+- Support fetching memory stats from `CISCO-ENHANCED-MEMPOOL-MIB` ([#2236](https://github.com/Uninett/nav/issues/2236), [#2439](https://github.com/Uninett/nav/pull/2439))
+- Added a flag to `navcheckservice` that shows all available handler plugins ([#2378](https://github.com/Uninett/nav/issues/2378), [#2437](https://github.com/Uninett/nav/pull/2437))
+- Post `deviceHwUpgrade`/`deviceSwUpgrade`/`deviceFwUpgrade` events when changes are detected to devices' hardware, software or firmware revisions ([#2393](https://github.com/Uninett/nav/issues/2393), [#2413](https://github.com/Uninett/nav/pull/2413))
+- Call a `cleanup()` method for individual container objects after ipdevpoll save stage ([#2421](https://github.com/Uninett/nav/pull/2421))
+- Added `Device` methods to resolve and return related objects/entities (chassis, modules, fans, power supplied) and extended device descriptions ([#2428](https://github.com/Uninett/nav/pull/2428))
+
+### Fixed
+
+- Avoid potential resource leaks by properly closing configuration files after reading them ([#2451](https://github.com/Uninett/nav/pull/2451))
+- Geomap "link to this configuration" now actually opens the correct location at the correct zoom level ([#2412](https://github.com/Uninett/nav/issues/2412), [#2488](https://github.com/Uninett/nav/pull/2488))
+- snmptrapd can now identify an SNMP agent from any of its interface addresses ([#2387](https://github.com/Uninett/nav/issues/2387), [#2461](https://github.com/Uninett/nav/pull/2461))
+- PortAdmin now ignores incorrectly configured VLAN tags (tagged as `NA`) on Juniper switches, instead of crashing ([#2452](https://github.com/Uninett/nav/issues/2452), [#2453](https://github.com/Uninett/nav/pull/2453))
+- Fix potential ipdevpoll crashes due to database fetches in wrong thread ([#2478](https://github.com/Uninett/nav/issues/2478), [#2480](https://github.com/Uninett/nav/pull/2480))
+- Handle Graphite connection issues gracefully in ranked statistics page ([#2459](https://github.com/Uninett/nav/pull/2459))
+- Handle Graphite connection issues gracefully in device group detail page ([#2345](https://github.com/Uninett/nav/issues/2345), [#2434](https://github.com/Uninett/nav/pull/2434))
+- Removed needless carbon data chunking from `activeipcollector` ([#1696](https://github.com/Uninett/nav/issues/1696), [#2462](https://github.com/Uninett/nav/pull/2462))
+- Evaluate `0.0` as a valid numeric metric value during threshold rule evaluations ([#2447](https://github.com/Uninett/nav/issues/2447)
+- Updated dead links in Geomap documentation ([#2419](https://github.com/Uninett/nav/pull/2419))
+- Link from IPAM to reserve prefixed in SeedDB now works again ([#2410](https://github.com/Uninett/nav/issues/2410), [#2422](https://github.com/Uninett/nav/pull/2422))
+- Improved inefficient database queries in Arnold ([#2425](https://github.com/Uninett/nav/pull/2425))
+- Updated tox examples in hacking documentation ([#2427](https://github.com/Uninett/nav/issues/2427), [#2430](https://github.com/Uninett/nav/pull/2430))
+- Fixed an `AttributeError` crash bug in the `naventity` command line program ([#2433](https://github.com/Uninett/nav/issues/2433), [#2444](https://github.com/Uninett/nav/pull/2444))
 
 ## [5.4.0] - 2022-05-19
 
-## Changed
+### Changed
 
 - The changelog format has changed from the legacy format into one based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Transceivers are no longer classed as modules ([#1154](https://github.com/Uninett/nav/issues/1154))
 - Generate more secure API tokens ([#2366](https://github.com/Uninett/nav/issues/2366))
 - Remaining instances of "Uninett" in footer FAQ have changed to Sikt ([#2367](https://github.com/Uninett/nav/pull/2367))
 - Upgrade to napalm 3.4.1 ([#2403](https://github.com/Uninett/nav/pull/2403))
 
-## Added
+### Added
 
 - Support more recent AKCP environment probes ([#2107](https://github.com/Uninett/nav/issues/2107)
 - Collect and graph temperature readings from JUNIPER-MIB ([#2342](https://github.com/Uninett/nav/issues/2342))
@@ -33,7 +87,7 @@ found in the [HISTORY](HISTORY) file.
 - Document NAV's various command line utilities ([#2368](https://github.com/Uninett/nav/pull/2368))
 - Add a contrib script to ship ISC DHCP server lease stats to NAV's Graphite instance ([#2371](https://github.com/Uninett/nav/issues/2371))
 
-## Fixed
+### Fixed
 
 - Don't display JavaScript alert dialog box when generating links to the current Geomap configuration ([#1016](https://github.com/Uninett/nav/issues/1016))
 - Optimize SeedDB prefix listing queries ([#2156](https://github.com/Uninett/nav/issues/2156))
@@ -52,7 +106,7 @@ found in the [HISTORY](HISTORY) file.
 - Remove unnecessary quotation marks in SeedDB ([#2416](https://github.com/Uninett/nav/pull/2416))
 - Fix incorrect handling of stateless event posting in internal APIs ([#2417](https://github.com/Uninett/nav/pull/2417))
 
-## Removed
+### Removed
 
 - Remaining Python 2 compatibility code ([#2319](https://github.com/Uninett/nav/issues/2319))
 - Dependency on the `six` Python module

Dockerfile

@@ -85,7 +85,7 @@ COPY requirements/ /requirements
 COPY requirements.txt /
 COPY tests/requirements.txt /test-requirements.txt
 # Since we used pip3 to install pip globally, pip should now be for Python 3
-RUN pip-compile --output-file /requirements.txt.lock /requirements.txt /test-requirements.txt
+RUN pip-compile --resolver=backtracking --output-file /requirements.txt.lock /requirements.txt /test-requirements.txt
 RUN pip install -r /requirements.txt.lock
 
 ARG CUSTOM_PIP=ipython

NOTES.rst

@@ -8,6 +8,69 @@ existing bug reports, go to https://github.com/uninett/nav/issues .
 To see an overview of upcoming release milestones and the issues they resolve,
 please go to https://github.com/uninett/nav/milestones .
 
+NAV 5.5
+=======
+
+Dependency changes
+------------------
+
+None :-)
+
+API changes
+-----------
+
+The ``/netbox/`` API endpoint adds a new read-only attribute:
+``mac_addresses``. This is a list of MAC addresses associated with an IP
+Device's chassis, typically collected from either ``LLDP-MIB`` or
+``BRIDGE-MIB``. In most cases, devices will only have a single address here,
+but sometimes, the two MIBs will disagree on what is the main "chassis"
+address.
+
+The ``/interface`` endpoint has gained two new read-only attributes for LAG
+information:
+
+* ``aggregator``: An interface that is part of an aggregate will have the
+  aggregate interface specified here. The aggregator will be identified by both
+  its ``id`` and ``ifname`` attributes.
+* ``bundled_interfaces``: An interface that aggregates multiple interfaces will
+  have those interfaces listed here. Each interface in the list will be
+  identified by their ``id`` and ``ifname`` attrbutes.
+
+Software upgrade history
+------------------------
+
+NAV has finally regained the ability to save device software, firmware and/or
+hardware upgrades as events to their history, as the new ``deviceSwUpgrade``,
+``deviceFwUpgrade`` and ``deviceHwUpgrade`` alert types have been added to the
+``deviceNotice`` event hierarchy.  These alerts can now be subscribed to in
+Alert Profiles, and will be searchable in the Device History tool.  See also
+:doc:`reference/alerttypes` for the full list of events/alerts NAV provides.
+
+Juniper ``BUILTIN`` devices
+---------------------------
+
+Juniper equipment tends to report soldered-on linecards as field-replaceable
+modules through their implementation of ``ENTITY-MIB::entPhysicalTable``. These
+modules are also all reported as having the same serial number: ``BUILTIN``.
+
+NAV versions prior to 5.5.1 did not safeguard against this Juniper bug. This
+would cause NAV installations that monitor Juniper equipment to have a single
+device with the ``BUILTIN`` serial number, which was shared between all
+monitored Juniper netboxes.  The attributes of ``BUILTIN`` devices (such as
+software or firmware revision) would be different across most Juniper netboxes,
+causing them to compete for updates of the attributes in the NAV database.
+
+This went under the radar until NAV 5.5.0 re-introduced the ``device*Upgrade``
+set of alerts. Now, every time a Juniper netbox is polleed and the shared
+``BUILTIN`` device's software/hardware/firmware revision was changed, an alert
+would be generated. For those unfortunate enough to subscribe to all NAV
+alerts, this would lead to a storm of alerts.
+
+Subsequently, NAV 5.5.1 deletes this shared ``BUILTIN`` device from the
+database, and adds functionality to ignore any module or entity that reports
+this as its serial number.
+
+
 NAV 5.4
 =======
 

README.rst

@@ -13,6 +13,9 @@
 .. image:: https://img.shields.io/badge/code%20style-black-000000.svg
    :alt: Code style: black
    :target: https://github.com/psf/black
+.. image:: https://img.shields.io/badge/irc.libera.chat-%23nav-blue.svg
+   :alt: #[email protected]
+   :target: https://web.libera.chat/?channel=#nav
 
 
 This is NAV - Network Administration Visualized - an advanced software suite

doc/faq/faq.rst

@@ -17,7 +17,7 @@ these have been a mainstay in the Norwegian higher education sector for years.
 
 For any other vendors, we suggest that you just add your devices to NAV and
 see whether NAV reports what you want it to report. If it doesn't, submit a
-question to the mailing list, nav-users@uninett.no.
+question to the mailing list, nav-users@lister.sikt.no.
 
 Why are there gaps in my graphs?
 --------------------------------

doc/index.rst

@@ -24,8 +24,8 @@ Getting help
 * Report a bug in our `bug tracker`_.
 
 .. _wiki pages: https://nav.uninett.no/wiki/
-.. _nav-users mailing list archives: https://sympa.uninett.no/lists/uninett.no/arc/nav-users
-.. _the nav-users mailing list: https://sympa.uninett.no/lists/uninett.no/info/nav-users
+.. _nav-users mailing list archives: https://lister.sikt.no/hyperkitty/list/nav-users@lister.sikt.no/
+.. _the nav-users mailing list: https://lister.sikt.no/postorius/lists/nav-users.lister.sikt.no/
 .. _#nav IRC channel: irc://irc.libera.chat/nav
 .. _bug tracker: https://github.com/Uninett/nav
 

python/nav/Snmp/pynetsnmp.py

@@ -200,7 +200,10 @@ def walk(self, query="1.3.6.1.2.1.1.1.0"):
         current_oid = root_oid
 
         while 1:
-            response = self.handle.sgetnext(current_oid)
+            try:
+                response = self.handle.sgetnext(current_oid)
+            except (EndOfMibViewError, NoSuchObjectError):
+                break
             if response is None:
                 break
             response_oid, value = list(response.items())[0]
@@ -260,9 +263,12 @@ def bulkwalk(self, query="1.3.6.1.2.1.1.1.0", strip_prefix=False):
         current_oid = root_oid
 
         while 1:
-            response = self.handle.sgetbulk(
-                self.NON_REPEATERS, self.MAX_REPETITIONS, [current_oid]
-            )
+            try:
+                response = self.handle.sgetbulk(
+                    self.NON_REPEATERS, self.MAX_REPETITIONS, [current_oid]
+                )
+            except EndOfMibViewError:
+                break
             if response is None:
                 break
             for response_oid, value in response:

python/nav/activeipcollector/manager.py

@@ -42,13 +42,8 @@ def store(data):
 
     """
 
-    # Suspecting package drop - dividing into chunks and giving some
-    # breathing room for each batch of updates.
-    chunks = [data[x : x + 100] for x in range(0, len(data), 100)]
-    for chunk in chunks:
-        for db_tuple in chunk:
-            store_tuple(db_tuple)
-        time.sleep(2)
+    for db_tuple in data:
+        store_tuple(db_tuple)
 
     _logger.info('Sent %s updates', len(data))
 

python/nav/config.py

@@ -81,20 +81,21 @@ def read_flat_config(config_file, delimiter='='):
     if isinstance(config_file, str):
         config_file = open_configfile(config_file)
 
-    configuration = {}
-    for line in config_file.readlines():
-        line = line.strip()
-        # Unless the line is a comment, we parse it
-        if line and line[0] != '#':
-            # Split the key/value pair (max 1 split)
-            try:
-                (key, value) = line.split(delimiter, 1)
-                value = value.split('#', 1)[0]  # Remove end-of-line comments
-                configuration[key.strip()] = value.strip()
-            except ValueError:
-                sys.stderr.write("Config file %s has errors.\n" % config_file.name)
-
-    return configuration
+    with config_file:
+        configuration = {}
+        for line in config_file.readlines():
+            line = line.strip()
+            # Unless the line is a comment, we parse it
+            if line and line[0] != '#':
+                # Split the key/value pair (max 1 split)
+                try:
+                    (key, value) = line.split(delimiter, 1)
+                    value = value.split('#', 1)[0]  # Remove end-of-line comments
+                    configuration[key.strip()] = value.strip()
+                except ValueError:
+                    sys.stderr.write("Config file %s has errors.\n" % config_file.name)
+
+        return configuration
 
 
 def getconfig(configfile, defaults=None):
@@ -110,17 +111,18 @@ def getconfig(configfile, defaults=None):
     if isinstance(configfile, str):
         configfile = open_configfile(configfile)
 
-    config = configparser.RawConfigParser(defaults)
-    config.read_file(configfile)
+    with configfile:
+        config = configparser.RawConfigParser(defaults)
+        config.read_file(configfile)
 
-    sections = config.sections()
-    configdict = {}
+        sections = config.sections()
+        configdict = {}
 
-    for section in sections:
-        configsection = config.items(section)
-        configdict[section] = dict(configsection)
+        for section in sections:
+            configsection = config.items(section)
+            configdict[section] = dict(configsection)
 
-    return configdict
+        return configdict
 
 
 class NAVConfigParser(configparser.ConfigParser):

python/nav/etc/ipdevpoll.conf

@@ -198,6 +198,13 @@ plugins = statsystem statsensors statmulticast
 #
 ignored = <<=127.0.0.0/8, <<=fe80::/16, =128.0.0.0/2
 
+[modules]
+# A space separated list of serial numbers that will be ignored as
+# modules/devices. The default value is used to ignore soldered-on
+# linecards that Juniper erroneously reports as fieldreplaceable
+# units in ENTITY-MIB
+#ignored-serials = BUILTIN
+
 [linkstate]
 # Which ports to generate linkState events/alerts for.  Allowed values are
 # 'topology' for uplink/downlink ports, or 'any' for all ports.