Handle low level errors properly in ipdevpoll

[lunkwill42]

As proven by #2494, ipdevpoll seems unable to handle some types of low level exception that arise during the course of a job. Example in point, an exception thrown by the pynetsnmp-2 library during the course of memory collection in the statsystem.py plugin.

Using NAV 5.5.0 as a starting point to debug WTH is going on here would be prudent.

Normally, JobHandler would catch unhandled exceptions raised by a plugin, log the details and then fail the job, so that it may be rescheduled by the rescheduler. In the case of #2494, it seems all further scheduling stops at the point of the raised exception.

Traceback

The original traceback that showed this problem went something like this (this looks like the Twisted reactor printing the details to stderr, rather than ipdevpoll catching it somewhere):

Unhandled error in Deferred:

Traceback (most recent call last):
  File "/opt/venvs/nav/lib/python3.9/site-packages/pynetsnmp/tableretriever.py", line 30, in v2v3how
    return proxy._getbulk(0, min(maxRepetitions, limit), [oids])
  File "/opt/venvs/nav/lib/python3.9/site-packages/nav/ipdevpoll/snmp/common.py", line 65, in _wrapper
    return func(*args, **kwargs)
  File "/opt/venvs/nav/lib/python3.9/site-packages/nav/ipdevpoll/snmp/common.py", line 134, in _getbulk
    return super(AgentProxyMixIn, self)._getbulk(*args, **kwargs)
  File "/opt/venvs/nav/lib/python3.9/site-packages/pynetsnmp/twistedsnmp.py", line 385, in _getbulk
    return defer.fail(ex)
--- <exception caught here> ---
  File "/opt/venvs/nav/lib/python3.9/site-packages/pynetsnmp/twistedsnmp.py", line 381, in _getbulk
    self.defers[self.session.getbulk(nonrepeaters,
  File "/opt/venvs/nav/lib/python3.9/site-packages/pynetsnmp/netsnmp.py", line 759, in getbulk
    self._handle_send_status(req, send_status, 'get')
  File "/opt/venvs/nav/lib/python3.9/site-packages/pynetsnmp/netsnmp.py", line 717, in _handle_send_status
    raise SnmpError(msg_fmt % msg_args)
pynetsnmp.netsnmp.SnmpError: get: snmp_send cliberr=0, snmperr=-11, errstring=b"Error building ASN.1 representation (Can't build OID for variable)"

Unhandled Error
Traceback (most recent call last):
  File "/opt/venvs/nav/lib/python3.9/site-packages/nav/ipdevpoll/daemon.py", line 458, in start_ipdevpoll
    process.run()
  File "/opt/venvs/nav/lib/python3.9/site-packages/nav/ipdevpoll/daemon.py", line 101, in run
    reactor.run()
  File "/opt/venvs/nav/lib/python3.9/site-packages/twisted/internet/base.py", line 1283, in run
    self.mainLoop()
  File "/opt/venvs/nav/lib/python3.9/site-packages/twisted/internet/base.py", line 1292, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/opt/venvs/nav/lib/python3.9/site-packages/twisted/internet/base.py", line 913, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/opt/venvs/nav/lib/python3.9/site-packages/nav/mibs/mibretriever.py", line 479, in _schedule_next
    deferred = self.retrieve_column(column)
  File "/opt/venvs/nav/lib/python3.9/site-packages/nav/mibs/mibretriever.py", line 442, in retrieve_column
    deferred.addCallbacks(_result_formatter, _valueerror_handler)
builtins.AttributeError: 'NoneType' object has no attribute 'addCallbacks'

Originally posted by @lunkwill42 in #2494 (comment)