Merge pull request #72 from dns-stats/develop

Merge develop into master for 1.2.1-beta1 release

ChangeLog.txt

@@ -6,6 +6,18 @@
 # file, you can obtain one at https://mozilla.org/MPL/2.0/.
 #
 
+Version 1.2.1-beta1 - 2021-08-23
+--------------------------
+
+* Change behaviour of internal sniffer channel when overloaded to better
+  report and manage dropping/sampling of packets
+  * This fixes a bug where the 'Total Packets processed' statistic did not
+    include any packets dropped during overload
+* Improve `log-interval` statistics and packet drop logging
+* EXPERIMENTAL: New sampling mode option (disabled by default)
+  * sampling can be enabled at a specified rate for a specified time
+    when packet drops exceed a specified threshold (see config options)
+
 Version 1.2.0 - 2021-05-07
 --------------------------
 

doc/user-guide/compactor-capture-from-network.adoc.in

@@ -37,3 +37,18 @@
 *-L, --log-network-stats-period* _arg_::
   Every _arg_ seconds, log basic statistics on packet collection to the system log. The
   default value of 0 disables this logging.
+
+*--sampling-threshold* _arg_::
+  A threshold for the percentage of traffic dropped on the internal channels above which
+  sampling will be enabled (if *sampling-rate* is greater than 0). The
+  default value is 10.
+
+*--sampling-rate* _arg_::
+  The rate (1 in _arg_ packets) to be applied when sampling mode is enabled (this is an 
+  experimental feature). The rate is applied for *sampling-time* seconds and then sampling
+  is disabled. After this, depending on the traffic rate, sampling may be enabled again if the drops rise
+  above the *sampling-threshold*. The default value of 0 disables this option.
+
+*--sampling-time* _arg_::
+  The period of time to apply sampling mode for. To avoid accidentally setting a low value
+  that could result in instability this must be at least 10s. The default value is 100.

doc/user-guide/installation.adoc.in

@@ -4,11 +4,14 @@
 
 Binary install packages are available from a
 https://launchpad.net[Launchpad] PPA (Personal Package Archive)
-`ppa:dns-stats/compactor` for
+`ppa:dns-stats/compactor-bionic` for
 
-* http://releases.ubuntu.com/16.04/[Ubuntu 16.04 LTS 'Xenial Xerus']
+* http://releases.ubuntu.com/18.04/[Ubuntu 18.04 LTS 'Bionic Beaver']
 
-==== Ubuntu 16.04 LTS 'Xenial Xerus'
+Experimental packages for Ubuntu 20.04 are also available for early testing
+from `ppa:dns-stats/compactor-focal`.
+
+==== Ubuntu packages
 
 _compactor_ and  _inspector_  are supplied in separate packages named
 `dns-stats-compactor` and `dns-stats-inspector`.

doc/user-guide/overview.adoc.in

@@ -114,14 +114,15 @@ with the following changes:
 ** _BlockStatistics_:
 *** _compactor-non-dns-packets_ (-1): count of the number of received packets that
     could not be interpreted as DNS packets.
-*** _compactor-out-of-order-packets (-2): count of the number of received packets that
+*** _compactor-out-of-order-packets_ (-2): count of the number of received packets that
     were not received by _compactor_ in strict chronological order.
 *** _compactor-missing-pairs_ (-3): count of output query/response pairs not written
     by _compactor_ because they could not be processed quickly enough.
 *** _compactor-missing-packets_ (-4): count of output raw PCAP packets not written
     by _compactor_ because they could not be processed quickly enough.
 *** _compactor-missing-non-dns_ (-5): count of output ignored PCAP packets not written
     by _compactor_ because they could not be processed quickly enough.
+*** _compactor-packets_ (-6): total packets received by compactor
 
 [IMPORTANT]
 ====

doc/user-guide/running.adoc.in

@@ -297,32 +297,42 @@ messages.
 | Comment
 
 | ERROR
-| C-DNS overflow. Dropping address event(s).
-| An address event is a countable event associated with a client address, such as
-a TCP reset or ICMP event. These are happening faster than _compactor_  can
-record them, so at least one has been dropped.
+| Dropping on these channels: Sniffer C-DNS
+| Packets are arriving faster than _compactor_  can
+process them, so packets are being dropped internally on the specified
+channels. More information is available via the *--log-network-stats-period* logging output.
 
 | ERROR
-| C-DNS overflow. Dropping query/response(s).
-| DNS query/response matches are being generated faster than they can be
-recorded to the C-DNS output. At least one of these matches has been dropped.
-
-| ERROR
-| Ignored PCAP overflow. Dropping packet(s).
+| Ignored PCAP channel overflow. Dropping packet(s).
 | More ignored packets - that is, packets that do not appear to be DNS
 related or which are malformed - are arriving than can be processed
 and recorded to the ignored PCAP output. At least one has been
 dropped.
 
 | ERROR
-| Raw PCAP overflow. Dropping packet(s).
+| Raw PCAP channel overflow. Dropping packet(s).
 | More input packets are arriving than can be processed and recorded
 to the raw PCAP output. At least one has been dropped.
 
+| WARNING
+| Sampling mode switched on for 100s with rate of 1 in 10
+| Drops on at least one internal channel are occurring at a rate higher
+than specified by the *--sampling-threshold* option so sampling is enabled.
+
+| WARNING
+| Sampling mode extended as drops still occurring
+| Drops above the threshold level are still occurring after the specified
+*--sampling-time* period.
+
+| WARNING
+| Sampling mode switched off because time limit expired and not dropping above threshold
+| Drops are now below the sampling threshold and so sampling is disabled after the specified
+time limit. 
+
 | INFO
 | Total packet count, etc.
 | Basic statistics on the ongoing network capture requested by the
-*--log-network-stats-period* option.
+*--log-network-stats-period* option (see below).
 
 | INFO
 | Starting network capture
@@ -338,6 +348,17 @@ to the raw PCAP output. At least one has been dropped.
   stopping collection.
 |===
 
+Example statistics produced by enabling the *log-network-stats-period* option:
+
+----
+*Stats interval: average rate         480027 pps  over  1s
+ LIBPCAP : recv/OS drop/IF drop       481002/         0/         0
+ Sniffer : recv/dropped/queue         480701/         0/       917
+ Sampling: recv/discard               480027/         0
+ C-DNS   : recv/dropped/queue         480027/         0/         0
+ PCAP out: raw drop/ignored drop           0/         0/
+----
+
 Other error messages are reporting an internal error.
 
 === _compactor_  performance considerations

etc/compactor.conf.in

@@ -56,6 +56,16 @@
 # Log basic collection stats to syslog every n seconds. 0 (default) == never.
 # log-network-stats-period=0
 
+# (Sampling is an experimental feature)
+# Sampling threshold is percentage of traffic dropped above which sampling will be enabled. Default is 10.
+# sampling-threshold=10
+
+# Sampling rate (1 in n) to be applied if packets dropped internally. 0 (default) == none.
+# sampling-rate=0
+
+# Apply sampling for n seconds for before re-checking for dropped packets. Default is 100.
+# sampling-time=100
+
 # Output options.
 
 # Output file rotation period, in seconds.

src/channel.hpp

@@ -1,5 +1,5 @@
 /*
- * Copyright 2016-2017 Internet Corporation for Assigned Names and Numbers.
+ * Copyright 2016-2021 Internet Corporation for Assigned Names and Numbers.
  *
  * This Source Code Form is subject to the terms of the Mozilla Public
  * License, v. 2.0. If a copy of the MPL was not distributed with this
@@ -71,6 +71,15 @@ class Channel
         return closed_;
     }
 
+   /**
+    * \brief Return current queue size
+    */
+   unsigned get_length()
+   {
+       std::lock_guard<std::mutex> lock(m_);
+       return queue_.size();
+   }
+
     /**
      * \brief Add a new item to the channel.
      *