Merge pull request #79 from dns-stats/develop

1.2.1-rc2 release

ChangeLog.txt

@@ -6,6 +6,26 @@
 # file, you can obtain one at https://mozilla.org/MPL/2.0/.
 #
 
+Version 1.2.1-rc2 - 2022-01-26
+--------------------------
+
+* The behaviour of `compactor` is changed when an interrupt (SIGINT, SIGTERM)
+  is received e.g. when the service is restarted or a user enters ^C.
+  Previously all of the in progress C-DNS compression was aborted and all of
+  the in progress output files (compressed and uncompressed) were removed. Now
+  the `.raw` uncompressed output file is left on disk so it can be manually
+  recovered (i.e. renamed and compressed). Users will now need to choose how to
+  managed these uncompressed files. Behaviour on SIGHUP (re-reading of config
+  file without affecting any in progress compression) is unchanged.
+* Additional default logging is added of the specific signal received by
+  `compactor`, when file rotation occurs and when compression is aborted.
+* New `log-file-handing` option to log low level details of file
+  processing on file rotation and compression. This aids with debugging of file
+  processing problems and measurement of C-DNS file compression times.
+* Text added in the documentation to clarify time patterns are needed in file
+  output patterns for filename based file rotation to occur (based on the
+  `rotation-period` option).
+* Fix a bug relating to `compactor` start_time handling introduced in rc1.
 
 Version 1.2.1-rc1 - 2022-01-18
 --------------------------

doc/compactor.adoc.in

@@ -84,7 +84,8 @@ character to the output path.
 Two categories of expansions are available, time expansions and configuration
 expansions. Time expansions are of the form `%` followed by a single letter, and
 expand to a time component determined by the letter. For a list of the available letters,
-see *strftime*(3).
+see *strftime*(3). Time expansions must be included in the output file pattern for
+filename based rotation to occur using the *rotation-period* option.
 
 Configuration expansions are of the form `%{name}`, and substitute the value of the
 configuration item named. The configuration items that may be substituted are

doc/user-guide/compactor-capture-from-network.adoc.in

@@ -38,6 +38,10 @@
   Every _arg_ seconds, log basic statistics on packet collection to the system log. The
   default value of 0 disables this logging.
 
+*-F, --log-file-handling*::
+  Log detailed file processing when files are rotated and compressed. This facilitates
+  debugging of file processing issues and measurement of C-DNS file compression times.
+
 *--sampling-threshold* _arg_::
   A threshold for the percentage of traffic dropped on the internal channels above which
   sampling will be enabled (if *sampling-rate* is greater than 0). The

doc/user-guide/compactor-outputs.adoc.in

@@ -62,12 +62,17 @@
   a single digit `0` to `9`.  If not specified, the default level is `6`.
 
 *-t, --rotation-period* _SECONDS_::
-  Specify the frequency with which all output file path patterns should be re-examined.
-  If the file path has changed, the existing output file is closed and a new one opened
-  using the new pattern expansion. If the file path has not changed, the pattern
-  is re-examined every second until it changes. The default period is 300 seconds.
-  This may be combined with maximum output file size rotation, in which case
-  rotation happens when either condition is met.
+  Specify the frequency with which all output file path patterns should be
+  re-examined. If the file path has changed after this period (e.g. because it
+  contains a date/time element which has changed), the existing output file is
+  closed and a new one opened using the new pattern expansion. If the file path
+  has not changed, the pattern is re-examined every second until it changes.
+  Note that file path patterns that do not contain date/time elements will
+  therefore not trigger file rotation via this mechanism, so using a default
+  file pattern similar that that in the default configuration file is
+  recommended. The default period is 300 seconds. This may be combined with
+  maximum output file size rotation, in which case rotation happens when either
+  condition is met.
 
 *-n, --include* _SECTIONS_::
   Indicate which optional sections should be included in the main output. This argument

doc/user-guide/configuring.adoc.in

@@ -160,7 +160,7 @@ interface, `interface3` the third and so on.
 | `eth0-eth1`
 
 | `%{rotate-period}`
-| The file rotation period, in seconds.
+| The file rotation period used when file names contain time/date elements, in seconds.
 | `300`
 
 | `%{snaplen}`

doc/user-guide/installation.adoc.in

@@ -86,7 +86,7 @@ source, see the documentation for those items.
   {cpp}. Building them requires a {cpp} compiler and tool chain compatible
   with the 2011 ISO standard, otherwise {cpp}11.
 
-| pkg-config | Used for dependency management.
+| `pkg-config` | Used for dependency management.
 
 | `boost-log`
 .5+| Several libraries from http://www.boost.org[Boost
@@ -103,7 +103,7 @@ source, see the documentation for those items.
 
 | `boost-iostreams`
 
-| `boost-filesystem`
+| `boost-filesystem` |
 
 | `liblzma`| The compression library from http://tukaani.org/xz/[XZ utils].
 

doc/user-guide/running.adoc.in

@@ -57,8 +57,8 @@ Command options:
   -l [ --list-interfaces ]         list all network interfaces.
 
 Configuration:
-  -t [ --rotation-period ] arg (=300)   rotation period for all outputs, in 
-                                        seconds.
+  -t [ --rotation-period ] arg (=300)   rotation period for filename based rotation
+                                        for all outputs, in seconds.
   -q [ --query-timeout ] arg (=5)       timeout period for unanswered queries, 
                                         in seconds.
 ...(rest of output omitted for brevity)...
@@ -93,12 +93,10 @@ summary information on exit (configuration and basic statistics).
 
 [WARNING]
 ====
-If C-DNS compression is enabled, interrupting _compactor_ causes the
-current output C-DNS to be lost. For this reason you are recommended
-to have C-DNS output file rotation enabled when using compression. For
-more details, see <<C-DNS output compression>>.
-
-This behaviour is under review.
+If C-DNS compression is enabled, interrupting _compactor_ causes the current
+compression to be aborted (the '.raw' uncompressed file is retained on disk).
+For this reason you are recommended to have C-DNS output file rotation enabled
+when using compression. For more details, see <<C-DNS output compression>>.
 ====
 
 ==== Capturing DNSTAP traffic
@@ -121,20 +119,16 @@ summary information on exit (configuration and basic statistics).
 
 [WARNING]
 ====
-If C-DNS compression is enabled, interrupting _compactor_ causes the
-current output C-DNS to be lost. For this reason you are recommended
-to have C-DNS output file rotation enabled when using compression. For
-more details, see <<C-DNS output compression>>.
-
-This behaviour is under review.
+If C-DNS compression is enabled, interrupting _compactor_ causes the current
+compression to be aborted (the '.raw' uncompressed file is retained on disk).
+For this reason you are recommended to have C-DNS output file rotation enabled
+when using compression. For more details, see <<C-DNS output compression>>.
 ====
 
 [WARNING]
 ====
 When capturing traffic over DNSTAP, output of raw or ignored packets to PCAP
 is not currently supported. Any related _compactor_ options specified will be ignored.
-
-This behaviour is under review.
 ====
 
 ==== Capturing from PCAP files
@@ -331,6 +325,11 @@ than specified by the *--sampling-threshold* option so sampling is enabled.
 | Drops are now below the sampling threshold and so sampling is disabled after the specified
 time limit. 
 
+| WARNING
+| Aborting compression of <file>
+| On an interrupt of kill of the process, all in progress C-DNS file compression is
+  aborted. However the '.raw' (uncompressed) file is left on disk. 
+
 | INFO
 | Total packet count, etc.
 | Basic statistics on the ongoing network capture requested by the
@@ -340,10 +339,18 @@ time limit.
 | Starting network capture
 | _compactor_  is starting a network capture.
 
+| INFO
+| Rotating file to <filename>
+| _compactor_  is rotating the current collection file to the named file.
+
 | INFO
 | Re-reading configuration
 | _compactor_  has received a `HUP` signal and is re-reading its configuration.
 
+| INFO
+| Signal handler: Received - <signal>
+| _compactor_  has received a signal.
+
 | INFO
 | Collection interrupted
 | _compactor_  has received a signal requesting termination and is
@@ -376,6 +383,10 @@ line outputs data on sampling:
 Note that the LIBPCAP statistics provided here are information only and may not be
 reliable, particularly at high load.
 
+Additional file handling messages can be output by enabling the *--log-file-handling*
+option. This allows detailed debugging of file processing problems and also 
+measurements of file compression times.
+
 === _compactor_  performance considerations
 
 ==== Threading
@@ -408,8 +419,8 @@ configuration, increase the maximum throughput of _compactor_ .
 
 The processed used for C-DNS compression is designed for a scenario where
 _compactor_ is outputting data to a C-DNS file that is periodically rotated; that is,
-the existing output file is close and a new output file with a distinct name is started.
-C-DNS output is first written uncompressed to a temporary output file. When the file is
+the existing output file is closed and a new output file with a distinct name is started.
+C-DNS output is first written uncompressed to a '.raw' temporary output file. When the file is
 complete, either at the end of input from a PCAP file being converted, or after a file
 rotation, a new thread is spawned to compress the temporary output file to the final
 compressed C-DNS file. If strong compression is being used, the compression may
@@ -419,15 +430,20 @@ maximum number of compression threads that may be active at any time is set in
 configuration; if the limit is reached, C-DNS output blocks until one of the compression
 threads finishes.
 
+Detailed logs of file processing can be enabled with the *--log-file-handling* option.
+
 [WARNING]
 ====
-If _compactor_ is interrupted, e.g. by the user typing Control-C, the
-C-DNS output is stopped and all compressing threads are requested to
-abort. In this case all temporary output files and incomplete
-compressed output files are deleted and only completed compressed
-files are retained. This means that all data in the current output
-C-DNS file at the time of the interrupt is lost. This behaviour is
-under review.
+If _compactor_ is interrupted, e.g. by the user typing Control-C or the service
+being restarted, the C-DNS output is stopped and all compressing threads are
+requested to abort. In this case all incomplete compressed output files are
+deleted, but the temporary uncompressed files are retained (alongside the
+completed compressed files). The files will have a `.raw` extension and can be
+renamed and compresses manually. Since these files are uncompressed and could
+be large, users should choose how to manage such files.
+
+This behaviour was changed in 1.2.1. Prior to that all in progress output files
+were removed on interrupt.
 ====
 
 [WARNING]
@@ -441,6 +457,10 @@ another DNS packet is seen or b) in the case of PCAP output any packet is seen.
 If _compactor_ is requested to reload its configuration via `HUP`
 signal, existing compression threads are not affected.
 
+Note that if PCAP compression is specified, PCAP files are compressed as they
+are written (not via a two-stage processes as above). On interrupt these files
+are simply closed.
+
 [[runninginspector]]
 == Running _inspector_
 

etc/compactor.conf.in

@@ -56,6 +56,9 @@
 # Log basic collection stats to syslog every n seconds. 0 (default) == never.
 # log-network-stats-period=0
 
+# Log detailed file processing for debugging.
+# log-file-handling=false
+
 # (Sampling is an experimental feature)
 # Sampling threshold is percentage of traffic dropped above which sampling will be enabled. Default is 10.
 # sampling-threshold=10

src/blockcbordata.cpp

@@ -1726,15 +1726,23 @@ namespace block_cbor {
             Timestamp end_ts(*end_time, ticks_per_second);
             end_ts.writeCbor(enc);
         }
-        // There is a rare case where a 'live' capture is fed old data (e.g.
-        // via a DNSTAP socket) and in this case the start time can be later
-        // than the earliest data. Don't write the start time if this is the
-        // case
-        if ( start_time  && start_time <= earliest_time )
+        // There is a rare case where a 'live' capture is fed old data and in
+        // this case the start time can be a lot later than the earliest data,
+        // causing a negative file duration. Additionally, due to how block
+        // processing works, the start_time can be very slightly later than
+        // the earliest_time, so use earliest_time for both cases.
+        if ( start_time )
         {
             enc.write(start_time_index);
-            Timestamp start_ts(*start_time, ticks_per_second);
-            start_ts.writeCbor(enc);
+            if ( *start_time <= earliest_time )
+            {
+                Timestamp start_ts(*start_time, ticks_per_second);
+                start_ts.writeCbor(enc);
+            } else
+            {
+                Timestamp start_ts(earliest_time, ticks_per_second);
+                start_ts.writeCbor(enc);
+            }
         }
 
         if ( block_parameters_index > 0 )

src/blockcborwriter.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright 2016-2021 Internet Corporation for Assigned Names and Numbers.
+ * Copyright 2016-2022 Internet Corporation for Assigned Names and Numbers.
  *
  * This Source Code Form is subject to the terms of the Mozilla Public
  * License, v. 2.0. If a copy of the MPL was not distributed with this
@@ -25,6 +25,7 @@
 #include "blockcbor.hpp"
 #include "blockcbordata.hpp"
 #include "blockcborwriter.hpp"
+#include "log.hpp"
 
 namespace {
     byte_string addr_to_string(const IPAddress& addr, const Configuration& config, bool is_client = true)
@@ -104,7 +105,8 @@ void BlockCborWriter::checkForRotation(const std::chrono::system_clock::time_poi
             data_->start_time = timestamp;
         }
         filename_ = output_pattern_.filename(timestamp, config_);
-        enc_->open(filename_);
+        LOG_INFO << "Rotating C_DNS file to " << filename_;
+        enc_->open(filename_, config_.log_file_handling);
         writeFileHeader();
     }
 }

src/cbordecoder.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright 2016-2019 Internet Corporation for Assigned Names and Numbers.
+ * Copyright 2016-2019, 2022 Internet Corporation for Assigned Names and Numbers.
  *
  * This Source Code Form is subject to the terms of the Mozilla Public
  * License, v. 2.0. If a copy of the MPL was not distributed with this

src/cbordecoder.hpp

@@ -1,5 +1,5 @@
 /*
- * Copyright 2016-2019 Internet Corporation for Assigned Names and Numbers.
+ * Copyright 2016-2019, 2022 Internet Corporation for Assigned Names and Numbers.
  *
  * This Source Code Form is subject to the terms of the Mozilla Public
  * License, v. 2.0. If a copy of the MPL was not distributed with this

src/cborencoder.cpp

@@ -15,6 +15,7 @@
 #include <limits>
 
 #include "cborencoder.hpp"
+#include "log.cpp"
 
 void CborBaseEncoder::writeTypeValue(unsigned cbor_type, unsigned long value)
 {
@@ -196,6 +197,9 @@ void CborBaseEncoder::writeBreak()
 template<>
 void ParallelWriterPool<StreamWriter>::compressFile(const std::string& input, const std::string& output)
 {
+
+    if (logging_)
+        LOG_INFO << "File handling: Renaming file:                 " << input.c_str() << " to " << output.c_str();
     if ( std::rename(input.c_str(), output.c_str()) != 0 )
         throw std::runtime_error("Can't rename " + input + " to " + output);
 }