diff --git a/ChangeLog.txt b/ChangeLog.txt index c1b88d4..545cd73 100644 --- a/ChangeLog.txt +++ b/ChangeLog.txt @@ -6,6 +6,26 @@ # file, you can obtain one at https://mozilla.org/MPL/2.0/. # +Version 1.2.1-rc2 - 2022-01-26 +-------------------------- + +* The behaviour of `compactor` is changed when an interrupt (SIGINT, SIGTERM) + is received e.g. when the service is restarted or a user enters ^C. + Previously all of the in progress C-DNS compression was aborted and all of + the in progress output files (compressed and uncompressed) were removed. Now + the `.raw` uncompressed output file is left on disk so it can be manually + recovered (i.e. renamed and compressed). Users will now need to choose how to + managed these uncompressed files. Behaviour on SIGHUP (re-reading of config + file without affecting any in progress compression) is unchanged. +* Additional default logging is added of the specific signal received by + `compactor`, when file rotation occurs and when compression is aborted. +* New `log-file-handing` option to log low level details of file + processing on file rotation and compression. This aids with debugging of file + processing problems and measurement of C-DNS file compression times. +* Text added in the documentation to clarify time patterns are needed in file + output patterns for filename based file rotation to occur (based on the + `rotation-period` option). +* Fix a bug relating to `compactor` start_time handling introduced in rc1. Version 1.2.1-rc1 - 2022-01-18 -------------------------- diff --git a/doc/compactor.adoc.in b/doc/compactor.adoc.in index d7449a0..81400de 100644 --- a/doc/compactor.adoc.in +++ b/doc/compactor.adoc.in @@ -84,7 +84,8 @@ character to the output path. Two categories of expansions are available, time expansions and configuration expansions. Time expansions are of the form `%` followed by a single letter, and expand to a time component determined by the letter. For a list of the available letters, -see *strftime*(3). +see *strftime*(3). Time expansions must be included in the output file pattern for +filename based rotation to occur using the *rotation-period* option. Configuration expansions are of the form `%{name}`, and substitute the value of the configuration item named. The configuration items that may be substituted are diff --git a/doc/user-guide/compactor-capture-from-network.adoc.in b/doc/user-guide/compactor-capture-from-network.adoc.in index 8b30a37..ae99658 100644 --- a/doc/user-guide/compactor-capture-from-network.adoc.in +++ b/doc/user-guide/compactor-capture-from-network.adoc.in @@ -38,6 +38,10 @@ Every _arg_ seconds, log basic statistics on packet collection to the system log. The default value of 0 disables this logging. +*-F, --log-file-handling*:: + Log detailed file processing when files are rotated and compressed. This facilitates + debugging of file processing issues and measurement of C-DNS file compression times. + *--sampling-threshold* _arg_:: A threshold for the percentage of traffic dropped on the internal channels above which sampling will be enabled (if *sampling-rate* is greater than 0). The diff --git a/doc/user-guide/compactor-outputs.adoc.in b/doc/user-guide/compactor-outputs.adoc.in index 5c11b63..f3f40c0 100644 --- a/doc/user-guide/compactor-outputs.adoc.in +++ b/doc/user-guide/compactor-outputs.adoc.in @@ -62,12 +62,17 @@ a single digit `0` to `9`. If not specified, the default level is `6`. *-t, --rotation-period* _SECONDS_:: - Specify the frequency with which all output file path patterns should be re-examined. - If the file path has changed, the existing output file is closed and a new one opened - using the new pattern expansion. If the file path has not changed, the pattern - is re-examined every second until it changes. The default period is 300 seconds. - This may be combined with maximum output file size rotation, in which case - rotation happens when either condition is met. + Specify the frequency with which all output file path patterns should be + re-examined. If the file path has changed after this period (e.g. because it + contains a date/time element which has changed), the existing output file is + closed and a new one opened using the new pattern expansion. If the file path + has not changed, the pattern is re-examined every second until it changes. + Note that file path patterns that do not contain date/time elements will + therefore not trigger file rotation via this mechanism, so using a default + file pattern similar that that in the default configuration file is + recommended. The default period is 300 seconds. This may be combined with + maximum output file size rotation, in which case rotation happens when either + condition is met. *-n, --include* _SECTIONS_:: Indicate which optional sections should be included in the main output. This argument diff --git a/doc/user-guide/configuring.adoc.in b/doc/user-guide/configuring.adoc.in index 137307a..e573064 100644 --- a/doc/user-guide/configuring.adoc.in +++ b/doc/user-guide/configuring.adoc.in @@ -160,7 +160,7 @@ interface, `interface3` the third and so on. | `eth0-eth1` | `%{rotate-period}` -| The file rotation period, in seconds. +| The file rotation period used when file names contain time/date elements, in seconds. | `300` | `%{snaplen}` diff --git a/doc/user-guide/installation.adoc.in b/doc/user-guide/installation.adoc.in index a8ca0a8..906463e 100644 --- a/doc/user-guide/installation.adoc.in +++ b/doc/user-guide/installation.adoc.in @@ -86,7 +86,7 @@ source, see the documentation for those items. {cpp}. Building them requires a {cpp} compiler and tool chain compatible with the 2011 ISO standard, otherwise {cpp}11. -| pkg-config | Used for dependency management. +| `pkg-config` | Used for dependency management. | `boost-log` .5+| Several libraries from http://www.boost.org[Boost @@ -103,7 +103,7 @@ source, see the documentation for those items. | `boost-iostreams` -| `boost-filesystem` +| `boost-filesystem` | | `liblzma`| The compression library from http://tukaani.org/xz/[XZ utils]. diff --git a/doc/user-guide/running.adoc.in b/doc/user-guide/running.adoc.in index 0a2943f..64151b1 100644 --- a/doc/user-guide/running.adoc.in +++ b/doc/user-guide/running.adoc.in @@ -57,8 +57,8 @@ Command options: -l [ --list-interfaces ] list all network interfaces. Configuration: - -t [ --rotation-period ] arg (=300) rotation period for all outputs, in - seconds. + -t [ --rotation-period ] arg (=300) rotation period for filename based rotation + for all outputs, in seconds. -q [ --query-timeout ] arg (=5) timeout period for unanswered queries, in seconds. ...(rest of output omitted for brevity)... @@ -93,12 +93,10 @@ summary information on exit (configuration and basic statistics). [WARNING] ==== -If C-DNS compression is enabled, interrupting _compactor_ causes the -current output C-DNS to be lost. For this reason you are recommended -to have C-DNS output file rotation enabled when using compression. For -more details, see <>. - -This behaviour is under review. +If C-DNS compression is enabled, interrupting _compactor_ causes the current +compression to be aborted (the '.raw' uncompressed file is retained on disk). +For this reason you are recommended to have C-DNS output file rotation enabled +when using compression. For more details, see <>. ==== ==== Capturing DNSTAP traffic @@ -121,20 +119,16 @@ summary information on exit (configuration and basic statistics). [WARNING] ==== -If C-DNS compression is enabled, interrupting _compactor_ causes the -current output C-DNS to be lost. For this reason you are recommended -to have C-DNS output file rotation enabled when using compression. For -more details, see <>. - -This behaviour is under review. +If C-DNS compression is enabled, interrupting _compactor_ causes the current +compression to be aborted (the '.raw' uncompressed file is retained on disk). +For this reason you are recommended to have C-DNS output file rotation enabled +when using compression. For more details, see <>. ==== [WARNING] ==== When capturing traffic over DNSTAP, output of raw or ignored packets to PCAP is not currently supported. Any related _compactor_ options specified will be ignored. - -This behaviour is under review. ==== ==== Capturing from PCAP files @@ -331,6 +325,11 @@ than specified by the *--sampling-threshold* option so sampling is enabled. | Drops are now below the sampling threshold and so sampling is disabled after the specified time limit. +| WARNING +| Aborting compression of +| On an interrupt of kill of the process, all in progress C-DNS file compression is + aborted. However the '.raw' (uncompressed) file is left on disk. + | INFO | Total packet count, etc. | Basic statistics on the ongoing network capture requested by the @@ -340,10 +339,18 @@ time limit. | Starting network capture | _compactor_ is starting a network capture. +| INFO +| Rotating file to +| _compactor_ is rotating the current collection file to the named file. + | INFO | Re-reading configuration | _compactor_ has received a `HUP` signal and is re-reading its configuration. +| INFO +| Signal handler: Received - +| _compactor_ has received a signal. + | INFO | Collection interrupted | _compactor_ has received a signal requesting termination and is @@ -376,6 +383,10 @@ line outputs data on sampling: Note that the LIBPCAP statistics provided here are information only and may not be reliable, particularly at high load. +Additional file handling messages can be output by enabling the *--log-file-handling* +option. This allows detailed debugging of file processing problems and also +measurements of file compression times. + === _compactor_ performance considerations ==== Threading @@ -408,8 +419,8 @@ configuration, increase the maximum throughput of _compactor_ . The processed used for C-DNS compression is designed for a scenario where _compactor_ is outputting data to a C-DNS file that is periodically rotated; that is, -the existing output file is close and a new output file with a distinct name is started. -C-DNS output is first written uncompressed to a temporary output file. When the file is +the existing output file is closed and a new output file with a distinct name is started. +C-DNS output is first written uncompressed to a '.raw' temporary output file. When the file is complete, either at the end of input from a PCAP file being converted, or after a file rotation, a new thread is spawned to compress the temporary output file to the final compressed C-DNS file. If strong compression is being used, the compression may @@ -419,15 +430,20 @@ maximum number of compression threads that may be active at any time is set in configuration; if the limit is reached, C-DNS output blocks until one of the compression threads finishes. +Detailed logs of file processing can be enabled with the *--log-file-handling* option. + [WARNING] ==== -If _compactor_ is interrupted, e.g. by the user typing Control-C, the -C-DNS output is stopped and all compressing threads are requested to -abort. In this case all temporary output files and incomplete -compressed output files are deleted and only completed compressed -files are retained. This means that all data in the current output -C-DNS file at the time of the interrupt is lost. This behaviour is -under review. +If _compactor_ is interrupted, e.g. by the user typing Control-C or the service +being restarted, the C-DNS output is stopped and all compressing threads are +requested to abort. In this case all incomplete compressed output files are +deleted, but the temporary uncompressed files are retained (alongside the +completed compressed files). The files will have a `.raw` extension and can be +renamed and compresses manually. Since these files are uncompressed and could +be large, users should choose how to manage such files. + +This behaviour was changed in 1.2.1. Prior to that all in progress output files +were removed on interrupt. ==== [WARNING] @@ -441,6 +457,10 @@ another DNS packet is seen or b) in the case of PCAP output any packet is seen. If _compactor_ is requested to reload its configuration via `HUP` signal, existing compression threads are not affected. +Note that if PCAP compression is specified, PCAP files are compressed as they +are written (not via a two-stage processes as above). On interrupt these files +are simply closed. + [[runninginspector]] == Running _inspector_ diff --git a/etc/compactor.conf.in b/etc/compactor.conf.in index dedfc3f..a58c3f9 100644 --- a/etc/compactor.conf.in +++ b/etc/compactor.conf.in @@ -56,6 +56,9 @@ # Log basic collection stats to syslog every n seconds. 0 (default) == never. # log-network-stats-period=0 +# Log detailed file processing for debugging. +# log-file-handling=false + # (Sampling is an experimental feature) # Sampling threshold is percentage of traffic dropped above which sampling will be enabled. Default is 10. # sampling-threshold=10 diff --git a/src/blockcbordata.cpp b/src/blockcbordata.cpp index 1038576..0b9e3fb 100644 --- a/src/blockcbordata.cpp +++ b/src/blockcbordata.cpp @@ -1726,15 +1726,23 @@ namespace block_cbor { Timestamp end_ts(*end_time, ticks_per_second); end_ts.writeCbor(enc); } - // There is a rare case where a 'live' capture is fed old data (e.g. - // via a DNSTAP socket) and in this case the start time can be later - // than the earliest data. Don't write the start time if this is the - // case - if ( start_time && start_time <= earliest_time ) + // There is a rare case where a 'live' capture is fed old data and in + // this case the start time can be a lot later than the earliest data, + // causing a negative file duration. Additionally, due to how block + // processing works, the start_time can be very slightly later than + // the earliest_time, so use earliest_time for both cases. + if ( start_time ) { enc.write(start_time_index); - Timestamp start_ts(*start_time, ticks_per_second); - start_ts.writeCbor(enc); + if ( *start_time <= earliest_time ) + { + Timestamp start_ts(*start_time, ticks_per_second); + start_ts.writeCbor(enc); + } else + { + Timestamp start_ts(earliest_time, ticks_per_second); + start_ts.writeCbor(enc); + } } if ( block_parameters_index > 0 ) diff --git a/src/blockcborwriter.cpp b/src/blockcborwriter.cpp index 2c09519..0f66592 100644 --- a/src/blockcborwriter.cpp +++ b/src/blockcborwriter.cpp @@ -1,5 +1,5 @@ /* - * Copyright 2016-2021 Internet Corporation for Assigned Names and Numbers. + * Copyright 2016-2022 Internet Corporation for Assigned Names and Numbers. * * This Source Code Form is subject to the terms of the Mozilla Public * License, v. 2.0. If a copy of the MPL was not distributed with this @@ -25,6 +25,7 @@ #include "blockcbor.hpp" #include "blockcbordata.hpp" #include "blockcborwriter.hpp" +#include "log.hpp" namespace { byte_string addr_to_string(const IPAddress& addr, const Configuration& config, bool is_client = true) @@ -104,7 +105,8 @@ void BlockCborWriter::checkForRotation(const std::chrono::system_clock::time_poi data_->start_time = timestamp; } filename_ = output_pattern_.filename(timestamp, config_); - enc_->open(filename_); + LOG_INFO << "Rotating C_DNS file to " << filename_; + enc_->open(filename_, config_.log_file_handling); writeFileHeader(); } } diff --git a/src/cbordecoder.cpp b/src/cbordecoder.cpp index 4aa39b7..7102a51 100644 --- a/src/cbordecoder.cpp +++ b/src/cbordecoder.cpp @@ -1,5 +1,5 @@ /* - * Copyright 2016-2019 Internet Corporation for Assigned Names and Numbers. + * Copyright 2016-2019, 2022 Internet Corporation for Assigned Names and Numbers. * * This Source Code Form is subject to the terms of the Mozilla Public * License, v. 2.0. If a copy of the MPL was not distributed with this diff --git a/src/cbordecoder.hpp b/src/cbordecoder.hpp index 3dce717..ebf4b87 100644 --- a/src/cbordecoder.hpp +++ b/src/cbordecoder.hpp @@ -1,5 +1,5 @@ /* - * Copyright 2016-2019 Internet Corporation for Assigned Names and Numbers. + * Copyright 2016-2019, 2022 Internet Corporation for Assigned Names and Numbers. * * This Source Code Form is subject to the terms of the Mozilla Public * License, v. 2.0. If a copy of the MPL was not distributed with this diff --git a/src/cborencoder.cpp b/src/cborencoder.cpp index 07939e8..4f70b11 100644 --- a/src/cborencoder.cpp +++ b/src/cborencoder.cpp @@ -15,6 +15,7 @@ #include #include "cborencoder.hpp" +#include "log.cpp" void CborBaseEncoder::writeTypeValue(unsigned cbor_type, unsigned long value) { @@ -196,6 +197,9 @@ void CborBaseEncoder::writeBreak() template<> void ParallelWriterPool::compressFile(const std::string& input, const std::string& output) { + + if (logging_) + LOG_INFO << "File handling: Renaming file: " << input.c_str() << " to " << output.c_str(); if ( std::rename(input.c_str(), output.c_str()) != 0 ) throw std::runtime_error("Can't rename " + input + " to " + output); } diff --git a/src/cborencoder.hpp b/src/cborencoder.hpp index 45c0f16..29d608b 100644 --- a/src/cborencoder.hpp +++ b/src/cborencoder.hpp @@ -308,7 +308,7 @@ class CborBaseStreamFileEncoder : public CborBaseEncoder * * \param name the filename. */ - virtual void open(const std::string& name) = 0; + virtual void open(const std::string& name, bool logging = false) = 0; /** * \brief Close the output file. @@ -374,12 +374,12 @@ class CborStreamFileEncoder : public CborBaseStreamFileEncoder * * \param name the base output filename. */ - virtual void open(const std::string& name) + virtual void open(const std::string& name, bool logging = false) { if ( writer_ ) throw std::runtime_error("Can't open file when one already open."); - writer_ = make_unique(name, level_); + writer_ = make_unique(name, level_, logging); bytes_written_ = 0; } @@ -525,11 +525,11 @@ class CborParallelStreamFileEncoder : public CborStreamFileEncoder * * \param name the base output filename. */ - virtual void open(const std::string& name) + virtual void open(const std::string& name, bool logging = false) { name_ = name; - CborStreamFileEncoder::open(name_ + ".raw"); + CborStreamFileEncoder::open(name_ + ".raw", logging); } /** @@ -585,8 +585,8 @@ class ParallelWriterPool : public BaseParallelWriterPool * compressing. * \param level the compression level to use. */ - ParallelWriterPool(unsigned max_threads, unsigned level) - : level_(level), max_threads_(max_threads), nthreads_(0), abort_(false) + ParallelWriterPool(unsigned max_threads, unsigned level, bool logging = false) + : level_(level), max_threads_(max_threads), nthreads_(0), abort_(false), logging_(logging) { } @@ -613,6 +613,12 @@ class ParallelWriterPool : public BaseParallelWriterPool */ virtual void compressFile(const std::string& input, const std::string& output) { + if (abort_) + { + LOG_WARN << "Aborting compression of " << input.c_str(); + // Leave the input file on disk so it can be recovered. + return; + } std::unique_lock lock(m_); if ( nthreads_ >= max_threads_ ) thread_finished_.wait(lock, [&](){ return nthreads_ < max_threads_; }); @@ -664,13 +670,15 @@ class ParallelWriterPool : public BaseParallelWriterPool try { + if (logging_) + LOG_INFO << "File handling: Starting compression of: " << input.c_str() << " to " << output.c_str(); std::ifstream ifs(input, std::ios::binary); if ( !ifs.is_open() ) throw std::runtime_error("Can't open file " + input); ifs.exceptions(std::ifstream::badbit); { - Writer writer(output, level_); + Writer writer(output, level_, logging_); uint8_t buf[OUTPUT_BUFFER_SIZE]; while ( !abort_ && !ifs.eof() ) @@ -681,10 +689,22 @@ class ParallelWriterPool : public BaseParallelWriterPool } ifs.close(); - if ( std::remove(input.c_str()) != 0 ) - throw std::runtime_error("Can't remove file " + input); - if ( abort_ && std::remove(output.c_str()) != 0 ) - throw std::runtime_error("Can't remove file " + output); + if ( !abort_) + { + if (logging_) + LOG_INFO << "File handling: Removing: " << input.c_str(); + if ( std::remove(input.c_str()) != 0 ) + throw std::runtime_error("Can't remove file " + input); + if (logging_) + LOG_INFO << "File handling: Finished compression of: " << input.c_str() << " to " << output.c_str(); + } else + { + // Leave the input file there so it can be recovered. + if (logging_) + LOG_INFO << "File handling: Removing file on abort: " << output.c_str(); + if (std::remove(output.c_str()) != 0 ) + throw std::runtime_error("Can't remove file " + output); + } } catch (const std::exception& err) { @@ -728,6 +748,12 @@ class ParallelWriterPool : public BaseParallelWriterPool * \brief condition variable to signal when a thread finishes. */ std::condition_variable thread_finished_; + + /** + * \brief logging + */ + bool logging_; + }; /** diff --git a/src/compactor.cpp b/src/compactor.cpp index 775b8a2..066d306 100644 --- a/src/compactor.cpp +++ b/src/compactor.cpp @@ -1,5 +1,5 @@ /* - * Copyright 2016-2021 Internet Corporation for Assigned Names and Numbers. + * Copyright 2016-2022 Internet Corporation for Assigned Names and Numbers. * * This Source Code Form is subject to the terms of the Mozilla Public * License, v. 2.0. If a copy of the MPL was not distributed with this @@ -620,17 +620,20 @@ static std::unique_ptr make_pcap_writer(const std::strin return make_unique>(pattern, std::chrono::seconds(config.rotation_period), config.xz_preset_pcap, - config.snaplen); + config.snaplen, + config.log_file_handling); else if ( config.gzip_pcap ) return make_unique>(pattern, std::chrono::seconds(config.rotation_period), config.gzip_level_pcap, - config.snaplen); + config.snaplen, + config.log_file_handling); else return make_unique>(pattern, std::chrono::seconds(config.rotation_period), 0, - config.snaplen); + config.snaplen, + config.log_file_handling); } /** @@ -879,6 +882,7 @@ static int run_configuration(const po::variables_map& vm, } signal_handler.wait_for_signals(); + LOG_INFO << "Signal handler: Received - " << strsignal(signal_received); switch(signal_received) { case 0: @@ -918,7 +922,6 @@ static int run_configuration(const po::variables_map& vm, stats.dump_stats(std::cout); } - LOG_INFO << "Compactor shutdown complete"; return res; } @@ -1033,15 +1036,15 @@ int main(int ac, char *av[]) { if ( configuration.xz_output ) { - writer_pool = std::make_shared>(configuration.max_compression_threads, configuration.xz_preset); + writer_pool = std::make_shared>(configuration.max_compression_threads, configuration.xz_preset, configuration.log_file_handling); } else if ( configuration.gzip_output ) { - writer_pool = std::make_shared>(configuration.max_compression_threads, configuration.gzip_level); + writer_pool = std::make_shared>(configuration.max_compression_threads, configuration.gzip_level, configuration.log_file_handling); } else { - writer_pool = std::make_shared>(configuration.max_compression_threads, 0); + writer_pool = std::make_shared>(configuration.max_compression_threads, 0, configuration.log_file_handling); } } @@ -1059,6 +1062,7 @@ int main(int ac, char *av[]) if ( thread.joinable() ) thread.join(); + LOG_INFO << "Compactor main thread shutdown complete. Other threads shutting down."; if ( res != 0 ) return 1; } diff --git a/src/configuration.cpp b/src/configuration.cpp index 4368ded..58e9f2f 100644 --- a/src/configuration.cpp +++ b/src/configuration.cpp @@ -304,6 +304,7 @@ Configuration::Configuration() max_block_items(5000), max_output_size(0), report_info(false), relaxed_mode(false), log_network_stats_period(0), + log_file_handling(false), sampling_threshold(10), sampling_rate(0), sampling_time(100), debug_dns(false), debug_qr(false), omit_hostid(false), omit_sysid(false), start_end_times_from_data(false), @@ -481,6 +482,9 @@ Configuration::Configuration() ("log-network-stats-period,L", po::value(&log_network_stats_period)->default_value(0), "log network collection stats period.") + ("log-file-handling,F", + po::value(&log_file_handling)->implicit_value(true), + "log details of file handling on rotation.") ("sampling-threshold", po::value(&sampling_threshold)->default_value(10), "sampling threshold - percentage of traffic dropped.") diff --git a/src/configuration.hpp b/src/configuration.hpp index a8d7ae7..06acbbc 100644 --- a/src/configuration.hpp +++ b/src/configuration.hpp @@ -857,6 +857,11 @@ class Configuration */ unsigned int log_network_stats_period; + /** + * \brief log detailed file handling + */ + bool log_file_handling; + /** * \brief sampling threshold above which to enable sampling */ diff --git a/src/pcapwriter.hpp b/src/pcapwriter.hpp index 4b18688..fdb7a39 100644 --- a/src/pcapwriter.hpp +++ b/src/pcapwriter.hpp @@ -1,5 +1,5 @@ /* - * Copyright 2016-2018 Internet Corporation for Assigned Names and Numbers. + * Copyright 2016-2018, 2022 Internet Corporation for Assigned Names and Numbers. * * This Source Code Form is subject to the terms of the Mozilla Public * License, v. 2.0. If a copy of the MPL was not distributed with this @@ -24,6 +24,7 @@ #include "makeunique.hpp" #include "nocopypacket.hpp" #include "rotatingfilename.hpp" +#include "log.hpp" /** * \class PcapBaseWriter @@ -112,9 +113,9 @@ class PcapWriter : public PcapBaseWriter * \param level compression level, if required. * \param snaplen snap length. */ - PcapWriter(const std::string& filename, unsigned level, unsigned snaplen) + PcapWriter(const std::string& filename, unsigned level, unsigned snaplen, bool logging = false) : filename_(filename), level_(level), - linktype_(NO_LINK_TYPE), snaplen_(snaplen) + linktype_(NO_LINK_TYPE), snaplen_(snaplen), logging_(logging) { } @@ -138,7 +139,7 @@ class PcapWriter : public PcapBaseWriter { if ( !writer_ ) { - writer_ = make_unique(filename_, level_); + writer_ = make_unique(filename_, level_, logging_); if ( linktype_ == NO_LINK_TYPE ) set_link_type(pdu); write_file_header(); @@ -185,6 +186,7 @@ class PcapWriter : public PcapBaseWriter { close(); filename_ = filename; + LOG_INFO << "Rotating PCAP file to " << filename_; } private: @@ -291,6 +293,12 @@ class PcapWriter : public PcapBaseWriter * \brief snap length. */ unsigned snaplen_; + + /** + * \brief logging + */ + unsigned logging_; + }; /** @@ -315,9 +323,9 @@ class PcapRotatingWriter : public PcapBaseRotatingWriter */ PcapRotatingWriter(const std::string& pattern, const std::chrono::seconds& period, - unsigned level, unsigned snaplen) + unsigned level, unsigned snaplen, bool logging = false) : fname_(make_unique(pattern + Writer::suggested_extension(), period)), - writer_("", level, snaplen) + writer_("", level, snaplen, logging) { } @@ -343,9 +351,8 @@ class PcapRotatingWriter : public PcapBaseRotatingWriter const std::chrono::system_clock::time_point& timestamp, const Configuration& config) { - if ( fname_->need_rotate(timestamp, config) ) + if ( fname_->need_rotate(timestamp, config) ) writer_.set_filename(fname_->filename(timestamp, config)); - writer_.write_packet(pdu, timestamp); } diff --git a/src/streamwriter.cpp b/src/streamwriter.cpp index ce817b7..10d84e3 100644 --- a/src/streamwriter.cpp +++ b/src/streamwriter.cpp @@ -1,5 +1,5 @@ /* - * Copyright 2016-2018 Internet Corporation for Assigned Names and Numbers. + * Copyright 2016-2018, 2022 Internet Corporation for Assigned Names and Numbers. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -27,12 +27,18 @@ const std::string& StreamWriter::STDOUT_FILE_NAME = "-"; -StreamWriter::StreamWriter(const std::string& name, unsigned) - : os_(&std::cout), name_(name), temp_name_(name + ".tmp") +StreamWriter::StreamWriter(const std::string& name, unsigned level, bool logging) + : os_(&std::cout), name_(name), temp_name_(name + ".tmp"), logging_(logging) { if ( name_ != STDOUT_FILE_NAME ) { ofs_.open(temp_name_, std::ofstream::binary); + if (logging_) { + if (level == 0 ) + LOG_INFO << "File handling: Opening tmp file (no compression): " << temp_name_ ; + else + LOG_INFO << "File handling: Opening tmp file (for compression): " << temp_name_ ; + } if ( ofs_.fail() ) throw std::runtime_error("Can't open file " + temp_name_); os_ = &ofs_; @@ -46,6 +52,8 @@ StreamWriter::~StreamWriter() if ( ofs_.is_open() ) { ofs_.close(); + if (logging_) + LOG_INFO << "File handling: Closing and renaming: " << temp_name_.c_str() << " to " << name_.c_str(); if ( std::rename(temp_name_.c_str(), name_.c_str()) != 0 ) LOG_ERROR << "file rename from " << temp_name_ << " to " << name_ << " failed"; } @@ -61,8 +69,8 @@ void StreamWriter::writeBytes(const uint8_t *p, std::ptrdiff_t n_bytes) os_->write(reinterpret_cast(p), n_bytes); } -GzipStreamWriter::GzipStreamWriter(const std::string& name, unsigned level) - : StreamWriter(name, level) +GzipStreamWriter::GzipStreamWriter(const std::string& name, unsigned level, bool logging) + : StreamWriter(name, level, logging) { boost::iostreams::gzip_params gzparams; @@ -115,8 +123,8 @@ const char* XzException::msg(lzma_ret err) } } -XzStreamWriter::XzStreamWriter(const std::string& name, unsigned level) - : StreamWriter(name, level), xz_stream_(LZMA_STREAM_INIT) +XzStreamWriter::XzStreamWriter(const std::string& name, unsigned level, bool logging) + : StreamWriter(name, level, logging), xz_stream_(LZMA_STREAM_INIT) { lzma_ret ret = lzma_easy_encoder(&xz_stream_, level, LZMA_CHECK_CRC64); if ( ret != LZMA_OK ) diff --git a/src/streamwriter.hpp b/src/streamwriter.hpp index 3bf2519..b4ea0ed 100644 --- a/src/streamwriter.hpp +++ b/src/streamwriter.hpp @@ -1,5 +1,5 @@ /* - * Copyright 2016-2018 Internet Corporation for Assigned Names and Numbers. + * Copyright 2016-2018, 2022 Internet Corporation for Assigned Names and Numbers. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -52,7 +52,7 @@ class StreamWriter * \param name filename. * \param level compression level */ - StreamWriter(const std::string& name, unsigned level); + StreamWriter(const std::string& name, unsigned level, bool logging = false); /** * \brief Destructor. @@ -106,6 +106,12 @@ class StreamWriter * \brief The temporary file to write to, renamed to final on close. */ std::string temp_name_; + + /** + * \brief The temporary file to write to, renamed to final on close. + */ + bool logging_; + }; /** @@ -123,7 +129,7 @@ class GzipStreamWriter : public StreamWriter * \param name filename. * \param level compression level */ - GzipStreamWriter(const std::string& name, unsigned level); + GzipStreamWriter(const std::string& name, unsigned level, bool logging = false); /** * \brief Destructor. @@ -199,7 +205,7 @@ class XzStreamWriter : public StreamWriter * \param name filename. * \param level compression level */ - XzStreamWriter(const std::string& name, unsigned level); + XzStreamWriter(const std::string& name, unsigned level, bool logging = false); /** * \brief Destructor. diff --git a/tests/blockcbordata_test.cpp b/tests/blockcbordata_test.cpp index 1cbba3c..49f43ba 100644 --- a/tests/blockcbordata_test.cpp +++ b/tests/blockcbordata_test.cpp @@ -1860,6 +1860,128 @@ SCENARIO("BlockData items can be written", "[block]") REQUIRE(tcbe.compareBytes(EXPECTED, sizeof(EXPECTED))); } } + + WHEN("start time is present") + { + BlockParameters bp; + std::vector bpv; + bpv.push_back(bp); + /* 2nd params have 10x ticks per second than default. */ + bp.storage_parameters.ticks_per_second = 10000000; + bpv.push_back(bp); + BlockData cd(bpv); + cd.earliest_time = std::chrono::system_clock::time_point(std::chrono::seconds(1) + std::chrono::microseconds(2)); + cd.end_time = std::chrono::system_clock::time_point(std::chrono::seconds(1) + std::chrono::microseconds(3)); + cd.start_time = std::chrono::system_clock::time_point(std::chrono::seconds(1) + std::chrono::microseconds(1)); + + + TestCborEncoder tcbe; + cd.writeCbor(tcbe); + tcbe.flush(); + + THEN("the encoding is as expected") + { + const uint8_t EXPECTED[] = + { + (5 << 5) | 31, + 0, (5 << 5) | 3, + 0, (4 << 5) | 2, 1, 2, + (1 << 5), (4 << 5) | 2, 1, 3, + (1 << 5) | 1, (4 << 5) | 2, 1, 1, + + 1, + (5 << 5) | 31, + 0, 0, + 1, 0, + 2, 0, + 3, 0, + 4, 0, + 5, 0, + (1 << 5) | 0, 0, + (1 << 5) | 1, 0, + (1 << 5) | 2, 0, + (1 << 5) | 3, 0, + (1 << 5) | 4, 0, + (1 << 5) | 5, 0, + (1 << 5) | 6, 0, + (1 << 5) | 7, 0, + (1 << 5) | 8, 0, + (1 << 5) | 9, 0, + (1 << 5) | 10, 0, + (1 << 5) | 11, 0, + 0xff, + + 2, + (5 << 5) | 31, + 0xff, + + 0xff + }; + + REQUIRE(tcbe.compareBytes(EXPECTED, sizeof(EXPECTED))); + } + } + + WHEN("start time is present but too late") + { + BlockParameters bp; + std::vector bpv; + bpv.push_back(bp); + /* 2nd params have 10x ticks per second than default. */ + bp.storage_parameters.ticks_per_second = 10000000; + bpv.push_back(bp); + BlockData cd(bpv); + cd.earliest_time = std::chrono::system_clock::time_point(std::chrono::seconds(1) + std::chrono::microseconds(2)); + cd.end_time = std::chrono::system_clock::time_point(std::chrono::seconds(1) + std::chrono::microseconds(3)); + cd.start_time = std::chrono::system_clock::time_point(std::chrono::seconds(1) + std::chrono::microseconds(5)); + + + TestCborEncoder tcbe; + cd.writeCbor(tcbe); + tcbe.flush(); + + THEN("the encoding is as expected") + { + const uint8_t EXPECTED[] = + { + (5 << 5) | 31, + 0, (5 << 5) | 3, + 0, (4 << 5) | 2, 1, 2, + (1 << 5), (4 << 5) | 2, 1, 3, + (1 << 5) | 1, (4 << 5) | 2, 1, 2, + + 1, + (5 << 5) | 31, + 0, 0, + 1, 0, + 2, 0, + 3, 0, + 4, 0, + 5, 0, + (1 << 5) | 0, 0, + (1 << 5) | 1, 0, + (1 << 5) | 2, 0, + (1 << 5) | 3, 0, + (1 << 5) | 4, 0, + (1 << 5) | 5, 0, + (1 << 5) | 6, 0, + (1 << 5) | 7, 0, + (1 << 5) | 8, 0, + (1 << 5) | 9, 0, + (1 << 5) | 10, 0, + (1 << 5) | 11, 0, + 0xff, + + 2, + (5 << 5) | 31, + 0xff, + + 0xff + }; + + REQUIRE(tcbe.compareBytes(EXPECTED, sizeof(EXPECTED))); + } + } } }