v4.0.3

• The input framework's handling of unset fields (commonly expressed as "-")
  in ingested data is now safer and more consistent. When reading data into
  records, Zeek now accepts unset fields in the input data only when the
  corresponding record field is &optional. Unset fields for non-optional
  fields cause the input line to be skipped. Reading data into tables with complex
  index types (such as table[count, count] of string) now also skips lines
  with unset fields, since such indexes require fields to be present.

  Note that this may change the behavior of existing scripts if you have unset
  fields in your input data.

• The version field in ssh.log is now optional and will not be set if we cannot
  determine the version that was negotiated by the client and server.

  #1605

• Zeekctl could crash at startup on certain compilers and platforms due to a
  memory corruption issue in the Broker python bindings.

  zeek/broker#187

• The highwayhash submodule was updated to fix a build failure on FreeBSD for
  PowerPC.

  #1591

This release deprecates the following functionality:

• The stepping-stone analyzer is marked as deprecated. It was partially marked
  as deprecated in 2.0, and will be fully removed in v4.1.

  #1573
  #1604

v4.0.2

This release fixes the following security issues:

• Fix potential Undefined Behavior in decode_netbios_name() and
  decode_netbios_name_type() BIFs. The latter has a possibility of a remote
  heap-buffer-overread, making this a potential DoS vulnerability.

  #1533
  #1563

• Add some extra length checking when parsing mobile ipv6 packets. Due to the
  possibility of reading invalid headers from remote sources, this is a
  potential DoS vulnerability.

  c5533f3

This release fixes the following bugs:

• Fix heap-use-after-free after clear_table() on a table that uses expiration
  attributes.

  d51bd4b

• Add fatal error for if table/Dictionary state ever becomes invalid since
  the behavior becomes unexpected/unclear at that point (e.g. when table
  bucket positions become large enough to overflow their 16-bit storage due
  to aggressive expiration-check settings preventing the re-positioning items)

  292e3e1

• Add missing "zeek/" to header includes, which can prevent external plugins
  from compiling against Zeek source-tree (e.g. via ./configure --zeek-dist=)

  #1547
  #1549

• Fix reading empty set[enum] values and any vector of enum values from config
  files

  #1555
  #1558
  #1559

• Fix type-checks related to list-type equality

  #1296
  #1358

v4.0.1

This release fixes the following security issue:

• Fix null-pointer dereference when encountering an invalid enum name in a
  config/input file that tries to read it into a set[enum]. For those
  that have such an input feed whose contents may come from external/remote
  sources, this is a potential DoS vulnerability.

  #1487
  #1488

This release fixes the following bugs:

• Fix mime type detection bug in IRC/FTP file_transferred event for file
  data containing null-bytes

  #1430

• Fix potential for missing timestamps in SMB logs

  #1436

• Remove use of LeakSanitizer API on FreeBSD where it's unsupported

  #1440

• Fix incorrect parsing of ERSPAN Type I

  #1445
  f53fb9a

• Fix incorrect/overflowed n value for SSL_Heartbeat_Many_Requests notices
  where number of server heartbeats is greater than number of client heartbeats.

  #1454
  #1459
  c23e3ca

• Fix missing user_agent existence check in smtp/software.zeek
  (causes reporter.log error noise, but no functional difference)

  #1455
  83d5b44

• Fix include order of bundled headers to avoid conflicts with
  pre-existing/system-wide installs

  #1465

• Fix musl build (e.g. Void, Alpine, etc.)

  #1469
  2ad4825

• Fix build with -DENABLE_MOBILE_IPV6 / ./configure --enable-mobile-ipv6

  #1493
  #1495

• Add check for null packet data in pcap IOSource, which is an observed state
  in Myricom libpcap that crashes Zeek via null-pointer dereference

  #1498

• Allow CRLF line-endings in Zeek scripts and signature files

  #1497
  #1499

• Fix armv7 build

  #1496
  #1502

• Fix unserialization of set[function], generally now used by connection
  record removal hooks, and specifically breaking intel.log of Zeek clusters

  #1506
  #1513

• Fix indexing of set/table types with a vector

  #1514

• Fix precision loss in ASCII logging/printing of large double, time, or
  interval values

  #1450
  #1494

• Improve handling of invalid SIP data before requests

  #1507
  #1511

• Fix copy()/cloning vectors that have holes (indices w/ null values)

  180ab31

Reminder: Zeek 4.0.x is a Long-Term Support (LTS) release, receiving bug fixes until at least May 2022 (estimate of 2 months after 5.0.0 release).

v3.0.14

This release fixes the following security issue:

• Fix null-pointer dereference when encountering an invalid enum name in a
  config/input file that tries to read it into a set[enum]. For those
  that have such an input feed whose contents may come from external/remote
  sources, this is a potential DoS vulnerability.

  #1487
  #1488

Reminder: Zeek 3.0.x is a Long-Term Support (LTS) release, receiving bug fixes until at least May 2021 (estimate of 2 months after 4.0.0 release).

v4.0.0

New Functionality

• Added support for EDNS0 Cookie and Keep-Alive options.

• Added new Packet Analysis plugin architecture for parsing packet headers
  at layers below the existing Session analysis plugins. This allows
  writing plugins to parse the various parts of a packet header separately,
  chaining down into other plugins as needed.

• Add dce_rpc_request_stub and dce_rpc_response_stub events for
  accessing the contents of DCE-RPC request/response stub data.

• Add support for log filter policy hooks, which supersede the current
  log predicates. The hook signature is as follows:

    hook(rec: any, id: Log::ID, filter: Log::Filter);
  

  The logging manager invokes hooks on each log record. Hooks can veto
  log records via a break, and modify them if necessary. Log filters
  inherit the stream-level hook, but can override or remove the hook
  as needed. The distribution's existing log streams now come with
  pre-defined hooks that users can add handlers to. The existing
  predicates are deprecated for removal in 4.1 but continue to work.

  See https://docs.zeek.org/en/master/frameworks/logging.html#filter-log-records
  for more details.

• Added a udp-state signature condition to enforce matching against
  either "originator" or "responder" flow direction of UDP packets.

• Improvements to capture-loss.zeek:

  • A new option, CaptureLoss::initial_watch_interval. When restarting a
    Zeek cluster, one usually wants some immediate feedback as to the health of
    the monitoring via capture loss. However, you previously needed to wait a
    full CaptureLoss::watch_interval, which defaults to 15 minutes. The
    new option specifies the interval for the first-time report. So the new
    default behavior provides stats after 1 minute and then after
    15 minutes afterward.

  • A new notice type, CaptureLoss::Too_Little_Traffic.
    If a Zeek process sees less than CaptureLoss::minimum_acks ACKs in a
    given interval, this notice gets raised. This can be a useful diagnostic
    if, for whatever reason, a Zeek process stops seeing traffic, but
    capture-loss.zeek would have previously only reported that "0 gaps and 0
    ACKs is 0% loss".

• A new zeek_script_args variable contains a list of arguments passed
  to a script. E.g. either when explicitly executing Zeek like
  zeek -- myscript.zeek -arg1 -arg2, or when using Zeek to interpret
  executable scripts that contain a hashbang line at the top like::

  #!/usr/local/zeek/bin/zeek --

• Added a new generate_all_events bif, which can be used to always raise
  events, even when they are not used by scripts. This can be used by the
  dump-events.zeek script to log all events that happen; the script
  got a new option to enable this behavior.

• Added new unknown_protocols.log that will log analyzer and protocol pairs
  via the packet analysis framework for packet protocols that aren't
  supported by Zeek. It can be enabled by loading the
  policy/misc/unknown-protocols script. The script adds a new
  unknown_protocol event.

• Added support for DNS resource records LOC, SSHFP, NSEC3PARAM, and custom
  BIND9 signaling. The associated events are:

  • dns_LOC
  • dns_SSHFP
  • dns_NSEC3PARAM
  • dns_BINDS

• Zeek now supports SSH clients/servers that advertise SSH version 1.99, which
  is a special version indicating that the server/client supports both SSH2 and
  SSH1.

• Added count_to_double() and int_to_double() type-conversion BIFs.

• Added these string-processing BIFs:

  • count_substr
  • find_str
  • rfind_str
  • starts_with
  • ends_with
  • is_num
  • is_alpha
  • is_alnum
  • ljust
  • rjust
  • swap_case
  • to_title
  • zfill
  • remove_prefix
  • remove_suffix

• Added a new Weird::sampling_global_list option to configure global
  rate-limiting of certain weirds instead of per connection/flow.

• Added a Pcap::findalldevs() for obtaining available network devices.

• Added enum_names() BIF to return names of an enum type's values

• Added type_aliases BIF for introspecting type-names of types/values

• Added composite-index support for &backend (Broker-backed tables).
  An example of a set with composite index is set[string, count, count].

• Sumstats now allows manual epochs. If an epoch interval of 0 is specified,
  epochs will have to be manually ended by callis SumStats::next_epoch. This
  can be convenient because epochs can be synced to other events.

• The Zeek distribution now includes Zeek's package manager, zkg. Its
  code, configuration, and state reside in Zeek's installation tree,
  as follows:

  • The toplevel script, zkg, installs alongside zeek in the
    distribution's $prefix/bin folder.

  • The config file installs into $prefix/etc/zkg/config. The
    distribution's zkg command uses it by default, but you can switch
    to a different one via the ZKG_CONFIG_FILE environment
    variable or the --configfile command-line flag.

  • zkg's package state resides in $prefix/var/lib/zkg. This
    implies that parallel Zeek installations now automatically
    separate their package installations.

  These folders have the same ownership and access permissions as the
  rest of the installation, , meaning that in order to manage zkg
  packages you need to run zkg as a user with corresponding access.
  Apart from these location overrides, the bundled zkg installation
  behaves as usual.

  local.zeek now contains a (commented out) @load statement you
  can use to source zkg's package state automatically.

  zkg's own Python module resides in ``zeek/python/zeekpkg`, in the
  installation tree's library folder. See below for additional changes
  around the library folder.

  zkg has external Python module dependencies. The Zeek configuration
  does not verify whether these dependencies are met. A new warning
  message at zkg launch flags missing packages and how to install them
  (e.g. via pip).

  Configuring with --disable-zkg disables the zkg inclusion. You
  can continue to install and use zkg independently. You're also free
  to use the config file in $prefix/etc/zkg/config with other zkg
  installations.

  The zkg source tree resides in auxil/package-manager as an
  additional Git submodule.

Changed Functionality

• NetControl::DROP had 3 conflicting definitions that could potentially
  be used incorrectly without any warnings or type-checking errors.
  Such enum redefinition conflicts are now caught and treated as errors,
  so the NetControl::DROP enums had to be renamed:

  • The use as enum of type Log::ID is renamed to NetControl::DROP_LOG

  • The use as enum of type NetControl::CatchReleaseInfo is renamed to
    NetControl::DROP_REQUESTED

  • The use as enum of type NetControl::RuleType is unchanged and still
    named NetControl::DROP

• The extract_email_addrs_vec() BIF now returns all occurrences of emails,
  including duplicates, with preserved order of occurrence. This seems like
  the original/documented intent of the function, but the previous
  implementation did not preserve ordering or duplicates.

• The Dictionary implementation is replaced (no API changes). The new version
  uses clustered hashing, a variation of Robinhood / Open Addressing hashing.
  This implementation generally performs better and utilizes less memory
  than the previous one. A detailed explanation of the implementation is here:
  https://jasonlue.github.io/algo/2019/08/20/clustered-hashing.html

• The p fields of Cluster::Node records now use a
  &default=0/unknown attribute with 0/unknown meaning that the node is
  not pre-configured to listen for incoming connections from other cluster
  nodes.

• The |x| operator, where x is an expression with an integral result,
  no longer performs an implicit coercion of that result into a signed
  int type. This was actually the behavior before Zeek 3.0 as well, but
  the attempt to prevent mistakes that easily result from integer literals in
  Zeek being unsigned like |5 - 9| causing an overflow/wraparound and
  yielding a very large number is not generally consistent since overflows
  are still generally able to happen in other ways and also in other contexts
  besides just the absolute-value operator. So the preference was to revert
  to a behavior that favors consistency. For reference, see
  #251 (comment)

• The Zeek installation tree is now more consistent in using a lib64/
  (rather than lib/) subdirectory for platforms where that's the common
  convention. If the old hardcoded lib/ path exists while installing Zeek
  4.0 and the new subdirectory differs, the old lib/ remains untouched.
  This clutters the installation but is safe: the new installation does not
  require the old location, and any files you might require still in the old
  tree (e.g. ZeekControl plugins) remain available.

  Due to Zeek 4's reorganization of the installation tree we recommend
  a clean-slate install when possible.

• Python modules installed with the Zeek distribution now reside in a
  common zeek/python directory below the library path (such as
  lib64/zeek/python) and no longer assume ZeekControl. The
  zeek/python/zeekctl folder now contains only ZeekControl's own
  functionality, zeek/python/zeekpkg contains zkg's Python module, and
  Broker's Python bindings live in zeek/python/broker. zeek-config --python_dir now reports this new zeek/python folder. Several
  new configure options allow you to customize the Python folder location,
  ...

v3.2.4

This release fixes the following security issues:

• Fix ASCII Input reader's treatment of input files containing null-bytes
  #1398
  a636f8e

  This is mostly only significant for deployments that utilize input data feeds
  whose content is controlled by external sources: an input file containing
  null-bytes could lead to a buffer-over-read, crash Zeek, and be exploited
  to cause Denial of Service.

This release fixes the following bugs:

• MIME sub-entities overwrote top-level header values cause misleading SMTP log
  #1352
  #1365

• Fix incorrect major_subsys_version field in pe_optional_header event
  #1401

Reminder: Zeek 3.0.x is the Long-Term Support (LTS) release, receiving bug fixes until at least May 2021 (estimate of 2 months after 4.0.0 release) while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately March 2021 when the next 4.0.x LTS release series is expected to begin.

v3.0.13

This release fixes the following security issues:

• Fix ASCII Input reader's treatment of input files containing null-bytes
  #1398
  a636f8e

  This is mostly only significant for deployments that utilize input data feeds
  whose content is controlled by external sources: an input file containing
  null-bytes could lead to a buffer-over-read, crash Zeek, and be exploited
  to cause Denial of Service.

This release fixes the following bugs:

• MIME sub-entities overwrote top-level header values cause misleading SMTP log
  #1352
  #1365

• Fix incorrect major_subsys_version field in pe_optional_header event
  #1401

Reminder: Zeek 3.0.x is a Long-Term Support (LTS) release, receiving bug fixes until at least May 2021 (estimate of 2 months after 4.0.0 release).

v3.2.3

This release fixes the following security issues:

• In the parsing of IPv6 addresses within EDNS ECS options, data was written
  to a stack-buffer using as many bytes as supplied in the option even if it
  was in excess of the desired address prefix or maximum IPv6 address size.
  This could result in an overflow of that stack-buffer. This may be
  remotely exploitable by anyone creating such a DNS message.

  Credit to OSS-Fuzz for discovery
  https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28336
  (Link to details becomes public 30 days after patch release)

This release fixes the following bugs:

• The parsing of IPv4 addresses within EDNS ECS options would overwrite the
  storage used for that address as many times as there were bytes in the option
  in excess of the desired address prefix length or maximum IPv4 address size.
  This could cause the resulting IPv4 address to be derived from the incorrect
  data.

• In parsing EDNS ECS options, upon encountering unexpected/excessive
  option-length or source-prefix parameters, the data pointer used for parsing
  was also not always advanced to the start of the next alleged option's data.
  Assuming all other parsing code correctly guards against invalid input,
  there's no further harm from that other than the subsequent parsing being
  more likely to encounter unexpected values and emitting more Weirds.

• Incorrect ICMP Neighbor Discovery Option length calculation
  #1225
  #1228

• Fix memory leak in deprecated Analyzer::ConnectionEvent()
  #1294

• Fix SMB2 response status parsing
  #1311
  0b8535b
  07c4662

• Fix excessive connection_status_update events for ICMP connections
  #1322

Reminder: Zeek 3.0.x is the Long-Term Support (LTS) release, receiving bug fixes until at least February 2021 while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately February 2021 when the next 4.0.x LTS release series is expected to begin.

v3.0.12

This release fixes the following bugs:

• Incorrect ICMP Neighbor Discovery Option length calculation
  #1225
  #1228

• Fix SMB2 response status parsing
  #1311
  0b8535b
  07c4662

• Fix excessive connection_status_update events for ICMP connections
  #1322

Reminder: Zeek 3.0.x is a Long-Term Support (LTS) release, receiving bug fixes until at least February 2021 (estimate of 2 months after 4.0.0 release).

v3.2.2

This release fixes the following security issues:

• Fix multipart MIME leak of sub-part found after closing-boundary

  After detecting a closing-boundary for a given multipart MIME entity, it
  enters into an "end of data" state, however any subsequent boundary
  delimiter could still cause the allocation of a sub-entity object that
  is never released due to cleanup logic being bypassed upon finding the
  "end of data" state already reached.

  This change prevents allocation/processing of sub-entities after the
  "end of data" state is reached (e.g. from detecting a multipart
  closing-boundary). This new behavior still aligns with RFC 2046
  expectations:

  "There appears to be room for additional information prior to the first
  boundary delimiter line and following the final boundary delimiter line.
  These areas should generally be left blank, and implementations must
  ignore anything that appears before the first boundary delimiter line or
  after the last one."

  This leak has potential for remote exploitation and cause for Denial of
  Service via resource exhaustion.

  Credit to OSS-Fuzz for discovery
  https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26027
  (Link to details becomes public 30 days after patch release)

This release fixes the following bugs:

• Fix incorrect RSTOS0 conn_state determinations
  #1164
  #1166

Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least December 2020 while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately December 2020 when the next 4.0.x LTS release series is expected to begin.