forked from zeek/zeek
-
Notifications
You must be signed in to change notification settings - Fork 0
/
NEWS
5358 lines (4064 loc) · 229 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This document summarizes the most important changes in the current Zeek
release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as Broker, come with their own ``CHANGES``.)
Zeek 6.0.0
==========
Breaking Changes
----------------
- Zeek now treats private address space (i.e., non-routable IP address ranges)
as local by default, matching the intuition of many users that e.g. a
192.168/16 IP address should show up as local in the logs. To do this, Zeek
automatically adds ``Site::private_address_space`` to ``Site::local_nets`` at
startup. Subsequent runtime updates to ``Site::private_address_space``
propagate to ``Site::local_nets``, while updates to the latter don't affect
the former.
You're free to define ``Site::local_nets`` as before and do not need to update
your configurations. If you added standard private address space to
``Site::local_nets`` in the past, you no longer need to do so. This also
applies to zeekctl's ``networks.cfg`` file.
The new global Boolean ``Site::private_address_space_is_local``, true by
default, controls the behavior. A redef to false brings back Zeek's prior
behavior of considering private address space an unrelated concept, which will
come in handy for example when working with tests that compare results against
log baselines that have not yet been updated.
- Custom source tarballs require a ``repo-info.json`` file.
Note, should you be using official Zeek release tarballs only, or build
Zeek solely from git checkouts, this does not affect you.
However, if you're building your own Zeek source tarballs, it is now required
that a ``repo-info.json`` file exists at the top-level. The ``dist`` target was
extended to add this file and official Zeek release source tarballs will
contain it going forward.
The following command can be used to produce ``repo-info.json``:
python3 ./ci/collect-repo-info.py --only-git > ../path/to/tarballdir/repo-info.json
This is required to support the new ``-V`` / ``--build-info`` option that
provides information about git submodules and included plugins used during
the build. The ``ci/collect-repo-info.py`` tool runs at ``./configure`` time
and either collects the required information from a git clone (when git is
installed), or otherwise uses the content of a file named ``repo-info.json``.
If you see opportunities to extend ``repo-info.json`` with further information,
please get in touch.
- Plugin authors should raise the minimum required CMake version to 3.15 to
ensure compatibility with new CMake scaffolding included in this
release. Older versions will trigger a warning at configuration time and,
depending on the functionality included in the plugin, may trigger subsequent
errors during configuration or build.
- The IRC_Data analyzer declaration has been moved to protocols/irc/IRC.h.
- The error message returned when using ``bro_init``, ``bro_done``, and
``bro_script_loaded`` events is now removed. removed. Usage of these events
has returned that error during script parsing for a few years, and time has
come to finally remove it.
New Functionality
-----------------
- Zeek now features experimental JavaScript support:
/* hello.js */
zeek.on('zeek_init', () => {
console.log('Hello, Zeek!');
});
$ zeek ./hello.js
Hello, Zeek!
When building Zeek on a system that features a recent (16.13+) version of the
libnode package with development headers, Zeek automatically includes the
externally-maintained ZeekJS plugin (https://github.com/corelight/zeekjs) as a
builtin plugin. This allows Zeek to load and execute JavaScript code located
in ``.js`` or ``.cjs`` files. When no such files are passed to Zeek, the
JavaScript engine and Node.js environment aren't initialized and there is no
runtime impact.
The Linux distributions Fedora 37 & 38, Ubuntu 22.10, and the upcoming Debian
12 release provide suitable packages. On other platforms, Node.js can be built
from source with the ``--shared`` option.
To disable this functionality, pass ``--disable-javascript`` to configure.
- Zeek events now hold network timestamps. For scheduled events, the timestamp
represents the network time for which the event was scheduled for, otherwise
it is the network time at event creation. A new bif ``current_event_time()``
allows to retrieve the current event's network timestamp within the script-layer.
When Zeek sends events via Broker to other nodes in a cluster, an event's network
timestamp is attached to the Broker messages. On a receiving Zeek node executing a
handler for a remote event, ``current_event_time()`` returns the network time of
the sending node at the time the event was created.
The Broker level implementation allows to exchange arbitrary event metadata, but
Zeek's script and C++ APIs currently only expose network timestamp functionality.
- A new bif ``from_json()`` can be used to parse JSON strings into records.
type A: record { a: addr; };
local p = from_json({\"a\": \"192.168.0.1\"}", A);
if ( p$valid )
print (p$v as A)
Implicit conversion from JSON to Zeek types is implemented for bool, int, count,
real, interval (number as seconds) and time (number as unix timestamp), port
(strings in "80/tcp" notation), patterns, addr, subnet, enum, sets, vectors
and records similar to the rules of the input framework. Optional or default
record fields are allowed to be missing or null in the input.
- Zeek now provides native "Community ID" support with a new bif called
``community_id_v1()``. Two policy scripts ``protocols/conn/community-id-logging``
and ``frameworks/notice/community-id`` extend the respective logs with a
``community_id`` field the same way as the external zeek-community-id plugin
provides. A main difference to the external ``hash_conn()`` bif is that the
``community_id_v1()`` takes a ``conn_id`` record instead of a ``connection``.
Loading the new policy scripts and using the external zeek-community-id
plugin at the same time is unsupported.
- ZeekControl is now multi-logger aware. When multiple logger nodes are configured
in ZeekControl's node.cfg, by default the log archival logic adds a logger's name
as suffix to the rotated file name:
stats.11:18:57-11:19:00-logger-1.log.gz
stats.11:18:57-11:19:00-logger-2.log.gz
Previously, in a multi-logger setup, individual logger processes would overwrite
each other's log files during rotation, causing data loss.
For setups with a single logger, there's no change in behavior. The naming
of the final logs can be customized by providing an alternative
``make-archive-name`` script and using the new ``ZEEK_ARG_LOG_SUFFIX``
environment variable.
- A supervisor controlled Zeek cluster is now multi-logger aware. This avoids
loggers overwriting each other's log files within a single log-queue directory.
By default, a logger's name is appended to the rotated logs by zeek-archiver.
- Introduce a new command-line option ``-V`` / ``--build-info``. It produces
verbose output in JSON format about the repository state and any included
plugins.
- The X.509 certificate parser now exposes the signature type that is given inside
the signed portion of the certificate.
- The SSL parser now parses the CertificateRequest handshake message. There is a new
``ssl_certificate_request`` event and a new ``parse_distinguished_name`` function.
We also added the ``protocols/ssl/certificate-request-info`` policy script, that
adds some additional information to ``ssl.log``.
- Add logging metrics for streams (``zeek-log-stream-writes``) and writers
(``zeek-log-writer-writes-total``).
- The DNS analyzer now parses RFC 2535's AD ("authentic data") and CD ("checking
disabled") flags from DNS requests and responses, making them available in
the ``dns_msg`` record provided by many of the ``dns_*`` events. The existing
``Z`` field remains unchanged and continues to subsume the two flags, for
backward compatibility.
- The supervisor framework can now start worker nodes that read from a trace file.
- Zeek can be prevented from updating ``network_time()`` to the current time
by setting ``allow_network_time_forward=F``. Together with ``set_network_time()``
or a custom plugin, this allows control of ``network_time()`` without Zeek
interfering.
- The setting ``Pcap::non_fd_timeout`` can be used to configure the timeout
used by non-selectable packet sources in the idle case (default 20usec).
This value has previously been hard-coded, but increasing it can significantly
reduce idle CPU usage in low packet rate deployments.
- Zeek now supports a new ``@pragma`` directive. It currently allows suppressing
deprecation warnings in Zeek scripts by opening with
``@pragma push ignore-deprecations`` and closing with ``@pragma pop``.
This particularly helps in situations where use of the Zeek base scripts, for
example to populate a deprecated field for API compatibility, would otherwise
trigger deprecation warnings.
- The ``Reporter`` class was extended by a ``Deprecation()`` method to use
for logging deprecations rather than using ad-hoc ``Warning()`` calls.
- The network statistics record type features a new ``pkts_filtered`` field for
reporting the number of packets that the interface filtered before hand-off to
Zeek. Packet source implementations are free to fill this field as
feasible. The default pcap packet source does not provide this information
because its availability depends on the libpcap version.
- Packet statistics (packets received, packets dropped, bytes received, packets
seen on link, and packets filtered) are now reported to the Telemetry
framework, under the ``zeek_net`` prefix.
- Zeek's cluster framework provides the new ``get_node_count(node_type: NodeType)``
function to obtain the number of nodes for a given node type as defined in the
cluster layout. Furthermore, ``broadcast_topics`` was added as a collection of
broker topics that can be used to reach all nodes in a cluster.
- The new ``Cluster::Experimental`` namespace has been introduced to Zeek's cluster
framework to provide experimental features. Based on practical experiences and the
adoption of an experimental feature, it may become a regular feature or be removed
in future releases. Experimental features are loaded via:
``@load policy/frameworks/cluster/experimental``
- Zeek's cluster framework provides two new experimental events:
- ``cluster_started``: This event will be broadcasted from the manager once all
cluster-level connections have been established based on the given cluster layout.
If any node restarts (including the manager itself), the event will neither be
rebroadcasted nor raised locally for the restarted node.
- ``node_fully_connected``: This event will be sent to the manager and raised
locally once a cluster node has successfully conducted cluster-level handshakes
for all its outgoing connections to other cluster nodes based on the given cluster
layout.
Note: There is no tracking of cluster node connectivity. Thus, there is no guarantee
that all peerings still exist at the time of these events being raised.
- The IEEE 802.11 packet analyzer gains the ability to parse encapsulated A-MSDU
packets, instead of just dropping them. It also gains the ability to properly
recognize CCMP-encrypted packets. These encrypted packets are currently
dropped to Zeek's inability to do anything with them.
- Add packet analzyers for LLC, SNAP, and Novell 802.3, called from the Ethernet
and VLAN analyzers by default.
- Environment variables for the execution of log rotation postprocessors can
be set via ``Log::default_rotation_postprocessor_cmd_env``.
- The ``record_field`` record was extended by ``optional`` and ``record_fields()``
can now be used to determine the optionality of record fields.
- The ``ip4_hdr`` record was extended by ``DF``, ``MF``, ``offset`` and ``sum``
to aid packet-level analysis use-cases.
- Zeek now supports parsing the recently standardized DTLS 1.3. Besides the protocol
messages being correctly parsed and raising the typical SSL/TLS events, the biggest
visible change is the newly added ``ssl_extension_connection_id`` event.
- The NTP analyzer now recognizes when client and server mode messages disagree
with the notion of "originator" and "responder" and flips the connection. This
can happen in packet loss or packet re-ordering scenarios. Such connections will
have a ``^`` added to their history.
- New bifs for ``ceil()`` and ``log2()`` have been added.
- Seeds for deterministic processing can now also be set through a new environment
variable called ``ZEEK_SEED_VALUES``. The format is expected to contain 21
positive numbers separated by spaces.
Changed Functionality
---------------------
- When ``get_file_handle()`` is invoked for an analyzer that did not register
an appropriate callback function, log a warning and return a generic handle
value based on the analyzer and connection information.
- The ``&on_change`` attribute of set and tables is propagated through ``copy()``.
- Revert back to old method of preallocating ``PortVal`` objects for all valid
port numbers, as it was implemented prior to the Windows port. Not
preallocating these objects saves a minor amount of memory for short runs of
Zeek, but comes at a performance cost for having to allocate the objects every
time a new port is seen plus do map lookups for each port. This memory savings
is mostly lost for long runs of Zeek, since all of the ports will likely end
up allocated in time.
If the version from the Windows port is desired, a new configure option
``--disable-port-prealloc`` will disable the preallocation and enable the map
lookup version.
- The main-loop has been changed to process all ready IO sources with a
zero timeout in the same loop iteration. Previously, two zero-timeout
sources would require two main-loop iterations. Further, when the main-loop
is polling IO sources with file descriptors, zero timeout IO sources are
added to the list of sources to be processed as well.
The intervals to decide when Zeek checks FD-based IO sources for readiness
have been made configurable through ``io_poll_interval_default`` and
``io_poll_interval_live`` for ease of testing, development and debugging
of the main-loop.
- Zeek does not arbitrarily update ``network_time()`` to current time anymore.
When a packet source is providing a constant stream of packets, packets
drive network time. Previously, Zeek updated network time to current
time in various situations, disregarding timestamps of network packets.
Zeek will now update ``network_time()`` only when a packet source has been
inactive/idle for an interval of ``packet_source_inactivity_timeout``
(default 100msec). When a worker process suddenly observes no packets, timer
expiration may initially be delayed by ``packet_source_inactivity_timeout``.
- Calling ``suspend_processing()`` when reading traces does not update network
time to the current time anymore. Instead, Zeek keeps ``network_time()``
according to the trace file. This causes scheduled events to not fire once
``suspend_processing()`` is called, which seems more reasonable than
arbitrarily setting ``network_time()`` to current time. Processing can still
be continued from broker events or input readers.
- Previously, Zeek would process and dispatch events for the very first packet
in a trace file in order to initialize time, even if ``suspend_processing()``
was called in a ``zeek_init()`` handler. This has been changed such that the
first packet will only be processed once ``continue_processing()`` has been
invoked again. Some background around the previous behavior can be found
in GH-938. Given that the ``network_time_init()`` event explicitly signals
initialization of network time, this behavior seems more reasonable.
- If an event is scheduled with a 0.0sec timeout from a ``zeek_init()`` handler
that also invokes ``suspend_processing()``, the scheduled event will fire
immediately with ``network_time()`` still yielding ``0.0``. Previously,
``network_time()`` was set to the current time. The new behavior provides
more deterministic operation and aligns with timers stopping during a
``suspend_processing()``.
- Broker no longer initializes network time to current time when processing
input. Particularly in combination with pcap processing this was not desirable
behavior.
- The IO loop's poll interval is now correctly reduced from 100 to 10 for
live packet sources. This should lower CPU usage for deployments with
non-selectable packet sources.
- Zeek's CMake scaffolding has received an overhaul for modernizing the build
system and to make it easier to maintain going forward. Plugins can now use a
declarative interface for adding all sources, BIFs, etc. in one block instead
of using the previous begin/end functions. While the old plugin functions
still exist for backward compatibility, the underlying codebase requires newer
CMake features. Plugin authors should raise their minimum required CMake
version to 3.15, to match Zeek's.
- The IRC data analyzer does not extract DCC acknowledgements to files anymore.
Instead, ``irc_dcc_send_ack`` is raised with the bytes acknowledged by the
recipient.
- The IRC base script now use ``file_sniff()`` instead of ``file_new()`` for
DCC file transfers to capture ``fuid`` and inferred MIME type in irc.log.
- The ``ignore_checksums`` script variable now reflects the correct value
when using the ``-C`` command-line flag.
- Support for ARUBA GRE tunnels now covers all of the known protocol type values
for those tunnels.
- The vlan field reported by the AF_PACKET packet source is now properly
masked to exclude PCP and DEI bits. Previously, these bits were included
and could cause invalid vlan values > 4095 to be reported.
- Libpcap based packet source now avoids the 32bit wraparound of link and
dropped packet counters as reported by users.
- The `ssl_history` field in ssl.log indicates that the letter `j` is reserved
for hello retry requests. However, this logging was never fully implemented;
instead, hello retry requests were logged like as a server hello (with the letter
`s`). This oversight was fixed, and hello retry requests are now correctly logged.
- When per-connection SMB parser state (read offsets, tree ids, ...) exceeds
``SMB::max_pending_messages`` (default 1000), Zeek discards such per-connection
state and raises a new ``smb2_discarded_messages_state()`` event. This event is
used to reset script-layer SMB state. This change provides protection against
unbounded state growth due to partial or one-sided SMB connections.
Setting ``SMB::max_pending_messages`` to 0 can be used to switch back to the
previous behavior of not discarding state. Setting ``SMB::enable_state_clear``
to ``F`` skips the script-layer state clearing logic.
Removed Functionality
---------------------
- Mixing vector and scalar operands for binary expressions, like addition,
multiplication, etc., is now an error.
- Using deprecated ``when`` semantics without capturing variables is now an error.
- Referencing local variables in a more outer scope than where they were declared
is now an error
Deprecated Functionality
------------------------
- The cluster framework's ``worker_count`` has been deprecated in favor of the
new function ``get_active_node_count(node_type: NodeType)`` that can be used
to obtain the number of nodes of a given type the calling node is currently
connected to.
Zeek 5.2.0
==========
Breaking Changes
----------------
- Zeekctl now assigns network ports to workers starting at port 27760. This
fixes an issue where workers were starting up with ports within Linux's
ephemeral port range, and were potentially failing to startup due the ports
already being in use. This change may require changes in firewall/routing
configurations between hosts in a Zeek cluster. This should not affect
clusters running on FreeBSD, as that OS uses a different range for ephemeral
ports.
- Zeekctl support for the AF_PACKET plugin specific options (af_packet_*) has
been integrated into zeekctl directly. Upgrading to Zeek 5.2 with a builtin
AF_PACKET packet source (default on Linux) requires an upgrade of zeekctl
to the version bundled with Zeek to continue using these options.
- The blank identifier ``_`` cannot be used in expressions and options anymore.
Outside of obfuscation exercises, this should have little real-world impact.
- A new ``mysql_eof`` event has been introduced and the ``mysql_ok`` event
is not raised in its place or artificially anymore. The base scripts were
adapted accordingly. Users of ``mysql_ok()`` likely need to switch to
``mysql_eof()``.
- Zeek will now exit at startup if an external plugin (e.g. from a package) is
discovered to have the same name as a built-in plugin. See below for the
change regarding the AF_PACKET plugin now being built-in for an example of
this potentially being triggered.
- DNS query type strings were updated to match the current standardized list of
strings. This changes the string reported for a small subset of query types:
30: Changed from "EID" to "NXT"
31: Changed from "NIMLOC" to "EID"
32: Changed from "NB" to "NIMLOC"
- The ``--with-caf`` option for the ``configure`` script was removed. Broker now
requires specific versions of CAF per Zeek release, and passing an
externally-built version of CAF often lead to build failures.
New Functionality
-----------------
- Experimental support added for building and running Zeek on Microsoft Windows
environments. This is considered experimental due to the fact that our
standard testing setup (btest) doesn't run properly on Windows. This will be
fixed in the future. In the meantime we have done some basic testing against
builds done with Visual Studio 2019. Information on how to build on Windows is
available in the Zeek documentation. Note also that Spicy is currently
unsupported and will be fixed in the future.
The feature as checked into the repository is not considered production-ready.
There are many bugs to squash and features to improve, and we will be steadily
fixing things over the next few months.
The Zeek team wants to give a huge thank you to the team at Microsoft for all
of their effort in completing this port.
- Zeek container images are now being published to zeek/zeek and zeek/zeek-dev
rather than zeekurity/zeek and zeekurity/zeek-dev on Docker Hub (and continue
to be published to public.ecr.aws) Further, container images for amd64 and
arm64 platforms are now available. Main driver for the latter was to allow
usage of the official container images on Apple's M1 systems.
- Zeekctl support for using ``af_packet`` as ``lb_method`` has been added.
- New ``analyzer_confirmation_info`` and ``analyzer_violation_info`` events with
accompanying record types ``AnalyzerConfirmationInfo`` and
``AnalyzerViolationInfo`` have been added. These supersede
``analyzer_confirmation`` and ``analyzer_violation``, which have been
deprecated.
- Added helpers to determine protocol, packet or file analyzer based on
``AllAnalyzers::Tag`` values named ``is_protocol_analyzer()``,
``is_packet_analyzer()`` and ``is_file_analyzer()``.
- File analyzers can now raise analyzer violations to the script-layer via the
new ``AnalyzerViolation()`` method.
- Packet and file analyzers can now be disabled and enabled at runtime using the
``Analyzer::enable_analyzer()`` and ``Analyzer::disable_analyzer()``
wrappers. While initially for protocol analyzers only, these have been
extended to work for packet and file analyzers. This now allows to leverage
``Analyzer::disabled_analyzers`` for these kinds of analyzers.
- The blank identifier ``_`` can now be used to ignore loop variables of
different types without type clash errors. This allows to do the following
within the same scope:
local vec = vector("a", "b", "c");
for ( _, v in vec )
print v;
for ( i, _ in vec )
print v;
Iterating over only the values of a table can be done by ignoring the full
index with a single blank identifier. Due to the internal structure of Zeek
tables, this can result in a performance improvement.
local tab = table(["a", 1, T] = "a1T", ["b", 2, F] = "b2f");
for ( _, v in tab )
print v;
It's also possible ignore individual indices of different types with the blank
identifier ``_`` as follows:
for ( [_, i, _], v in tab )
print i, v;
As noted under breaking changes, the blank identifier ``_`` cannot be
referenced in expression anymore.
- It is now possible to put trailing commas within table, vector, set and record
construction. For example, the following code is now valid, which can make for
more uniform style and smaller diffs.
local vec = vector(
"1",
"2",
);
local tab: table[string] of count = [
["a"] = 1,
["b"] = 2,
];
Function calls and record constructors can have a trailing comma after the
last argument.
Analyzer::schedule_analyzer(
chan$orig_h,
chan$resp_h,
chan$resp_p,
Analyzer::ANALYZER_FTP_DATA,
5mins,
);
- Re-introduce event groups. Allow the ``&group`` attribute on event and hook
handlers for annotating them with one or more event groups. These groups can
be disabled and enable during runtime. Disabling an event group implies
disabling all event and hook handlers that are part of it.
The main difference to a previous implementation in (very) old Zeek versions
is its granularity: It is now possible to toggle individual event handlers
(event handler bodies). The original implementation worked at the level of
events, disabling or enabling all event handlers for a given event at once.
Additionally, support for hooks was added as these are structurally similar to
events.
The BIFs ``disable_event_group()`` and ``enable_event_group()`` are
re-instantiated and allow controlling event groups based on the group
attribute.
Additionally, event and hook handlers are implicitly placed into event module
groups based on the module they are implemented in. All events implemented in
a given module can be toggled with ``disable_module_events()`` and
``enable_module_events()``.
- Extend the ``Logging::Stream`` record with an ``event_groups`` field and
toggle these during ``Log::disable_stream`` and ``Log::enable_stream``
invocations. This allows for explicit/manual opt-in performance optimizations
by turning off event handlers at runtime that are only needed for log
generation.
- On Linux, the AF_PACKET packet source plugin
(https://github.com/zeek/zeek-af_packet-plugin) is included as a builtin
plugin by default. To select this packet source, prefix the interface name
with ``af_packet``.
zeek -i af_packet::eth0
- Usage of ``break`` and ``next`` statements is now validated. It was previously
possible to place these outside of ``for``, ``while`` or ``switch`` statements
without any error indication.
- Add two BIFs ``get_identifier_declaring_script()`` and
``get_record_field_declaring_script()`` to query the declaring scripts for
identifiers and record fields from Zeek scripts.
- Extend the SSH analyzer to produce new events (``ssh2_ecc_init``,
``ssh2_gh_gex_init``, ``ssh2_gss_init``, ssh2_rsa_secret``) to detect when SSH
client and server roles are reversed.
- Analyzers found in the new ``Analyzer::requested_analyzers`` set will be
enabled at ``zeek_init()`` time. The set can be populated via
:zeek:see:`redef`. This change only has an effect in settings where
``Analyzer::disable_all`` is changed to ``T``. By default, all analyzers
continue to be enabled.
- A new ``analyzer.log`` was added to log all analyzer violations and optionally
analyzer confirmations. This log can be useful during development of new
analyzers as well as for collecting operational data in production
environments.
- Expose configurability of for SQLite's synchronous and journal_mode PRAGMAs
for SQLite backed Broker data stores. Setting these to synchronous=normal
and journal_mode=wal can significantly improve throughput at the cost of
some durability in the presence of power loss or OS crash. In the context
of Zeek, this is likely more than acceptable.
Additionally, add integrity_check and failure_mode options to support
detecting and deleting corrupted SQLite database at store initialization.
- A new ``join_string_set`` BIF was added, replacing the existing script-level
version from utils/strings.zeek.
- A new ``&ordered`` attribute for tables and sets was added. This attribute
causes iteration over a table/set to return elements in the order of their
insertion.
- A new ``-D`` argument was added to the ``configure`` script to allow passing
parameters directly to the underlying CMake call.
- Added parsing for the challenge and response fields to the NTLM analyzer.
- A new ``FTP::max_command_length`` value was added to script-land, defaulting
to 100. This value is used by the FTP analyzer to limit the size of commands
accepted by the analyzer. A ``FTP_max_command_length_exceeded`` weird is
raised for any violations of that length.
- The MySQL analyzer has been extended to detect when client and server negotiate
to use a SSL encrypted session. This allows analysis of the subsequent SSL
handshake. The service field for encrypted MySQL connections in the conn.log
will have entries for both, mysql and ssl.
Changed Functionality
---------------------
- Violations for packet analyzers that have sessions attached with them will be
raised once only. Further, analyzer confirmations are not raised after a
violation.
- The parameter given to ``enum_names()`` can now be a string naming the enum
type, rather than the type itself.
- The ``type_name`` of enum types produced by ``record_fields()`` now includes
the actual type name rather than just ``"enum"``.
- Passing non-string ``sep`` and ``def`` arguments to ``cat_sep()`` isn't a
fatal error anymore. More descriptive error messages are produced, too.
- The number of analyzer violation events that can be raised by protocol
analyzer instances is now capped by the const
``max_analyzer_violation_events``.
- The number of analyzer violation events that can be raised by protocol and
file analyzer instances is now capped by the const
``max_analyzer_violation_events``. Its default is 1000 and the main purpose
is to prevent analyzers from scheduling too many ``analyzer_violation_info``
events before the DPD ``max_violations`` script-level logic has a chance to
run and disable the problematic analyzer.
- The TCP analyzer now continues processing payload for some
connections missing initial packets where it would previously have
stopped. This fixes a few cases where we already had the logic to
continue in place, but we still ended up considering them partial.
- Count underflows via ``--c`` or subtract from statements (``c = c - 1``) are
now consistently warned about. Previously, underflows through ``--c`` were
treated as runtime errors, while "subtract from" underflows were silently
accepted. The following (surprising behavior) now causes a warning, too:
$ zeek -e 'print 1 - 2'
expression warning in <command line>, line 1: count underflow (1 - 2)
18446744073709551615
- The MQTT scripts registering the analyzer and DPD signatures have been moved
from the policy folder to base and are loaded by default.
- Notices created for files transferred over multiple connections will now be
associated with one of the connections rather than none.
- The MySQL analyzer has been switched to parse in little endian. This avoids
analyzer violations due to out of bound errors for length encoded strings.
- Non-fatal errors when setting up BPF filtering will no longer cause Zeek to
exit, but instead will log the error in reporter.log and continue processing.
- The languages reported for the ``keyboard_layout`` field in rdp.log were
updated to match the current standardized set of languages. Unknown layout
values now attempt to fallback to a "parent" layout if one is available.
- In the cluster management framework, the controller now supports Broker's
WebSocket data transport for communication with clients. It listens on TCP
port 2149 for this purpose. zeek-client now likewise uses the WebSocket
transport, removing its runtime dependency on the Broker library and enabling
standalone installation. The client still bundles with Zeek by default but is
now also available on PyPI and installable via ``pip install zeek-client``.
The documentation provides additional details.
Deprecated Functionality
------------------------
- The global ``disabling_analyzer()`` hook has been deprecated and replaced
with ``Analyzer::disabling_analyzer()`` that has the same semantics.
- The ``analyzer_confirmation`` and ``analyzer_violation`` events have been
deprecated in favor of the more generic ``analyzer_confirmation_info`` and
``analyzer_violation_info`` events.
- The const values for toggling individual tunnel packet analyzers have been
deprecated in favor of using ``Analyzer::disable_analyzer()`` directly. This
affects:
Tunnel::enable_ip
Tunnel::enable_ayiya
Tunnel::enable_teredo
Tunnel::enable_gtpv1
Tunnel::enable_gre
Setting these explicitly to F can be achieved by leveraging
``Analyzers::disabled_analyzers``, for example:
redef Analyzer::disabled_analyzers += { PacketAnalyzer::ANALYZER_GRE };
- The ``zeek::merge_type_list()`` function has been deprecated. Please consider
the partially compatible and saner ``zeek::maximal_type()`` instead. See
GH-2604 for context.
- The pre-authentication data field (pa_data) available in certain Kerberos
events now exposes the (encrypted) PA-ENC-TIMESTAMP field (padata-type=2).
- The ``SupressWeirds()`` method in the ContentLine analyzer was deprecated in
favor of the correctly-spelled ``SuppressWeirds()`` method.
- The `bro` symlink has finally been removed.
Zeek 5.1.0
==========
Breaking Changes
----------------
- The ``Packet::{l2,l3}_checksummed`` variables were reworked to correctly match
the network layers that they apply to. A new ``Packet::l4_checksummed``
variable was added to cover the transport layer. See this GitHub issue for
more detail: https://github.com/zeek/zeek/issues/2183.
- The STREAM mode of the ASCII reader now behaves like `tail -F`: when file is
removed/replaced, it will start tracking the new file. See
https://github.com/zeek/zeek/pull/2097 for more detail
- The Dictionary and PDict classes are now C++ templates. This may cause
plugin/package builds to fail due to needing to modify uses of them to match.
- By default, ``files.log`` does not have the fields ``tx_hosts``, ``rx_hosts``
and ``conn_uids`` anymore. These have been replaced with the more commonly
used ``uid`` and ``id`` fields. They can be re-instantiated by loading the
following policy script through ``local.zeek``:
@load frameworks/files/deprecated-txhosts-rxhosts-connuids
Note, however, that this script will be removed with Zeek 6.1. Consumers
of ``files.log`` should convert to using the singular ``uid`` and ``id``
fields instead.
- The ``files.log`` is now unrolled consistently. That is, when Zeek associates
multiple connections with a single file, each of these connections will result
in individual ``files.log`` entries with unique connection uids, all sharing
the same file uid.
This unrolling behavior always existed in a Zeek cluster when the network
connections involved in a file transfer are load-balanced to different
workers. Due to this affecting only a marginal ratio of files on real-world
networks, unrolling the log was chosen as the more efficient approach over
making the current logic cluster aware.
The ``seen_bytes`` and ``missing_bytes`` fields of a ``File::Info`` record
continue to represent the total number across all connections seen by the
current instance of Zeek.
- The barnyard2 policy scripts have been removed. The integration with the
Barnyard2 project used the pre-Broker Broccoli library, which got removed in
Zeek 3.0.
- The unified2 analyzer and accompanying scripts have been removed without
deprecation.
- The return value of ``packet_analysis::IP::ParsePacket`` has changed to return
enum values. This makes it easier to look at the result and immediately know
what it means. Unfortunately, because we can't overload a method on the return
value alone, we aren't able to deprecate the original version of the method.
This may cause build of packages to fail if they were using this method.
- Conditional directives (``@if``, ``@ifdef``, ``@ifndef``, ``@else`` and
``@endif``) can not be placed directly following ``if``, ``for`` or ``while``
statements anymore. This was interpreted non-intuitively and could lead to
subtle bugs. The statement following the directive was placed outside of its
intended block. Placing braces after ``if``, ``for`` or ``while`` should
result in the intended behavior.
- The ``bro`` symlink to the ``zeek`` binary has finally been removed.
- The connection's service field does not hold "negated" analyzer names
anymore (analyzers that triggered violations after a confirmation).
These analyzers are now tracked in a separate ``service_violation``
field with their respective enum values instead.
New Functionality
-----------------
- Added support for the /s regular expression modifier. Using this modifier in
patterns in Zeek scripts will cause the '.' character to also match newline
characters.
- Added a new telemetry framework for providing high-level access to Zeek's
metric subsystem. This framework allows script writers to use different metric
types (counters, gauges and histograms) for tracking metrics without using
lower-level BiFs from ``telemetry.bif``. Additionally, metrics can now be
accessed from script land using ``Telemetry::collect_metrics()`` and
``Telemetry::collect_histogram_metrics()``.
The framework is located in ``base/frameworks/telemetry``.
In addition to the Prometheus endpoint for metrics export that has existed
since Zeek 4.1, two new log streams, ``telemetry.log`` and
``telemetry_histogram.log``, can be enabled by loading
``policy/frameworks/telemetry/log``. This policy script is included in
``local.zeek`` by default.
For further details on the framework and examples, please refer to the
Zeek documentation.
- Allow redef'ing the ``&log`` attribute of record fields:
redef Notice::Info$email_dest -= { &log };
While the syntax allows for any attribute, only ``&log`` is supported. The
semantics for other record field attributes are not easy to grasp and there
were no obvious use-cases identified.
- Introduced a global ``disabling_analyzer()`` hook to allow vetoing calls
to ``disable_analyzer()``.
The contract is simple: Any script can veto a ``disable_analyzer()`` call by
breaking from this hook. The decision is local to the script taking into
account any state attached to the connection or state stored elsewhere.
A script breaking from the hook takes over responsibility to call
``disable_analyzer()`` at a later point when it finds the condition due
to which it vetoed fulfilled (which may be never).
- Add support for iterating over indices and values of a vector using the
same syntax as used for iterating over key-value pairs of tables, where
``value`` will be set to ``vec[idx]``.
local vec = vector("zero", "one", "two");
for ( idx, value in vec )
print idx, value;
- The Supervisor framework now allows better control over where to place
additional scripts in the load sequence of new nodes. It previously always
loaded such scripts after any other user scripts, which could create pitfalls
when users expected their scripts to run last. Scripts placed in
``NodeConfig``'s new ``addl_base_scripts`` and ``addl_user_scripts`` fields
will be loaded after the base scripts (and thus before any user scripts) and
after any user scripts, respectively. The old ``NodeConfig$scripts` field
still adds to the very end and is deprecated.
- Added a new script-level option ``max_changes_per_connection`` to limit the
number of ``tunnel_changed`` events that can be sent for a connection. This
helps prevent log spam from connections that regularly swap. The option
defaults to 5, and can be set to zero do disable the limiting.
- Added a new BIF ``bytestring_to_float`` for converting 4-byte bytestrings to
float values.
- Added a new BIF ``pow``.
- Added new bit-shift operators ``<<`` and ``>>`` for use in scripts.
- Added a new BIF ``table_keys`` which returns a ``set`` of keys from a table.
- Added a new BIF ``table_values`` which returns a ``vector`` of keys from a
table.
- Added new fields to the Modbus log for the Modbus PDU type, the transaction
ID, and the unit ID. See https://github.com/zeek/zeek/pull/2281 for more
information.
- Added support for parsing TCP option 27, and fixed validation of lengths for
TCP options 28, 29, and 34.
- Added new packet-analyzer to handle the DLT_LINUX_SLL2 PCAP link type.
Changed Functionality
---------------------
- The SSL analyzer now determines the direction of the SSL/TLS session by examining
the packets, and no longer assumes that the connection originator is the client.
Due to this, the ``is_orig`` field in all SSL/TLS events was renamed to ``is_client``.
Furthermore, the ``ssl_history`` now can indicate that the connection was flipped
(meaning that it is not in the normal order of the originator is the client) using
the ``^`` character. A new ``ssl_connection_flipped`` is raised when the connection
is flipped. Furthermore, a ``SSL_unclear_connection_direction`` weird is raised when
we cannot determine the connection direction, because both sides of the connection
send packets that are associated with being a client/server.
- The default logging directory is now set globally across all log
writers through ``Log::default_logdir``.
- Calling `Option::set()` when Zeek is terminating is now a noop and returns `F`.
This prevents callbacks into script-land through change handlers when parts
of the environment have already been torn down.
- When running in cluster mode, the manager by default now imports metrics from
all other cluster nodes and opens port 9911/tcp for Prometheus metrics exposition.
- The ``smb2_file_delete`` event will now be raised for SMB2 ``CREATE`` requests
marked with the ``FILE_DELETE_ON_CLOSE`` option.
- Fixed ``bytestring_to_count`` to handle 3-, 5-, 6-, and 7-byte strings.
Removed Functionality
---------------------
- The barnyard2 policy scripts have been removed.
- The unified2 analyzer and accompanying scripts have been removed.
Deprecated Functionality
------------------------
- The ``PDict`` class is now an alias to ``Dictionary`` and has been
deprecated. Use ``Dictionary`` directly, passing a pointer type to the
template.
- ``LogAscii::logdir`` and per-writer log directories have been deprecated in
favor of the new ``Log::default_logdir``.
- The ``HOOK_BRO_OBJ_DTOR`` hook and associated methods have been
deprecated. They are replaced by the ``HOOK_OBJ_DTOR`` hook and methods.
- The ``bro_int_t`` and ``bro_uint_t`` types have been deprecated and replaced
by ``zeek_int_t`` and ``zeek_uint_t``.
- The ``bro_inet_ntop.h`` and ``bro_inet_ntop.c`` files have been deprecated and
replaced by ``zeek_*`` files.
- The ``BRO_PLUGIN_API_VERSION`` has been deprecated and replaced by
``zeek::PLUGIN_API_VERSION``.
- The ``misc/scan.zeek`` script has been marked for removal in Zeek 6.1. Use
github.com/ncsa/bro-simple-scan instead.
- The Supervisor framework's ``NodeConfig$scripts`` field has been deprecated
and marked for removal in Zeek 6.1. Use ``NodeConfig$addl_user_scripts``
instead.
Zeek 5.0.0
==========
Breaking Changes
----------------
- Zeek now requires at least CMake version 3.15.0.
- If Zeek is configured with support for included Spicy (the default) we now
require at least Flex version 2.6 and its development headers, at least Bison
version 3.3, and GCC version 8.3 or Clang version 9.0 or higher.
- The script-land ``union`` and ``timer`` types have been removed. They haven't
had any actual semantics backing them for some time and shouldn't have
functioned in any useable way. We opted to skip the deprecation cycle for
these types for that reason.
- Broker now uses a new network backend with a custom network protocol that is
incompatible with the pre-5.0 backend. In practice, this means Zeek 4.x will
not be able to exchange events with Zeek 5.x. Going forward, this new backend
will allow us to keep the Broker protocol more stable and add new capabilities
in a backwards compatible way.
New Functionality
-----------------
- The Zeek cluster management framework now supports operational use of
single-instance clusters, meaning setups in which all Zeek cluster nodes
(manager, workers, etc) reside on a single machine. The framework builds upon
Zeek's Supervisor (``-j`` is a requirement) and includes three components:
(1) A cluster controller, loaded via ``policy/frameworks/management/controller`.
The controller is the central management entity for a Zeek cluster. It
interacts with one or more agents to enact instructions delivered by a
management client. The controller is stateful and can persist cluster
state to disk.
(2) One or more cluster agents, loaded via ``policy/frameworks/management/agent`.
Each agent assists the controller in managing the local instance (typically
a machine or container) of Zeek cluster nodes. Agents interact with the
local Supervisor to control nodes and peer directly with running Zeek
nodes via Broker to carry out node-local tasks.
(3) zeek-client, the cluster management client. A standalone client running
outside of Zeek, it's now installed by default, alongside the other
executables. The client supports uploading cluster configurations to the
controller, deploying them, retrieving them, checking cluster node status,
restarting individual cluster nodes, and retrieving the current value(s)
of global identifiers in one or more Zeek nodes.
Controller and agent come pre-configured for single-instance deployments of
Zeek clusters, with automated log rotation and archival via Zeek's
``zeek-archiver`` tool. For further details on the framework, please refer
to the Zeek documentation.