Skip to content

Latest commit

 

History

History
762 lines (687 loc) · 58.5 KB

CHANGELOG.next.asciidoc

File metadata and controls

762 lines (687 loc) · 58.5 KB
Raw

Beats version HEAD

Breaking changes

Affecting all Beats

  • The document id fields has been renamed from @metadata.id to @metadata._id 15859

  • Variable substitution from environment variables is not longer supported. https://github.com/elastic/beats/pull/15937{15937}

  • Change aws_elb autodiscover provider field name from elb_listener.* to aws.elb.*. 16219 https://github.com/elastic/beats/pull/16402{16402}

  • Remove AddDockerMetadata and AddKubernetesMetadata processors from the script processor. They can still be used as normal processors in the configuration. 16349 16514

  • Introduce APM libbeat instrumentation, active when running the beat with ELASTIC_APM_ACTIVE=true. 17938

  • Remove the non-ECS agent.hostname field. Use the agent.name or agent.id fields for an identifier. 16377 18328

  • Make error message about locked data path actionable. 18667

  • Ensure dynamic template names are unique for the same field. 18849

  • Remove the deprecated xpack.monitoring. settings. Going forward only monitoring. settings may be used. 9424 18608

  • Added certificate TLS verification mode to ignore server name mismatch. 12283 20293

  • Autodiscover doesn’t generate any configuration when a variable is missing. Previously it generated an incomplete configuration. 20898

  • Remove redundant cloudfoundry.*.timestamp fields. This value is set in @timestamp. 21175

Auditbeat

  • File integrity dataset (macOS): Replace unnecessary file.origin.raw (type keyword) with file.origin.text (type text). 12423 15630

  • Change event.kind=error to event.kind=event to comply with ECS. 18870 20685

  • Change network.direction values to ECS recommended values (inbound, outbound). 12445 20695

Filebeat

  • Improve ECS field mappings in panw module. event.outcome now only contains success/failure per ECS specification. 16025 17910

  • Improve ECS categorization field mappings for nginx module. http.request.referrer only populated when nginx sets a value 16174 17844

  • Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . 16180 17982

  • With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta) will no longer send the host field that contains information about the host Filebeat is running on. This is because the host field specifies the host on which the event happened. 13920 18223

  • With the default configuration the following modules will no longer send the host field that contains information about the host on which Filebeat is running. You can revert this change by configuring tags for the module and omitting forwarded from the list. 13920

    • CEF 18223

    • PANW 18223

    • Cisco 18753

    • CrowdStrike 19132

    • Fortinet 19133

    • iptables 18756

    • Checkpoint 18754

    • Netflow 19087

    • Zeek 19113 (forwarded tag is not included by default)

    • Suricata 19107 (forwarded tag is not included by default)

    • CoreDNS 19134 (forwarded tag is not included by default)

    • Envoy Proxy 19134 (forwarded tag is not included by default)

  • Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. 18154 18359

  • Adds check on <no value> config option value for the azure input resource_manager_endpoint. 18890

  • Okta module now requires objects instead of JSON strings for the http_headers, http_request_body, pagination, rate_limit, and ssl variables. 18953

  • Adds oauth support for httpjson input. 18415 18892

  • Adds split_events_by option to httpjson input. 19246

  • Adds date_cursor option to httpjson input. 19483

  • Adds Gsuite module with SAML support. 19329

  • Adds Gsuite User Accounts support. 19329

  • Adds Gsuite Login audit support. 19702

  • Adds Gsuite Admin support. 19769

  • Adds Gsuite Drive support. 19704

  • Adds Gsuite Groups support. 19725

  • Move file metrics to dataset endpoint 19977

  • Add while_pattern type to multiline reader. 19662

  • Tracking session end reason in panw module. 18705

  • Fix PANW field spelling "veredict" to "verdict" on event.action 18808

  • Removed experimental modules citrix, kaspersky, rapid7 and tenable. 20706

  • Add support for GMT timezone offsets in decode_cef. 20993

Heartbeat

Journalbeat

  • Improve parsing of syslog.pid in journalbeat to strip the username when present 16116

Metricbeat

  • Make use of secure port when accessing Kubelet API 16063

  • Add Tomcat overview dashboard 14026

  • Move service config under metrics and simplify metric types. 18691

  • Fix ECS compliance of user.id field in system/users metricset 19019

  • Rename googlecloud stackdriver metricset to metrics. 19718

Packetbeat

  • Redis: fix incorrectly handle with two-words redis command. 14872 14873

  • event.category no longer contains the value network_traffic because this is not a valid ECS event category value. 20556

Winlogbeat

  • Add support to Sysmon file delete events (event ID 23). 18094

  • Improve ECS field mappings in Sysmon module. related.hash, related.ip, and related.user are now populated. 18364

  • Improve ECS field mappings in Sysmon module. Hashes are now also populated to the corresponding process.hash, process.pe.imphash, file.hash, or file.pe.imphash. 18364

  • Improve ECS field mappings in Sysmon module. file.name, file.directory, and file.extension are now populated. 18364

  • Improve ECS field mappings in Sysmon module. rule.name is populated for all events when present. 18364

  • Add Powershell module. Support for event ID’s: 400, 403, 600, 800, 4103, 4014, 4105, 4106. 16262 18526

  • Fix Powershell processing of downgraded engine events. 18966

  • Fix unprefixed fields in fields.yml for Powershell module 18984

  • Remove top level hash property from sysmon events 20653

Functionbeat

Bugfixes

Affecting all Beats

  • Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data 17223

  • Fix add_cloud_metadata to better support modifying sub-fields with other processors. 13808

  • TLS or Beats that accept connections over TLS and validate client certificates. 14146

  • Fix panics that could result from invalid TLS certificates. This can affect Beats that connect over TLS, or Beats that accept connections over TLS and validate client certificates. 14146

  • Fix panic in the Logstash output when trying to send events to closed connection. 15568

  • Fix missing output in dockerlogbeat 15719

  • Fix logging target settings being ignored when Beats are started via systemd or docker. 12024 15442

  • Do not load dashboards where not available. 15802

  • Fix issue where TLS settings would be ignored when a forward proxy was in use. https://github.com/elastic/beats/pull/15516{15516}

  • Update replicaset group to apps/v1 15802

  • Fix issue where default go logger is not discarded when either * or stdout is selected. 10251 15708

  • Upgrade go-ucfg to latest v0.8.1. https://github.com/elastic/beats/pull/15937{15937}

  • Fix index names for indexing not always guaranteed to be lower case. 16081

  • Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

  • Fix loading processors from annotation hints. 16348

  • Fix an issue that could cause redundant configuration reloads. 16440

  • Fix k8s pods labels broken schema. 16480

  • Fix k8s pods annotations broken schema. 16554

  • Upgrade go-ucfg to latest v0.8.3. https://github.com/elastic/beats/pull/16450{16450}

  • Fix NewContainerMetadataEnricher to use default config for kubernetes module. 16857

  • Improve some logging messages for add_kubernetes_metadata processor 16866

  • Fix k8s metadata issue regarding node labels not shown up on root level of metadata. 16834

  • Fail to start if httpprof is used and it cannot be initialized. 17028

  • Fix concurrency issues in convert processor when used in the global context. 17032

  • Fix bug with monitoring.cluster_uuid setting not always being exposed via GET /state Beats API. 16732 17420

  • Fix building on FreeBSD by removing build flags from add_cloudfoundry_metadata processor. 17486

  • Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. 17613

  • Fix goroutine leak and Elasticsearch output file descriptor leak when output reloading is in use. 10491 17381

  • Fix setup.dashboards.index setting not working. 17749

  • Fix Elasticsearch license endpoint URL referenced in error message. 17880 18030

  • Fix panic when assigning a key to a nil value in an event. 18143

  • Gives monitoring reporter hosts, if configured, total precedence over corresponding output hosts. 17937 17991

  • Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • [Autodiscover] Check if runner is already running before starting again. 18564

  • Fix keystore add hanging under Windows. 18649 18654

  • Fix an issue where error messages are not accurate in mapstriface. 18662 18663

  • Fix regression in add_kubernetes_metadata, so configured indexers and matchers are used if defaults are not disabled. 18481 18818

  • Fix potential race condition in fingerprint processor. 18738

  • Add better handling for Kubernetes Update and Delete watcher events. 18882

  • Fix the translate_sid processor’s handling of unconfigured target fields. 18990 18991

  • Fixed a service restart failure under Windows. 18914 18916

  • The monitoring.elasticsearch.api_key value is correctly base64-encoded before being sent to the monitoring Elasticsearch cluster. 18939 18945

  • Fix kafka topic setting not allowing upper case characters. 18854 18640

  • Fix redis key setting not allowing upper case characters. 18854 18640

  • Fix config reload metrics (libbeat.config.module.start/stops/running). 19168

  • Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed 18979

  • Server-side TLS config now validates certificate and key are both specified 19584

  • Fix terminating pod autodiscover issue. 20084

  • Fix seccomp policy for calls to chmod and chown. 20054

  • Remove unnecessary restarts of metricsets while using Node autodiscover 19974

  • Output errors when Kibana index pattern setup fails. 20121

  • Fix issue in autodiscover that kept inputs stopped after config updates. 20305

  • Log debug message if the Kibana dashboard can not be imported from the archive because of the invalid archive directory structure 12211, 13387

  • Add service resource in k8s cluster role. 20546

  • [Metricbeat][Kubernetes] Change cluster_ip field from ip to keyword. 20571

  • Rename cloud.provider az value to azure inside the add_cloud_metadata processor. 20689

  • Add missing country_name geo field in add_host_metadata and add_observer_metadata processors. 20796 20811

  • [Autodiscover] Handle input-not-finished errors in config reload. 20915

  • Explicitly detect missing variables in autodiscover configuration, log them at the debug level. 20568 20898

Auditbeat

  • system/socket: Fixed compatibility issue with kernel 5.x. 15771

  • system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

  • system module: Fix panic during initialisation when /proc/stat can’t be read. 17569

  • system/package: Fix an error that can occur while trying to persist package metadata. 18536 18887

  • system/socket: Fix dataset using 100% CPU and becoming unresponsive in some scenarios. 19033 19764

  • system/socket: Fixed tracking of long-running connections. 19033

  • system/package: Fix librpm loading on Fedora 31/32. NNNN

  • file_integrity: Create fsnotify watcher only when starting file_integrity module 19505

  • auditd: Fix spelling of anomaly in event.category.

  • auditd: Fix typo in event.action of removed-user-role-from. 19300

  • auditd: Fix typo in event.action of used-suspicious-link. 19300

  • system/socket: Fix kprobe grouping to allow running more than one instance. 20325

Filebeat

  • Fix mapping of fortinet.firewall.mem as integer. 19335

  • Ensure all zeek timestamps include millisecond precision. 14599 16766

  • Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. 15502 15590

  • Add shared_credential_file to cloudtrail config 15652 15656

  • Fix typos in zeek notice fileset config file. 15764 15765

  • Fix mapping error when zeek weird logs do not contain IP addresses. 15906

  • Improve elasticsearch/audit fileset to handle timestamps correctly. 15942

  • Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the elasticsearch module. 15840 15900

  • Fix mapping error for cloudtrail additionalEventData field 16088

  • Fix a connection error in httpjson input. 16123

  • Fix s3 input with cloudtrail fileset reading json file. 16374 16441

  • Rewrite azure filebeat dashboards, due to changes in kibana. 16466

  • Adding the var definitions in azure manifest files, fix for errors when executing command setup. 16270 16468

  • Fix merging of fileset inputs to replace paths and append processors. https://github.com/elastic/beats/pull/16450{16450}

  • Add queue_url definition in manifest file for aws module. https://github.com/elastic/beats/pull/16640{16640}

  • Fix issue where autodiscover hints default configuration was not being copied. 16987

  • Fix Elasticsearch _id field set by S3 and Google Pub/Sub inputs. 17026

  • Fixed various Cisco FTD parsing issues. 16863 16889

  • Fix default index pattern in IBM MQ filebeat dashboard. 17146

  • Fix elasticsearch.gc fileset to not collect all logs when Elasticsearch is running in Docker. 13164 16583 17164

  • Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. 17216 17220

  • CEF: Fixed decoding errors caused by trailing spaces in messages. 17253

  • Fixed a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. 17242 17243

  • Fixed MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17086 17156

  • Fix elasticsearch.audit data ingest pipeline to be more forgiving with date formats found in Elasticsearch audit logs. 17406

  • Fixed activemq module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17428

  • Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. 17425

  • Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. 17734 17735

  • Fixed cloudfoundry.access to have the correct cloudfoundry.app.id contents. 17847

  • Fixing ingress_controller. fields to be of type keyword instead of text. 17834

  • Fixed typo in log message. 17897

  • Fix Cisco ASA ASA 3020** and 106023 messages 17964

  • Unescape file name from SQS message. 18370

  • Improve cisco asa and ftd pipelines' failure handler to avoid mapping temporary fields. 18391 18392

  • Fix source.address not being set for nginx ingress_controller 18511

  • Fix PANW module wrong mappings for bytes and packets counters. 18522 18525

  • Fixed ingestion of some Cisco ASA and FTD messages when a hostname was used instead of an IP for NAT fields. 14034 18376

  • Fix a rate limit related issue in httpjson input for Okta module. 18530 18534

  • Fix googlecloud.audit pipeline to only take in fields that are explicitly defined by the dataset. 18465 18472

  • Fix o365.audit failing to ingest events when ip address is surrounded by square brackets. 18587 18591

  • Fix Kubernetes Watcher goroutine leaks when input config is invalid and input.reload is enabled. 18629 18630

  • Okta module now sets the Elasticsearch _id field to the Okta UUID value contained in each system log to minimize the possibility of duplicating events. 18953

  • Fix improper nesting of session_issuer object in aws cloudtrail fileset. 18894 18915

  • Fix o365 module ignoring var.api settings. 18948

  • Fix netflow module to support 7 bytepad for IPFIX template. 18098

  • Fix Cisco ASA dissect pattern for 313008 & 313009 messages. 19149

  • Fix date and timestamp formats for fortigate module 19316

  • Fix memory leak in tcp and unix input sources. 19459

  • Add missing default_field: false to aws filesets fields.yml. 19568

  • Fix tls mapping in suricata module 19492 19494

  • Update container name for the azure filesets. 19899

  • Fix bug with empty filter values in system/service 19812

  • Fix S3 input to trim delimiter /n from each log line. 19972

  • Ignore missing in Zeek module when dropping unecessary fields. 19984

  • Fix auditd module syscall table for ppc64 and ppc64le. 20052

  • Fix Filebeat OOMs on very long lines 19500, 19552

  • Fix s3 input parsing json file without expand_event_list_from_field. 19902 19962 20370

  • Fix millisecond timestamp normalization issues in CrowdStrike module 20035, 20138

  • Fix support for message code 106100 in Cisco ASA and FTD. 19350 20245

  • Fix event.outcome logic for azure/siginlogs fileset 20254

  • Fix fortinet setting event.timezone to the system one when no tz field present 20273

  • Fix okta geoip lookup in pipeline for destination.ip 20454

  • Fix mapping exception in the googlecloud/audit dataset pipeline. 18465 20465

  • Fix cisco asa and ftd parsing of messages 106102 and 106103. 20469

  • Improve validation checks for Azure configuration 20369 20389

  • Fix event.kind for system/syslog pipeline 20365 20390

  • Clone value when copy fields in processors to avoid crash. 19206 20500

  • Fix event.type for zeek/ssl and duplicate event.category for zeek/connection 20696

  • Fix long registry migration times. 20717 20705

  • Fix event types and categories in auditd module to comply with ECS 20652

  • Update documentation in the azure module filebeat. 20815

  • Provide backwards compatibility for the set processor when Elasticsearch is less than 7.9.0. 20908

  • Remove wrongly mapped tls.client.server_name from fortinet/firewall fileset. 20983

  • Fix an error updating file size being logged when EOF is reached. 21048

  • Fix error when processing AWS Cloudtrail Digest logs. 21086 20943

  • Provide backwards compatibility for the append processor when Elasticsearch is less than 7.10.0. 21159

Heartbeat

  • Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. 15639

  • Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. 17549

  • Add support for new service_name option to all monitors. 19932.

  • Stop rescheduling tasks of stopped monitors. 20570

Heartbeat

Journalbeat

Metricbeat

  • Add dedot for tags in ec2 metricset and cloudwatch metricset. 15843 15844

  • Use RFC3339 format for timestamps collected using the SQL module. 15847

  • Avoid parsing errors returned from prometheus endpoints. 15712

  • Change lookup_fields from metricset.host to service.address 15883

  • Add dedot for cloudwatch metric name. 15916 15917

  • Fixed issue logstash-xpack module suddenly ceasing to monitor Logstash. 15974 16044

  • Fix skipping protocol scheme by light modules. pull

  • Made logstash-xpack module once again have parity with internally-collected Logstash monitoring data. 16198

  • Change sqs metricset to use average as statistic method. 16438

  • Revert changes in docker module: add size flag to docker.container. 16600

  • Fix diskio issue for windows 32 bit on disk_performance struct alignment. 16680

  • Fix detection and logging of some error cases with light modules. 14706

  • Fix imports after PR was merged before rebase. 16756

  • Add dashboard for redisenterprise module. 16752

  • Dynamically choose a method for the system/service metricset to support older linux distros. 16902

  • Use max in k8s apiserver dashboard aggregations. 17018

  • Reduce memory usage in elasticsearch/index metricset. 16503 16538

  • Check if CCR feature is available on Elasticsearch cluster before attempting to call CCR APIs from elasticsearch/ccr metricset. 16511 17073

  • Use max in k8s overview dashboard aggregations. 17015

  • Fix Disk Used and Disk Usage visualizations in the Metricbeat System dashboards. 12435 17272

  • Fix missing Accept header for Prometheus and OpenMetrics module. 16870 17291

  • Further revise check for bad data in docker/memory. 17400

  • Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

  • Combine cloudwatch aggregated metrics into single event. 17345

  • Fix how we filter services by name in system/service 17400

  • Fix cloudwatch metricset missing tags collection. 17419 17424

  • check if cpuOptions field is nil in DescribeInstances output in ec2 metricset. 17418

  • Fix aws.s3.bucket.name terms_field in s3 overview dashboard. 17542

  • Fix Unix socket path in memcached. 17512

  • Fix vsphere VM dashboard host aggregation visualizations. 17555

  • Fix azure storage dashboards. 17590

  • Metricbeat no longer needs to be started strictly after Logstash for logstash-xpack module to report correct data. 17261 17497

  • Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. 17154 17600

  • Add privileged option so as mb to access data dir in Openshift. 17606

  • Fix "ID" event generator of Google Cloud module 17160 17608

  • Add privileged option for Auditbeat in Openshift 17637

  • Fix storage metricset to allow config without region/zone. 17623 17624

  • Fix overflow on Prometheus rates when new buckets are added on the go. 17753

  • Remove specific win32 api errors from events in perfmon. 18292 18361

  • Fix application_pool metricset after pdh changes. 18477

  • Fix tags_filter for cloudwatch metricset in aws. 18524

  • Fix panic on metricbeat test modules when modules are configured in metricbeat.modules. 18789 18797

  • Fix getting gcp compute instance metadata with partial zone/region in config. 18757

  • Add missing network.sent_packets_count metric into compute metricset in googlecloud module. 18802

  • Fix compute and pubsub dashboard for googlecloud module. 18962 18980

  • Fix crash on vsphere module when Host information is not available. 18996 19078

  • Fix incorrect usage of hints builder when exposed port is a substring of the hint 19052

  • Remove dedot for tag values in aws module. 19112 19221

  • Stop counterCache only when already started 19103

  • Fix empty field name errors in the application pool metricset. 19537

  • Set tags correctly if the dimension value is ARN 19111 19433

  • Fix bug incorrect parsing of float numbers as integers in Couchbase module 18949 19055

  • Fix mapping of service start type in the service metricset, windows module. 19551

  • Fix config example in the perfmon configuration files. 19539

  • Add missing info about the rest of the azure metricsets in the documentation. 19601

  • Fix k8s scheduler compatibility issue. 19699

  • Fix SQL module mapping NULL values as string 18955 elastic#18898[18898

  • Fix ec2 disk and network metrics to use Sum statistic method. 20680

  • Fill cloud.account.name with accountID if account alias doesn’t exist. 20736

  • The Kibana collector applies backoff when errored at getting usage stats 20772

  • Update fields.yml in the azure module, missing metrics field. 20918

  • The elasticsearch/index metricset only requests wildcard expansion for hidden indices if the monitored Elasticsearch cluster supports it. 20938

  • Disable Kafka metricsets based on Jolokia by default. They require a different configuration. 20989

  • Fix panic index out of range error when getting AWS account name. 21101 21095

  • Handle missing counters in the application_pool metricset. 21071

  • Fix remote_write flaky test. 21173

Packetbeat

  • Enable setting promiscuous mode automatically. 11366

  • Fix process monitoring when ipv6 is disabled under Linux. 19941 19945

  • Add "network" to event.category 20364 20392

Winlogbeat

  • Fix invalid IP addresses in DNS query results from Sysmon data. 18432 18436

  • Fields from Winlogbeat modules were not being included in index templates and patterns. 18983

Functionbeat

  • Fix timeout option of GCP functions. 16282 16287

  • Do not need Google credentials if not required for the operation. 17329 21072

  • Fix dependency issues of GCP functions. 20830 21070

  • Fix catchall bucket config errors by adding more validation. 16282 16287

Added

Affecting all Beats

  • Add configuration for APM instrumentation and expose the tracer trough the Beat object. 17938

  • Add document_id setting to decode_json_fields processor. 15859

  • Include network information by default on add_host_metadata and add_observer_metadata. 15347 16077

  • Add aws_ec2 provider for autodiscover. 12518 14823

  • Add monitoring variable libbeat.config.scans to distinguish scans of the configuration directory from actual reloads of its contents. 16440

  • Add support for multiple password in redis output. 16058 16206

  • Add support for Histogram type in fields.yml 16570

  • Windows .exe files now have embedded file version info. 15232t

  • Remove experimental flag from setup.template.append_fields 16576

  • Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. 16621

  • Add Kerberos support to Kafka input and output. 16781

  • Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. elastic#16621[16621

  • Add support for kubernetes provider to recognize namespace level defaults 16321

  • Add translate_sid processor on Windows for converting Windows security identifier (SID) values to names. 7451 16013

  • Add capability of enrich container.id with process id in add_process_metadata processor 15947

  • Update RPM packages contained in Beat Docker images. 17035

  • Update supported versions of redis output. 17198

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

  • Add replace processor for replacing string values of fields. 17342

  • Add optional regex based cid extractor to add_kubernetes_metadata processor. 17360

  • Add urldecode processor to for decoding URL-encoded fields. 17505

  • Add support for AWS IAM role_arn in credentials config. 17658 12464

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • Add Kerberos support to Elasticsearch output. 17927

  • Add k8s keystore backend. 18096

  • Add support for fixed length extraction in dissect processor. 17191

  • Set agent.name to the hostname by default. 16377 18000

  • Add support for basic ECS logging. 17974

  • Add config example of how to skip the add_host_metadata processor when forwarding logs. 13920 18153

  • When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

  • Add backoff configuration options for the Kafka output. 16777 17808

  • Add TLS support to Kerberos authentication in Elasticsearch. 18607

  • Change ownership of files in docker images so they can be used in secured environments. 12905

  • Upgrade k8s.io/client-go and k8s keystore tests. 18817

  • Add support for multiple sets of hints on autodiscover 18883

  • Add a configurable delay between retries when an app metadata cannot be retrieved by add_cloudfoundry_metadata. 19181

  • Add data type conversion in dissect processor for converting string values to other basic data types. 18683

  • Add the ignore_failure configuration option to the dissect processor. 19464

  • Add the overwrite_keys configuration option to the dissect processor. 19464

  • Add support to trim captured values in the dissect processor. 19464

  • Added the max_cached_sessions option to the script processor. 19562

  • Add support for DNS over TLS for the dns_processor. 19321

  • Add minimum cache TTL for successful DNS responses. 18986

  • Set index.max_docvalue_fields_search in index template to increase value to 200 fields. 20215

  • Add leader election for Kubernetes autodiscover. 20281

  • Add capability of enriching process metadata with contianer id also for non-privileged containers in add_process_metadata processor. 19767

  • Add replace_fields config option in add_host_metadata for replacing host fields. 20490 20464

  • Add container ECS fields in kubernetes metadata. 20984

  • Add ingress controller dashboards. 21052

  • Added experimental citrix module. 20820

  • Added experimental cyberark module. 20820

  • Added experimental proofpoint module. 20820

  • Added experimental snort module. 20820

  • Added experimental symantec module. 20820

  • Added experimental dataset barracuda/spamfirewall. 20820

  • Added experimental dataset cisco/meraki. 20820

  • Added experimental dataset f5/bigipafm. 20820

  • Added experimental dataset fortinet/fortimail. 20820

  • Added experimental dataset fortinet/fortimanager. 20820

  • Added experimental dataset juniper/netscreen. 20820

  • Added experimental dataset sophos/utm. 20820

Auditbeat

  • Reference kubernetes manifests include configuration for auditd and enrichment with kubernetes metadata. 17431

  • Reference kubernetes manifests mount data directory from the host, so data persist between executions in the same node. 17429

  • Log to stderr when running using reference kubernetes manifests. 174443

  • Fix syscall kprobe arguments for 32-bit systems in socket module. 17500

  • Fix memory leak on when we miss socket close kprobe events. 17500

  • Add system module process dataset ECS categorization fields. 18032

  • Add system module socket dataset ECS categorization fields. 18036

  • Add ECS categories for system module host dataset. 18031

  • Add system module package dataset ECS categorization fields. 18033

  • Add system module login dataset ECS categorization fields. 18034

  • Add system module user dataset ECS categorization fields. 18035

  • Add file integrity module ECS categorization fields. 18012

  • Add file.mime_type, file.extension, and file.drive_letter for file integrity module. 18012

  • Add ECS categorization info for auditd module 18596

  • Add enrichment of auditd seccomp events with name of the architecture, syscall, and signal. 14055 19300

Filebeat

  • Set event.outcome field based on googlecloud audit log output. 15731

  • Add dashboard for AWS ELB fileset. 15804

  • Add dashboard for AWS vpcflow fileset. 16007

  • Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb 15757 15936

  • Add custom string mapping to CEF module to support Forcepoint NGFW 14663 15910

  • Add ingress nginx controller fileset 16197

  • move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

  • Add ECS tls and categorization fields to apache module. 16032 16121

  • Work on e2e ACK’s for the azure-eventhub input 15671 16215

  • Add MQTT input. 15602 16204

  • Add ECS categorization fields to activemq module. 16151 16201

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Improve ECS categorization, container & process field mappings in auditd module. 16153 16280

  • Improve ECS field mappings in aws module. 16154 16307

  • Improve ECS categorization field mappings in googlecloud module. 16030 16500

  • Improve ECS field mappings in haproxy module. 16162 16529

  • Add cloudwatch fileset and ec2 fileset in aws module. 13716 16579

  • Improve ECS categorization field mappings in kibana module. 16168 16652

  • Improve the decode_cef processor by reducing the number of memory allocations. 16587

  • Add cloudfoundry input to send events from Cloud Foundry. 16586

  • Improve ECS categorization field mappings in iis module. 16165 16618

  • Improve ECS categorization field mapping in kafka module. 16167 16645

  • Allow users to override pipeline ID in fileset input config. 9531 16561

  • Add o365audit input type for consuming events from Office 365 Management Activity API. 16196 16244

  • Improve ECS categorization field mappings in logstash module. 16169 16668

  • Update filebeat httpjson input to support pagination via Header and Okta module. 16354

  • Improve ECS categorization field mapping in icinga module. 16164 16533

  • Improve ECS categorization field mappings in ibmmq module. 16163 16532

  • Improve ECS categorization, host field mappings in elasticsearch module. 16160 16469

  • Add ECS related fields to CEF module 16157 16338

  • Improve ECS categorization field mappings in suricata module. 16181 16843

  • Release ActiveMQ module as GA. 17047 17049

  • Improve ECS categorization field mappings in iptables module. 16166 16637

  • Add Filebeat Okta module. 16362

  • Add custom string mapping to CEF module to support Check Point devices. 16041 16907

  • Add pattern for Cisco ASA / FTD Message 734001 16212 16612

  • Added new module o365 for ingesting Office 365 management activity API events. 16196 16386

  • Add source field in k8s events 17209

  • Improve AWS cloudtrail field mappings 16086 16110 17155

  • Added new module crowdstrike for ingesting Crowdstrike Falcon streaming API endpoint event data. 16988

  • Added documentation for running Filebeat in Cloud Foundry. 17275

  • Move azure-eventhub input to GA. 15671 17313

  • Improve ECS categorization field mappings in mongodb module. 16170 17371

  • Improve ECS categorization field mappings for mssql module. 16171 17376

  • Added access_key_id, secret_access_key and session_token into aws module config. 17456

  • Add dashboard for Google Cloud Audit and AWS CloudTrail. 17379

  • Improve ECS categorization field mappings for mysql module. 16172 17491

  • Release Google Cloud module as GA. 17511

  • Add config option to select a different azure cloud env in the azure-eventhub input and azure module. 17649 17659

  • Added new Checkpoint Syslog filebeat module. 17682

  • Improve ECS categorization field mappings for nats module. 16173 17550

  • Add support for v10, v11 and v12 logs on Postgres 13810 17732

  • Enhance elasticsearch/server fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17714

  • Add support for Google Application Default Credentials to the Google Pub/Sub input and Google Cloud modules. 15668

  • Enhance elasticsearch/deprecation fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17728

  • Enhance elasticsearch/slowlog fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17729

  • Improve ECS categorization field mappings in misp module. 16026 17344

  • Added Unix stream socket support as an input source and a syslog input source. 17492

  • Added new Fortigate Syslog filebeat module. 17890

  • Improve ECS categorization field mappings in postgresql module. 16177 17914

  • Improve ECS categorization field mappings in rabbitmq module. 16178 17916

  • Make decode_cef processor GA. 17944

  • Improve ECS categorization field mappings in redis module. 16179 17918

  • Improve ECS categorization field mappings for zeek module. 16029 17738

  • Improve ECS categorization field mappings for netflow module. 16135 18108

  • Added an input option publisher_pipeline.disable_host to disable host.name from being added to events by default. 18159

  • Improve ECS categorization field mappings in system module. 16031 18065

  • Change the json.* input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Improve ECS categorization field mappings in osquery module. 16176 17881

  • Add http_endpoint input. 18298

  • Add support for array parsing in azure-eventhub input. 18585

  • Added observer.vendor, observer.product, and observer.type to PANW module events. 18223

  • The logstash module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. 9964 18095

  • Improve ECS categorization field mappings in envoyproxy module. 16161 18395

  • Improve ECS categorization field mappings in coredns module. 16159 18424

  • Improve ECS categorization field mappings in cisco module. 16028 18537

  • The s3 input can now automatically detect gzipped objects. 18283 18764

  • Add geoip AS lookup & improve ECS categorization in aws cloudtrail fileset. 18644 18958

  • Improved performance of PANW sample dashboards. 19031 19032

  • Add support for v1 consumer API in Cloud Foundry input, use it by default. 19125

  • Explicitly set ECS version in all Filebeat modules. 19198

  • Add new mode to multiline reader to aggregate constant number of lines 18352

  • Add automatic retries and exponential backoff to httpjson input. 18956

  • Add awscloudwatch input. 19025

  • Changed the panw module to pass through (rather than drop) message types other than threat and traffic. 16815 19375

  • Add support for timezone offsets and Z to decode_cef timestamp parser. 19346

  • Improve ECS categorization field mappings in traefik module. 16183 19379

  • Improve ECS categorization field mappings in azure module. 16155 19376

  • Add text & flattened versions of fields with unknown subfields in aws cloudtrail fileset. 18866 19121

  • Added Microsoft Defender ATP Module. 17997 19197

  • Add experimental dataset tomcat/log for Apache TomCat logs 19713

  • Add experimental dataset netscout/sightline for Netscout Arbor Sightline logs 19713

  • Add experimental dataset barracuda/waf for Barracuda Web Application Firewall logs 19713

  • Add experimental dataset f5/bigipapm for F5 Big-IP Access Policy Manager logs 19713

  • Add experimental dataset bluecoat/director for Bluecoat Director logs 19713

  • Add experimental dataset cisco/nexus for Cisco Nexus logs 19713

  • Add experimental dataset citrix/virtualapps for Citrix Virtual Apps logs 19713

  • Add experimental dataset cylance/protect for Cylance Protect logs 19713

  • Add experimental dataset fortinet/clientendpoint for Fortinet FortiClient Endpoint Protection logs 19713

  • Add experimental dataset imperva/securesphere for Imperva Secure Sphere logs 19713

  • Add experimental dataset infoblox/nios for Infoblox Network Identity Operating System logs 19713

  • Add experimental dataset juniper/junos for Juniper Junos OS logs 19713

  • Add experimental dataset kaspersky/av for Kaspersky Anti-Virus logs 19713

  • Add experimental dataset microsoft/dhcp for Microsoft DHCP Server logs 19713

  • Add experimental dataset tenable/nessus_security for Tenable Nessus Security Scanner logs 19713

  • Add experimental dataset rapid7/nexpose for Rapid7 Nexpose logs 19713

  • Add experimental dataset radware/defensepro for Radware DefensePro logs 19713

  • Add experimental dataset sonicwall/firewall for Sonicwall Firewalls logs 19713

  • Add experimental dataset squid/log for Squid Proxy Server logs 19713

  • Add experimental dataset zscaler/zia for Zscaler Internet Access logs 19713

  • Add initial support for configurable file identity tracking. 18748

  • Add support for reading auditd logs that are prefixed with node=. 19659

  • Add event.ingested for CrowdStrike module 20138

  • Add support for additional fields and FirewallMatchEvent type events in CrowdStrike module 20138

  • Add event.ingested for Suricata module 20220

  • Add support for custom header and headersecret for filebeat http_endpoint input 20435

  • Add event.ingested to all Filebeat modules. 20386

  • Return error when log harvester tries to open a named pipe. 18682 20450

  • Avoid goroutine leaks in Filebeat readers. 19193 20455

  • Convert httpjson to v2 input 20226

  • Improve Zeek x509 module with x509 ECS mappings 20867

  • Improve Zeek SSL module with x509 ECS mappings 20927

  • Improve Zeek Kerberos module with x509 ECS mappings 20958

  • Improve Fortinet firewall module with x509 ECS mappings 20983

  • Improve Santa module with x509 ECS mappings 20976

  • Improve Suricata Eve module with x509 ECS mappings 20973

  • Added new module for Zoom webhooks 20414

  • Add type and sub_type to panw panos fileset 20912

  • Always attempt community_id processor on zeek module 21155

Heartbeat

  • Allow a list of status codes for HTTP checks. 15587

  • Add additional ECS compatible fields for TLS information. 17687

  • Record HTTP response headers. 18327

  • Add index and pipeline settings to monitor configurations. 20610

Journalbeat

  • Added an id config option to inputs to allow running multiple inputs on the same journal. 18467

  • Add basic ECS categorization and log.syslog fields. 19176

Metricbeat

  • Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

  • Add lambda metricset in aws module. 15260

  • Expand data for the system/memory metricset 15492

  • Add azure storage metricset in order to retrieve metric values for storage accounts. 14548 15342

  • Add cost warnings for the azure module. 15356

  • Add DynamoDB AWS Metricbeat light module 15097

  • Release elb module as GA. 15485

  • Add a system/network_summary metricset 15196

  • Add mesh metricset for Istio Metricbeat module 15535

  • Add mixer metricset for Istio Metricbeat module 15696

  • Add pilot metricset for Istio Metricbeat module 15761

  • Make the system/cpu metricset collect normalized CPU metrics by default. 15618 15729

  • Add galley metricset for Istio Metricbeat module 15857

  • Add key/value mode for SQL module. 15770 {pull]15845[15845]

  • Add STAN dashboard 15654

  • Add support for Unix socket in Memcached metricbeat module. 13685 15822

  • Add up metric to prometheus metrics collected from host 15948

  • Add citadel metricset for Istio Metricbeat module 15990

  • Add support for processors in light modules. 14740 15923

  • Add collecting AuroraDB metrics in rds metricset. 14142 16004

  • Reuse connections in SQL module. 16001

  • Improve the logstash module (when xpack.enabled is set to true) to use the override cluster_uuid returned by Logstash APIs. 15772 15795

  • Add region parameter in googlecloud module. 15780 16203

  • Add kubernetes storage class support via kube-state-metrics. 16145

  • Add database_account azure metricset. 15758

  • Add support for NATS 2.1. 16317

  • Add Load Balancing metricset to GCP 15559

  • Add support for Dropwizard metrics 4.1. 16332

  • Add azure container metricset in order to monitor containers. 15751 16421

  • Improve the haproxy module to support metrics exposed via HTTPS. 14579 16333

  • Add filtering option for prometheus collector. 16420

  • Add metricsets based on Ceph Manager Daemon to the ceph module. 7723 16254

  • Release statsd module as GA. 16447 14280

  • Add collecting tags and tags_filter for rds metricset in aws module. 16605 16358

  • Add OpenMetrics Metricbeat module 16596

  • Add cloudfoundry module to send events from Cloud Foundry. 16671

  • Add redisenterprise module. 16482 15269

  • Add system/users metricset as beta 16569

  • Align fields to ECS and add more tests for the azure module. 16024 16754

  • Add additional cgroup fields to docker/diskiohttps://github.com/elastic/pull/16638[16638]

  • Add PubSub metricset to Google Cloud Platform module 15536

  • Add overview dashboard for googlecloud compute metricset. 16534 16819

  • Add Prometheus remote write endpoint 16609

  • Release STAN module as GA. 16980

  • Add query metricset for prometheus module. 17104

  • Release ActiveMQ module as GA. 17047 17049

  • Release Zookeeper/connection module as GA. 14281 17043

  • Add support for CouchDB v2 16352 16455

  • Add dashboard for pubsub metricset in googlecloud module. 17161

  • Add dashboards for the azure container metricsets. 17194

  • Replace vpc metricset into vpn, transitgateway and natgateway metricsets. 16892

  • Use Elasticsearch histogram type to store Prometheus histograms 17061

  • Allow to rate Prometheus counters when scraping them 17061

  • Release Oracle module as GA. 14279 16833

  • Release vsphere module as GA. 15798 17119

  • Add Storage metricsets to GCP module 15598

  • Added documentation for running Metricbeat in Cloud Foundry. 17275

  • Add test for documented fields check for metricsets without a http input. 17315 17334

  • Add final tests and move label to GA for the azure module in metricbeat. 17319

  • Refactor windows/perfmon metricset configuration options and event output. 17596

  • Reference kubernetes manifests mount data directory from the host when running metricbeat as daemonset, so data persist between executions in the same node. 17429

  • Add state_statefulset metricset to Metricbeat recommended configuration for k8s. 17627

  • Add more detailed error messages, system tests and small refactoring to the service metricset in windows. 17725

  • Stack Monitoring modules now auto-configure required metricsets when xpack.enabled: true is set. [16471 17609

  • Add Metricbeat IIS module dashboards. 17966

  • Add dashboard for the azure database account metricset. 17901

  • Allow partial region and zone name in googlecloud module config. 17913

  • Add aggregation aligner as a config parameter for googlecloud stackdriver metricset. [17141 17719

  • Move the perfmon metricset to GA. 16608 17879

  • Add static mapping for metricsets under aws module. 17614 17650

  • Add dashboard for googlecloud storage metricset. 18172

  • Collect new bulk indexing metrics from Elasticsearch when xpack.enabled:true is set. https://github.com/elastic/beats/issues/ 17992

  • Remove requirement to connect as sysdba in Oracle module 15846 18182

  • Update MSSQL module to fix some SSPI authentication and add brackets to USE statements 17862]

  • Add client address to events from http server module 18336

  • Remove required for region/zone and make stackdriver a metricset in googlecloud. 16785 18398

  • Add memory metrics into compute googlecloud. 18802

  • Add new fields to HAProxy module. 18523

  • Add Tomcat overview dashboard 14026

  • Accept prefix as metric_types config parameter in googlecloud stackdriver metricset. 19345

  • Update Couchbase to version 6.5 18595 19055

  • Add dashboards for googlecloud load balancing metricset. 18369

  • Add support for v1 consumer API in Cloud Foundry module, use it by default. 19268

  • Add support for named ports in autodiscover. 19398

  • Add param aws_partition to support aws-cn, aws-us-gov regions. 18850 19423

  • Add support for wildcard * in dimension value of AWS CloudWatch metrics config. 18050 19660

  • The elasticsearch/index metricset now collects metrics for hidden indices as well. 18639 18703

  • Added performance and query metricsets to mysql module. 18955

  • The elasticsearch-xpack/index metricset now reports hidden indices as such. 18639 18706

  • Adds support for app insights metrics in the azure module. 18570 18940

  • Infer types in Prometheus remote_write. 19944

  • Added cache and connection_errors metrics to status metricset of MySQL module 16955 19844

  • Update MySQL dashboard with connection errors and cache metrics 19913 16955

  • Add cloud.instance.name into aws ec2 metricset. 20077

  • Add host inventory metrics into aws ec2 metricset. 20171

  • Add scope setting for elasticsearch module, allowing it to monitor an Elasticsearch cluster behind a load-balancing proxy. 18539 18547

  • Add state_daemonset metricset for Kubernetes Metricbeat module 20649

  • Add host inventory metrics to azure compute_vm metricset. 20641

  • Add host inventory metrics to googlecloud compute metricset. 20391

  • Add billing data collection from Cost Explorer into aws billing metricset. 20527 20103

  • Migrate compute_vm metricset to a light one, map cloud.instance.id field. 20889

  • Request prometheus endpoints to be gzipped by default 20766

  • Release all kubernetes state metricsets as GA 20901

  • Add billing metricset into googlecloud module. 20812 20738

  • Move compute_vm_scaleset to light metricset. 21038 20985

  • Sanitize event.host. 21022

  • Add overview and platform health dashboards to Cloud Foundry module. 21124

Packetbeat

  • Add an example to packetbeat.yml of using the forwarded tag to disable host metadata fields when processing network data from network tap or mirror port. 19209

  • Add ECS fields for x509 certs, event categorization, and related IP info. 19167

  • Add 100-continue support 15830 19349

Functionbeat - Add basic ECS categorization and cloud fields. 19174 - Add support for parallelization factor for kinesis. 20727

Winlogbeat

  • Add more DNS error codes to the Sysmon module. 15685

  • Add experimental event log reader implementation that should be faster in most cases. 6585 16849

  • Set process.command_line and process.parent.command_line from Sysmon Event ID 1. 17327

  • Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module 17517

  • Add registry and code signature information and ECS categorization fields for sysmon module 18058

  • Add new winlogbeat security dashboard 18775

  • Add event.outcome to events based on the audit success and audit failure keywords. 20564

Elastic Log Driver - Add support for docker logs command 19531 - Add support to change beat name, and support for Kibana Logs. 20522

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Journalbeat

Metricbeat

  • Deprecate tags config parameter in cloudwatch metricset. 16733

  • Deprecate tags.resource_type_filter config parameter and replace with resource_type. 19688

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat