Skip to content

Latest commit

 

History

History
435 lines (366 loc) · 30.6 KB

CHANGELOG.next.asciidoc

File metadata and controls

435 lines (366 loc) · 30.6 KB
Raw

Beats version HEAD

Breaking changes

Affecting all Beats

Auditbeat

  • File integrity dataset (macOS): Replace unnecessary file.origin.raw (type keyword) with file.origin.text (type text). 12423 15630

Filebeat - Improve ECS field mappings in panw module. event.outcome now only contains success/failure per ECS specification. 16025 17910 - Improve ECS categorization field mappings for nginx module. http.request.referrer is now lowercase & http.request.referrer only populated when nginx sets a value 16174 17844 - Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . 16180 17982

Heartbeat

Journalbeat

  • Improve parsing of syslog.pid in journalbeat to strip the username when present 16116

Metricbeat

  • Make use of secure port when accessing Kubelet API 16063

  • Add Tomcat overview dashboard 14026

Packetbeat

  • Redis: fix incorrectly handle with two-words redis command. 14872 14873

Winlogbeat

  • Add support to Sysmon file delete events (event ID 23). 18094

Functionbeat

Bugfixes

Affecting all Beats

  • Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data 17223

  • Fix add_cloud_metadata to better support modifying sub-fields with other processors. 13808

  • TLS or Beats that accept connections over TLS and validate client certificates. 14146

  • Fix panics that could result from invalid TLS certificates. This can affect Beats that connect over TLS, or Beats that accept connections over TLS and validate client certificates. 14146

  • Fix panic in the Logstash output when trying to send events to closed connection. 15568

  • Fix missing output in dockerlogbeat 15719

  • Fix logging target settings being ignored when Beats are started via systemd or docker. 12024 15442

  • Do not load dashboards where not available. 15802

  • Fix issue where TLS settings would be ignored when a forward proxy was in use. https://github.com/elastic/beats/pull/15516{15516}

  • Update replicaset group to apps/v1 15802

  • Fix issue where default go logger is not discarded when either * or stdout is selected. 10251 15708

  • Upgrade go-ucfg to latest v0.8.1. https://github.com/elastic/beats/pull/15937{15937}

  • Fix index names for indexing not always guaranteed to be lower case. 16081

  • Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

  • Fix loading processors from annotation hints. 16348

  • Fix an issue that could cause redundant configuration reloads. 16440

  • Fix k8s pods labels broken schema. 16480

  • Fix k8s pods annotations broken schema. 16554

  • Upgrade go-ucfg to latest v0.8.3. https://github.com/elastic/beats/pull/16450{16450}

  • Fix NewContainerMetadataEnricher to use default config for kubernetes module. 16857

  • Improve some logging messages for add_kubernetes_metadata processor 16866

  • Fix k8s metadata issue regarding node labels not shown up on root level of metadata. 16834

  • Fail to start if httpprof is used and it cannot be initialized. 17028

  • Fix concurrency issues in convert processor when used in the global context. 17032

  • Fix bug with monitoring.cluster_uuid setting not always being exposed via GET /state Beats API. 16732 17420

  • Fix building on FreeBSD by removing build flags from add_cloudfoundry_metadata processor. 17486

  • Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. 17613

  • Fix goroutine leak and Elasticsearch output file descriptor leak when output reloading is in use. 10491 17381

  • Fix setup.dashboards.index setting not working. 17749

  • Fix Elasticsearch license endpoint URL referenced in error message. 17880 18030

  • Fix panic when assigning a key to a nil value in an event. 18143

  • Gives monitoring reporter hosts, if configured, total precedence over corresponding output hosts. 17937 17991

  • Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

Auditbeat

  • system/socket: Fixed compatibility issue with kernel 5.x. 15771

  • system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

  • system module: Fix panic during initialisation when /proc/stat can’t be read. 17569

Filebeat

  • Ensure all zeek timestamps include millisecond precision. 14599 16766

  • Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. 15502 15590

  • Add shared_credential_file to cloudtrail config 15652 15656

  • Fix typos in zeek notice fileset config file. 15764 15765

  • Fix mapping error when zeek weird logs do not contain IP addresses. 15906

  • Improve elasticsearch/audit fileset to handle timestamps correctly. 15942

  • Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the elasticsearch module. 15840 15900

  • Fix mapping error for cloudtrail additionalEventData field 16088

  • Fix a connection error in httpjson input. 16123

  • Fix s3 input with cloudtrail fileset reading json file. 16374 16441

  • Rewrite azure filebeat dashboards, due to changes in kibana. 16466

  • Adding the var definitions in azure manifest files, fix for errors when executing command setup. 16270 16468

  • Fix merging of fileset inputs to replace paths and append processors. https://github.com/elastic/beats/pull/16450{16450}

  • Add queue_url definition in manifest file for aws module. https://github.com/elastic/beats/pull/16640{16640}

  • Fix issue where autodiscover hints default configuration was not being copied. 16987

  • Fix Elasticsearch _id field set by S3 and Google Pub/Sub inputs. 17026

  • Fixed various Cisco FTD parsing issues. 16863 16889

  • Fix default index pattern in IBM MQ filebeat dashboard. 17146

  • Fix elasticsearch.gc fileset to not collect all logs when Elasticsearch is running in Docker. 13164 16583 17164

  • Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. 17216 17220

  • CEF: Fixed decoding errors caused by trailing spaces in messages. 17253

  • Fixed a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. 17242 17243

  • Fixed MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17086 17156

  • Fix elasticsearch.audit data ingest pipeline to be more forgiving with date formats found in Elasticsearch audit logs. 17406

  • Fixed activemq module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17428

  • Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. 17425

  • Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. 17734 17735

  • Fixed cloudfoundry.access to have the correct cloudfoundry.app.id contents. 17847

  • Fixing ingress_controller. fields to be of type keyword instead of text. 17834

  • Fixed typo in log message. 17897

  • Fix Cisco ASA ASA 3020** and 106023 messages 17964

Heartbeat

  • Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. 15639

  • Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. 17549

Journalbeat

Metricbeat

  • Add dedot for tags in ec2 metricset and cloudwatch metricset. 15843 15844

  • Use RFC3339 format for timestamps collected using the SQL module. 15847

  • Avoid parsing errors returned from prometheus endpoints. 15712

  • Change lookup_fields from metricset.host to service.address 15883

  • Add dedot for cloudwatch metric name. 15916 15917

  • Fixed issue logstash-xpack module suddenly ceasing to monitor Logstash. 15974 16044

  • Fix skipping protocol scheme by light modules. pull

  • Made logstash-xpack module once again have parity with internally-collected Logstash monitoring data. 16198

  • Change sqs metricset to use average as statistic method. 16438

  • Revert changes in docker module: add size flag to docker.container. 16600

  • Fix diskio issue for windows 32 bit on disk_performance struct alignment. 16680

  • Fix detection and logging of some error cases with light modules. 14706

  • Fix imports after PR was merged before rebase. 16756

  • Add dashboard for redisenterprise module. 16752

  • Dynamically choose a method for the system/service metricset to support older linux distros. 16902

  • Use max in k8s apiserver dashboard aggregations. 17018

  • Reduce memory usage in elasticsearch/index metricset. 16503 16538

  • Check if CCR feature is available on Elasticsearch cluster before attempting to call CCR APIs from elasticsearch/ccr metricset. 16511 17073

  • Use max in k8s overview dashboard aggregations. 17015

  • Fix Disk Used and Disk Usage visualizations in the Metricbeat System dashboards. 12435 17272

  • Fix missing Accept header for Prometheus and OpenMetrics module. 16870 17291

  • Further revise check for bad data in docker/memory. 17400

  • Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

  • Combine cloudwatch aggregated metrics into single event. 17345

  • Fix how we filter services by name in system/service 17400

  • Fix cloudwatch metricset missing tags collection. 17419 17424

  • check if cpuOptions field is nil in DescribeInstances output in ec2 metricset. 17418

  • Fix aws.s3.bucket.name terms_field in s3 overview dashboard. 17542

  • Fix Unix socket path in memcached. 17512

  • Fix vsphere VM dashboard host aggregation visualizations. 17555

  • Fix azure storage dashboards. 17590

  • Metricbeat no longer needs to be started strictly after Logstash for logstash-xpack module to report correct data. 17261 17497

  • Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. 17154 17600

  • Add privileged option so as mb to access data dir in Openshift. 17606

  • Fix "ID" event generator of Google Cloud module 17160 17608

  • Add privileged option for Auditbeat in Openshift 17637

  • Fix storage metricset to allow config without region/zone. 17623 17624

  • Add a switch to the driver definition on SQL module to use pretty names. 17378

  • Fix overflow on Prometheus rates when new buckets are added on the go. 17753

Packetbeat

  • Enable setting promiscuous mode automatically. 11366

Winlogbeat

Functionbeat

  • Fix timeout option of GCP functions. 16282 16287

Added

Affecting all Beats

  • Add document_id setting to decode_json_fields processor. 15859

  • Include network information by default on add_host_metadata and add_observer_metadata. 15347 16077

  • Add aws_ec2 provider for autodiscover. 12518 14823

  • Add monitoring variable libbeat.config.scans to distinguish scans of the configuration directory from actual reloads of its contents. 16440

  • Add support for multiple password in redis output. 16058 16206

  • Add support for Histogram type in fields.yml 16570

  • Windows .exe files now have embedded file version info. 15232t

  • Remove experimental flag from setup.template.append_fields 16576

  • Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. 16621

  • Add Kerberos support to Kafka input and output. 16781

  • Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. elastic#16621[16621

  • Add support for kubernetes provider to recognize namespace level defaults 16321

  • Add translate_sid processor on Windows for converting Windows security identifier (SID) values to names. 7451 16013

  • Add capability of enrich container.id with process id in add_process_metadata processor 15947

  • Update RPM packages contained in Beat Docker images. 17035

  • Update supported versions of redis output. 17198

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

  • Add replace processor for replacing string values of fields. 17342

  • Add optional regex based cid extractor to add_kubernetes_metadata processor. 17360

  • Add urldecode processor to for decoding URL-encoded fields. 17505

  • Add support for AWS IAM role_arn in credentials config. 17658 12464

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • Add Kerberos support to Elasticsearch output. 17927

  • Add support for fixed length extraction in dissect processor. 17191

  • Set agent.name to the hostname by default. 16377 18000

  • Add support for basic ECS logging. 17974

  • Add config example of how to skip the add_host_metadata processor when forwarding logs. 13920 18153

  • When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

  • Add backoff configuration options for the Kafka output. 16777 17808

Auditbeat

  • Reference kubernetes manifests include configuration for auditd and enrichment with kubernetes metadata. 17431

  • Reference kubernetes manifests mount data directory from the host, so data persist between executions in the same node. 17429

  • Log to stderr when running using reference kubernetes manifests. 174443

  • Fix syscall kprobe arguments for 32-bit systems in socket module. 17500

  • Fix memory leak on when we miss socket close kprobe events. 17500

  • Add system module process dataset ECS categorization fields. 18032

  • Add system module socket dataset ECS categorization fields. 18036

  • Add ECS categories for system module host dataset. 18031

  • Add system module package dataset ECS categorization fields. 18033

  • Add system module login dataset ECS categorization fields. 18034

  • Add system module user dataset ECS categorization fields. 18035

  • Add file integrity module ECS categorization fields. 18012

  • Add file.mime_type, file.extension, and file.drive_letter for file integrity module. 18012

Filebeat

  • Set event.outcome field based on googlecloud audit log output. 15731

  • Add dashboard for AWS ELB fileset. 15804

  • Add dashboard for AWS vpcflow fileset. 16007

  • Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb 15757 15936

  • Add custom string mapping to CEF module to support Forcepoint NGFW 14663 15910

  • Add ingress nginx controller fileset 16197

  • move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

  • Add ECS tls and categorization fields to apache module. 16032 16121

  • Work on e2e ACK’s for the azure-eventhub input 15671 16215

  • Add MQTT input. 15602 16204

  • Add ECS categorization fields to activemq module. 16151 16201

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Improve ECS categorization, container & process field mappings in auditd module. 16153 16280

  • Improve ECS field mappings in aws module. 16154 16307

  • Improve ECS categorization field mappings in googlecloud module. 16030 16500

  • Improve ECS field mappings in haproxy module. 16162 16529

  • Add cloudwatch fileset and ec2 fileset in aws module. 13716 16579

  • Improve ECS categorization field mappings in kibana module. 16168 16652

  • Improve the decode_cef processor by reducing the number of memory allocations. 16587

  • Add cloudfoundry input to send events from Cloud Foundry. 16586

  • Improve ECS categorization field mappings in iis module. 16165 16618

  • Improve ECS categorization field mapping in kafka module. 16167 16645

  • Allow users to override pipeline ID in fileset input config. 9531 16561

  • Add o365audit input type for consuming events from Office 365 Management Activity API. 16196 16244

  • Improve ECS categorization field mappings in logstash module. 16169 16668

  • Update filebeat httpjson input to support pagination via Header and Okta module. 16354

  • Improve ECS categorization field mapping in icinga module. 16164 16533

  • Improve ECS categorization field mappings in ibmmq module. 16163 16532

  • Improve ECS categorization, host field mappings in elasticsearch module. 16160 16469

  • Add ECS related fields to CEF module 16157 16338

  • Improve ECS categorization field mappings in suricata module. 16181 16843

  • Release ActiveMQ module as GA. 17047 17049

  • Improve ECS categorization field mappings in iptables module. 16166 16637

  • Add Filebeat Okta module. 16362

  • Add custom string mapping to CEF module to support Check Point devices. 16041 16907

  • Add pattern for Cisco ASA / FTD Message 734001 16212 16612

  • Added new module o365 for ingesting Office 365 management activity API events. 16196 16386

  • Add source field in k8s events 17209

  • Improve AWS cloudtrail field mappings 16086 16110 17155

  • Added new module crowdstrike for ingesting Crowdstrike Falcon streaming API endpoint event data. 16988

  • Added documentation for running Filebeat in Cloud Foundry. 17275

  • Move azure-eventhub input to GA. 15671 17313

  • Improve ECS categorization field mappings in mongodb module. 16170 17371

  • Improve ECS categorization field mappings for mssql module. 16171 17376

  • Added access_key_id, secret_access_key and session_token into aws module config. 17456

  • Add dashboard for Google Cloud Audit and AWS CloudTrail. 17379

  • Improve ECS categorization field mappings for mysql module. 16172 17491

  • Release Google Cloud module as GA. 17511

  • Add config option to select a different azure cloud env in the azure-eventhub input and azure module. 17649 17659

  • Added new Checkpoint Syslog filebeat module. 17682

  • Improve ECS categorization field mappings for nats module. 16173 17550

  • Add support for v10, v11 and v12 logs on Postgres 13810 17732

  • Enhance elasticsearch/server fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17714

  • Add support for Google Application Default Credentials to the Google Pub/Sub input and Google Cloud modules. 15668

  • Enhance elasticsearch/deprecation fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17728

  • Enhance elasticsearch/slowlog fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17729

  • Improve ECS categorization field mappings in misp module. 16026 17344

  • Added Unix stream socket support as an input source and a syslog input source. 17492

  • Added new Fortigate Syslog filebeat module. 17890

  • Improve ECS categorization field mappings in postgresql module. 16177 17914

  • Improve ECS categorization field mappings in rabbitmq module. 16178 17916

  • Make decode_cef processor GA. 17944

  • Improve ECS categorization field mappings in redis module. 16179 17918

  • Improve ECS categorization field mappings for zeek module. 16029 17738

  • Improve ECS categorization field mappings for netflow module. 16135 18108

  • Added an input option publisher_pipeline.disable_host to disable host.name from being added to events by default. 18159

  • Improve ECS categorization field mappings in system module. 16031 18065

  • Change the json.* input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Improve ECS categorization field mappings in osquery module. 16176 17881

  • Add new mode to multiline reader to aggregate constant number of lines 18352

Heartbeat

  • Allow a list of status codes for HTTP checks. 15587

  • Add additional ECS compatible fields for TLS information. 17687

Journalbeat

Metricbeat

  • Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

  • Add lambda metricset in aws module. 15260

  • Expand data for the system/memory metricset 15492

  • Add azure storage metricset in order to retrieve metric values for storage accounts. 14548 15342

  • Add cost warnings for the azure module. 15356

  • Add DynamoDB AWS Metricbeat light module 15097

  • Release elb module as GA. 15485

  • Add a system/network_summary metricset 15196

  • Add mesh metricset for Istio Metricbeat module 15535

  • Add mixer metricset for Istio Metricbeat module 15696

  • Add pilot metricset for Istio Metricbeat module 15761

  • Make the system/cpu metricset collect normalized CPU metrics by default. 15618 15729

  • Add galley metricset for Istio Metricbeat module 15857

  • Add key/value mode for SQL module. 15770 {pull]15845[15845]

  • Add STAN dashboard 15654

  • Add support for Unix socket in Memcached metricbeat module. 13685 15822

  • Add up metric to prometheus metrics collected from host 15948

  • Add citadel metricset for Istio Metricbeat module 15990

  • Add support for processors in light modules. 14740 15923

  • Add collecting AuroraDB metrics in rds metricset. 14142 16004

  • Reuse connections in SQL module. 16001

  • Improve the logstash module (when xpack.enabled is set to true) to use the override cluster_uuid returned by Logstash APIs. 15772 15795

  • Add region parameter in googlecloud module. 15780 16203

  • Add kubernetes storage class support via kube-state-metrics. 16145

  • Add database_account azure metricset. 15758

  • Add support for NATS 2.1. 16317

  • Add Load Balancing metricset to GCP 15559

  • Add support for Dropwizard metrics 4.1. 16332

  • Add azure container metricset in order to monitor containers. 15751 16421

  • Improve the haproxy module to support metrics exposed via HTTPS. 14579 16333

  • Add filtering option for prometheus collector. 16420

  • Add metricsets based on Ceph Manager Daemon to the ceph module. 7723 16254

  • Release statsd module as GA. 16447 14280

  • Add collecting tags and tags_filter for rds metricset in aws module. 16605 16358

  • Add OpenMetrics Metricbeat module 16596

  • Add cloudfoundry module to send events from Cloud Foundry. 16671

  • Add redisenterprise module. 16482 15269

  • Add system/users metricset as beta 16569

  • Align fields to ECS and add more tests for the azure module. 16024 16754

  • Add additional cgroup fields to docker/diskiohttps://github.com/elastic/pull/16638[16638]

  • Add PubSub metricset to Google Cloud Platform module 15536

  • Add overview dashboard for googlecloud compute metricset. 16534 16819

  • Add Prometheus remote write endpoint 16609

  • Release STAN module as GA. 16980

  • Add query metricset for prometheus module. 17104

  • Release ActiveMQ module as GA. 17047 17049

  • Release Zookeeper/connection module as GA. 14281 17043

  • Add support for CouchDB v2 16352 16455

  • Add dashboard for pubsub metricset in googlecloud module. 17161

  • Add dashboards for the azure container metricsets. 17194

  • Replace vpc metricset into vpn, transitgateway and natgateway metricsets. 16892

  • Use Elasticsearch histogram type to store Prometheus histograms 17061

  • Allow to rate Prometheus counters when scraping them 17061

  • Release Oracle module as GA. 14279 16833

  • Release vsphere module as GA. 15798 17119

  • Add Storage metricsets to GCP module 15598

  • Added documentation for running Metricbeat in Cloud Foundry. 17275

  • Add test for documented fields check for metricsets without a http input. 17315 17334

  • Add final tests and move label to GA for the azure module in metricbeat. 17319

  • Refactor windows/perfmon metricset configuration options and event output. 17596

  • Reference kubernetes manifests mount data directory from the host when running metricbeat as daemonset, so data persist between executions in the same node. 17429

  • Add more detailed error messages, system tests and small refactoring to the service metricset in windows. 17725

  • Stack Monitoring modules now auto-configure required metricsets when xpack.enabled: true is set. [16471 17609

  • Add Metricbeat IIS module dashboards. 17966

  • Add dashboard for the azure database account metricset. 17901

  • Allow partial region and zone name in googlecloud module config. 17913

  • Add aggregation aligner as a config parameter for googlecloud stackdriver metricset. [17141 17719

  • Move the perfmon metricset to GA. 16608 17879

  • Add static mapping for metricsets under aws module. 17614 17650

  • Add dashboard for googlecloud storage metricset. 18172

  • Collect new bulk indexing metrics from Elasticsearch when xpack.enabled:true is set. https://github.com/elastic/beats/issues/ 17992

  • Remove requirement to connect as sysdba in Oracle module 15846 18182

  • Update MSSQL module to fix some SSPI authentication and add brackets to USE statements 17862]

  • Add client address to events from http server module 18336

Packetbeat

Functionbeat

Winlogbeat

  • Add more DNS error codes to the Sysmon module. 15685

  • Add experimental event log reader implementation that should be faster in most cases. 6585 16849

  • Set process.command_line and process.parent.command_line from Sysmon Event ID 1. 17327

  • Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module 17517

  • Add registry and code signature information and ECS categorization fields for sysmon module 18058

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat