Skip to content

Latest commit

 

History

History
400 lines (327 loc) · 26 KB

CHANGELOG.next.asciidoc

File metadata and controls

400 lines (327 loc) · 26 KB

Beats version HEAD

Breaking changes

Affecting all Beats

  • Update to Golang 1.12.1. 11330

  • Disable Alibaba Cloud and Tencent Cloud metadata providers by default. 12812

  • Allow Metricbeat’s beat module to read monitoring information over a named pipe or unix domain socket. 14558

  • Remove version information from default ILM policy for improved upgrade experience on custom policies. 14745

  • Running setup cmd respects setup.ilm.overwrite setting for improved support of custom policies. 14741

  • Libbeat: Do not overwrite agent.*, ecs.version, and host.name. 14407

  • Libbeat: Cleanup the x-pack licenser code to use the new license endpoint and the new format. 15091

  • Users can now specify monitoring.cloud. to override monitoring.elasticsearch. settings. 14399 15254

  • Refactor metadata generator to support adding metadata across resources 14875

  • Update to ECS 1.4.0. 14844

  • The document id fields has been renamed from @metadata.id to @metadata._id 15859

  • Variable substitution from environment variables is not longer supported. https://github.com/elastic/beats/pull/15937{15937}

  • Change aws_elb autodiscover provider field name from elb_listener.* to aws.elb.*. 16219 https://github.com/elastic/beats/pull/16402{16402}

  • Remove AddDockerMetadata and AddKubernetesMetadata processors from the script processor. They can still be used as normal processors in the configuration. 16349 16514

Auditbeat

Filebeat

  • Fix parsing of Elasticsearch node name by elasticsearch/slowlog fileset. 14547

  • CEF extensions are now mapped to the data types defined in the CEF guide. 14342

Heartbeat

Journalbeat

  • Remove broken dashboard. 15288

  • Improve parsing of syslog.pid in journalbeat to strip the username when present 16116

Metricbeat

  • Add new dashboard for VSphere host cluster and virtual machine 14135

  • kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. 11975

  • Update cloudwatch metricset mapping for both metrics and dimensions. 15245

  • Make use of secure port when accessing Kubelet API 16063

Packetbeat

  • TLS: Fields have been changed to adapt to ECS. 15497

  • TLS: The behavior of send_certificates and include_raw_certificates options has changed. 15497

  • Added redact_headers configuration option, to allow HTTP request headers to be redacted whilst keeping the header field included in the beat. 15353

  • Add dns.question.subdomain and dns.question.top_level_domain fields. 14578

Winlogbeat

Functionbeat

Bugfixes

Affecting all Beats

  • Fix a race condition with the Kafka pipeline client, it is possible that Close() get called before Connect() . 11945

  • Allow users to configure only cluster_uuid setting under monitoring namespace. 14338

  • Fix spooling to disk blocking infinitely if the lock file can not be acquired. 15338

  • Update replicaset group to apps/v1 15802

  • Fix metricbeat test output with an ipv6 ES host in the output.hosts. 15368

  • Fix convert processor conversion of string to integer with leading zeros. 15513 15557

  • Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data 17223

  • Fix add_cloud_metadata to better support modifying sub-fields with other processors. 13808

  • Fix panic in the Logstash output when trying to send events to closed connection. 15568

  • Fix missing output in dockerlogbeat 15719

  • Fix logging target settings being ignored when Beats are started via systemd or docker. 12024 15442

  • Do not load dashboards where not available. 15802

  • Fix issue where default go logger is not discarded when either * or stdout is selected. 10251 15708

  • Fix issue where TLS settings would be ignored when a forward proxy was in use. https://github.com/elastic/beats/pull/15516{15516}

  • Remove superfluous use of number_of_routing_shards setting from the default template. 16038

  • Fix index names for indexing not always guaranteed to be lower case. 16081

  • Upgrade go-ucfg to latest v0.8.1. https://github.com/elastic/beats/pull/15937{15937}

  • Fix loading processors from annotation hints. 16348

  • Fix an issue that could cause redundant configuration reloads. 16440

  • Fix k8s pods labels broken schema. 16480

  • Fix k8s pods annotations broken schema. 16554

  • Upgrade go-ucfg to latest v0.8.3. https://github.com/elastic/beats/pull/16450{16450}

  • Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

  • Fix NewContainerMetadataEnricher to use default config for kubernetes module. 16857

  • Improve some logging messages for add_kubernetes_metadata processor https://github.com/elastic/beats/pull/16866{16866}

  • Fix k8s metadata issue regarding node labels not shown up on root level of metadata. 16834

  • Fail to start if httpprof is used and it cannot be initialized. 17028

  • Fix concurrency issues in convert processor when used in the global context. 17032

Auditbeat

  • system/socket: Fixed compatibility issue with kernel 5.x. 15771

  • system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

Filebeat

  • cisco/asa fileset: Fix parsing of 302021 message code. 14519

  • Fix filebeat azure dashboards, event category should be Alert. 14668

  • Fix a problem in Filebeat input httpjson where interval is not used as time.Duration. 14728

  • Fix SSL config in input.yml for Filebeat httpjson input in the MISP module. 14767

  • Check content-type when creating new reader in s3 input. 15252 15225

  • Fix session reset detection and a crash in Netflow input. 14904

  • Handle errors in handleS3Objects function and add more debug messages for s3 input. 15545

  • netflow: Allow for options templates without scope fields. 15449

  • netflow: Fix bytes/packets counters on some devices (NSEL and Netstream). 15449

  • netflow: Fix compatibility with some Cisco devices by changing the field class_id from short to long. 15449

  • Fixed dashboard for Cisco ASA Firewall. 15420 15553

  • Ensure all zeek timestamps include millisecond precision. 14599 16766

  • Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. 15502 15590

  • Add shared_credential_file to cloudtrail config 15652 15656

  • Fix typos in zeek notice fileset config file. 15764 15765

  • Fix mapping error when zeek weird logs do not contain IP addresses. 15906

  • Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the elasticsearch module. 15840 15900

  • Fix mapping error for cloudtrail additionalEventData field 16088

  • Fix a connection error in httpjson input. 16123

  • Improve elasticsearch/audit fileset to handle timestamps correctly. 15942

  • Fix s3 input with cloudtrail fileset reading json file. 16374 16441

  • Rewrite azure filebeat dashboards, due to changes in kibana. 16466

  • Adding the var definitions in azure manifest files, fix for errors when executing command setup. 16270 16468

  • Fix merging of fileset inputs to replace paths and append processors. https://github.com/elastic/beats/pull/16450{16450}

  • Add queue_url definition in manifest file for aws module. https://github.com/elastic/beats/pull/16640{16640}

  • Fix issue where autodiscover hints default configuration was not being copied. 16987

  • Fix Elasticsearch _id field set by S3 and Google Pub/Sub inputs. 17026

  • Fixed various Cisco FTD parsing issues. 16863 16889

  • Fix default index pattern in IBM MQ filebeat dashboard. 17146

  • Fix elasticsearch.gc fileset to not collect all logs when Elasticsearch is running in Docker. 13164 16583 17164

  • Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. 17216 17220

  • Fixed a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. 17242 17243

  • Fixed MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17086 17156

Heartbeat

  • Fix recording of SSL cert metadata for Expired/Unvalidated x509 certs. 13687

  • Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. 15639

  • Fixed scheduler shutdown issues which would in rare situations cause a panic due to semaphore misuse. 16397

Journalbeat

Metricbeat

  • Fix checking tagsFilter using length in cloudwatch metricset. 14525

  • Fixed bug with elasticsearch/cluster_stats metricset not recording license expiration date correctly. 14541 14591

  • Log bulk failures from bulk API requests to monitoring cluster. 14303 14356

  • Fix regular expression to detect instance name in perfmon metricset. 14273 14666

  • Fixed bug with elasticsearch/cluster_stats metricset not recording license ID in the correct field. 14592

  • Fix docker.container.size fields values 14979 15224

  • Make kibana module more resilient to Kibana unavailability. 15258 15270

  • Fix panic exception with some unicode strings in perfmon metricset. 15264

  • Make logstash module more resilient to Logstash unavailability. 15276 15306

  • Add username/password in Metricbeat autodiscover hints 15349

  • Fix CPU count in docker/cpu in cases where no online_cpus are reported 15070

  • Add dedot for tags in ec2 metricset and cloudwatch metricset. 15843 15844

  • Use RFC3339 format for timestamps collected using the SQL module. 15847

  • Change lookup_fields from metricset.host to service.address 15883

  • Avoid parsing errors returned from prometheus endpoints. 15712

  • Add dedot for cloudwatch metric name. 15916 15917

  • Fixed issue logstash-xpack module suddenly ceasing to monitor Logstash. 15974 16044

  • Fix skipping protocol scheme by light modules. pull

  • Made logstash-xpack module once again have parity with internally-collected Logstash monitoring data. 16198

  • Change sqs metricset to use average as statistic method. 16438

  • Revert changes in docker module: add size flag to docker.container. 16600

  • Fix diskio issue for windows 32 bit on disk_performance struct alignment. 16680

  • Fix detection and logging of some error cases with light modules. 14706

  • Add dashboard for redisenterprise module. 16752

  • Convert increments of 100 nanoseconds/ticks to milliseconds for WriteTime and ReadTime in diskio metricset (Windows) for consistency. 14233

  • Dynamically choose a method for the system/service metricset to support older linux distros. 16902

  • Use max in k8s apiserver dashboard aggregations. 17018

  • Reduce memory usage in elasticsearch/index metricset. 16503 16538

  • Check if CCR feature is available on Elasticsearch cluster before attempting to call CCR APIs from elasticsearch/ccr metricset. 16511 17073

  • Use max in k8s overview dashboard aggregations. 17015

  • Fix Disk Used and Disk Usage visualizations in the Metricbeat System dashboards. 12435 17272

  • Fix missing Accept header for Prometheus and OpenMetrics module. 16870 17291

Packetbeat

  • Enable setting promiscuous mode automatically. 11366

Winlogbeat

Functionbeat

  • Fix timeout option of GCP functions. 16282 16287

Added

Affecting all Beats

  • Add a friendly log message when a request to docker has exceeded the deadline. 15336

  • Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors 12451

  • Allow a beat to ship monitoring data directly to an Elasticsearch monitoring cluster. 9260

  • Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. 11394

  • add_host_metadata is no GA. 13148

  • Add providers setting to add_cloud_metadata processor. 13812

  • GA the script processor. 14325

  • Add fingerprint processor. 11173 14205

  • Add support for API keys in Elasticsearch outputs. 14324

  • Ensure that init containers are no longer tailed after they stop 14394

  • Add consumer_lag in Kafka consumergroup metricset 14822

  • Make use of consumer_lag in Kafka dashboard 14863

  • Refactor kubernetes autodiscover to enable different resource based discovery 14738

  • Add add_id processor. 14524

  • Enable TLS 1.3 in all beats. 12973

  • Spooling to disk creates a lockfile on each platform. 15338

  • Fingerprint processor adds a new xxhash hashing algorithm 15418

  • Enable DEP (Data Execution Protection) for Windows packages. 15149

  • Add document_id setting to decode_json_fields processor. 15859

  • Include network information by default on add_host_metadata and add_observer_metadata. 15347 16077

  • Add aws_ec2 provider for autodiscover. 12518 14823

  • Add monitoring variable libbeat.config.scans to distinguish scans of the configuration directory from actual reloads of its contents. 16440

  • Add support for multiple password in redis output. 16058 16206

  • Add support for Histogram type in fields.yml 16570

  • Windows .exe files now have embedded file version info. 15232t

  • Remove experimental flag from setup.template.append_fields 16576

  • Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. 16621

  • Add translate_sid processor on Windows for converting Windows security identifier (SID) values to names. 7451 16013

  • Add support for kubernetes provider to recognize namespace level defaults 16321

  • Add capability of enrich container.id with process id in add_process_metadata processor 15947

  • Update RPM packages contained in Beat Docker images. 17035

  • Add Kerberos support to Kafka input and output. 16781

  • Update supported versions of redis output. 17198

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

Auditbeat

Filebeat

  • Add dashboard for AWS ELB fileset. 15804

  • container and docker inputs now support reading of labels and env vars written by docker JSON file logging driver. 8358

  • Add index option to all inputs to directly set a per-input index value. 14010

  • Add new fileset googlecloud/audit for ingesting Google Cloud Audit logs. 15200

  • Add dashboards to the CEF module (ported from the Logstash ArcSight module). 14342

  • Add expand_event_list_from_field support in s3 input for reading json format AWS logs. 15357 15370

  • Add azure-eventhub input which will use the azure eventhub go sdk. 14092 14882

  • Expose more metrics of harvesters (e.g. read_offset, start_time). 13395

  • Include log.source.address for unparseable syslog messages. 13268 15453

  • Release aws elb fileset as GA. 15426 15380

  • Integrate the azure-eventhub with filebeat azure module (replace the kafka input). 15480

  • Release aws s3access fileset to GA. 15431 15430

  • Add cloudtrail fileset to AWS module. 14657 15227

  • New fileset googlecloud/firewall for ingesting Google Cloud Firewall logs. 14553

  • google-pubsub input: ACK pub/sub message when acknowledged by publisher. 13346 14715

  • Remove Beta label from google-pubsub input. 13346 14715

  • Set event.outcome field based on googlecloud audit log output. 15731

  • Add dashboard for AWS vpcflow fileset. 16007

  • Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb 15757 15936

  • Add ingress nginx controller fileset 16197

  • move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

  • Add ECS tls and categorization fields to apache module. 16032 16121

  • Work on e2e ACK’s for the azure-eventhub input 15671 16215

  • Add MQTT input. 15602 16204

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Improve ECS categorization, container & process field mappings in auditd module. 16153 16280

  • Add ECS categorization fields to activemq module. 16151 16201

  • Improve ECS field mappings in aws module. 16154 16307

  • Improve ECS categorization field mappings in googlecloud module. 16030 16500

  • Add cloudwatch fileset and ec2 fileset in aws module. 13716 16579

  • Improve ECS categorization field mappings in kibana module. 16168 16652

  • Add cloudfoundry input to send events from Cloud Foundry. 16586

  • Improve ECS field mappings in haproxy module. 16162 16529

  • Allow users to override pipeline ID in fileset input config. 9531 16561

  • Improve ECS categorization field mappings in logstash module. 16169 16668

  • Improve ECS categorization field mappings in iis module. 16165 16618

  • Improve the decode_cef processor by reducing the number of memory allocations. 16587

  • Improve ECS categorization field mapping in kafka module. 16167 16645

  • Improve ECS categorization field mapping in icinga module. 16164 16533

  • Improve ECS categorization field mappings in ibmmq module. 16163 16532

  • Add custom string mapping to CEF module to support Forcepoint NGFW 14663 15910

  • Add ECS related fields to CEF module 16157 16338

  • Improve ECS categorization, host field mappings in elasticsearch module. 16160 16469

  • Improve ECS categorization field mappings in suricata module. 16181 16843

  • Release ActiveMQ module as GA. 17047 17049

  • Improve ECS categorization field mappings in iptables module. 16166 16637

  • Add pattern for Cisco ASA / FTD Message 734001 16212 16612

  • Add o365audit input type for consuming events from Office 365 Management Activity API. 16196 16244

  • Add custom string mapping to CEF module to support Check Point devices. 16041 16907

  • Added new module o365 for ingesting Office 365 management activity API events. 16196 16386

  • Add Filebeat Okta module. 16362

  • Add source field in k8s events 17209

  • Improve AWS cloudtrail field mappings 16086 16110 17155

  • Move azure-eventhub input to GA. 15671 17313

Heartbeat

  • Allow a list of status codes for HTTP checks. 15587

Heartbeat

Journalbeat

Metricbeat

  • Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

  • Add lambda metricset in aws module. 15260

  • Expand data for the system/memory metricset 15492

  • Add azure storage metricset in order to retrieve metric values for storage accounts. 14548 15342

  • Add cost warnings for the azure module. 15356

  • Add DynamoDB AWS Metricbeat light module 15097

  • Release elb module as GA. 15485

  • Add a system/network_summary metricset 15196

  • Add IBM MQ light-weight Metricbeat module 15301

  • Enable script processor. 14711

  • Add mixer metricset for Istio Metricbeat module 15696

  • Add mesh metricset for Istio Metricbeat modulehttps://github.com/elastic/pull/15535[15535]

  • Add pilot metricset for Istio Metricbeat module 15761

  • Add galley metricset for Istio Metricbeat module 15857

  • Add STAN dashboard 15654

  • Add key/value mode for SQL module. 15770 {pull]15845[15845]

  • Add support for Unix socket in Memcached metricbeat module. 13685 15822

  • Make the system/cpu metricset collect normalized CPU metrics by default. 15618 15729

  • Add kubernetes storage class support via kube-state-metrics. 16145

  • Add up metric to prometheus metrics collected from host 15948

  • Add citadel metricset for Istio Metricbeat module 15990

  • Add support for processors in light modules. 14740 15923

  • Add collecting AuroraDB metrics in rds metricset. 14142 16004

  • Reuse connections in SQL module. 16001

  • Improve the logstash module (when xpack.enabled is set to true) to use the override cluster_uuid returned by Logstash APIs. 15772 15795

  • Add region parameter in googlecloud module. 15780 16203

  • Add database_account azure metricset. 15758

  • Add support for Dropwizard metrics 4.1. 16332

  • Add support for NATS 2.1. 16317

  • Add azure container metricset in order to monitor containers. 15751 16421

  • Improve the haproxy module to support metrics exposed via HTTPS. 14579 16333

  • Add filtering option for prometheus collector. 16420

  • Add metricsets based on Ceph Manager Daemon to the ceph module. 7723 16254

  • Add Load Balancing metricset to GCP 15559

  • Release statsd module as GA. 16447 14280

  • Add collecting tags and tags_filter for rds metricset in aws module. 16605 16358

  • Add OpenMetrics Metricbeat module 16596

  • Add redisenterprise module. 16482 15269

  • Add cloudfoundry module to send events from Cloud Foundry. 16671

  • Add system/users metricset as beta 16569

  • Align fields to ECS and add more tests for the azure module. 16024 16754

  • Add additional cgroup fields to docker/diskiohttps://github.com/elastic/pull/16638[16638]

  • Add overview dashboard for googlecloud compute metricset. 16534 16819

  • Add Prometheus remote write endpoint 16609

  • Release STAN module as GA. 16980

  • Add query metricset for prometheus module. 17104

  • Release ActiveMQ module as GA. 17047 17049

  • Add support for CouchDB v2 16352 16455

  • Release Zookeeper/connection module as GA. 14281 17043

  • Add dashboards for the azure container metricsets. 17194

  • Replace vpc metricset into vpn, transitgateway and natgateway metricsets. 16892

  • Use Elasticsearch histogram type to store Prometheus histograms 17061

  • Allow to rate Prometheus counters when scraping them 17061

  • Release Oracle module as GA. 14279 16833

  • Add Storage metricsets to GCP module 15598

  • Release vsphere module as GA. 15798 17119

  • Add PubSub metricset to Google Cloud Platform module 15536

  • Add final tests and move label to GA for the azure module in metricbeat. 17319

Packetbeat

Functionbeat

  • Add monitoring info about triggered functions. 14876

  • Add Google Cloud Platform support. 13598

Winlogbeat

  • Add more DNS error codes to the Sysmon module. 15685

  • Add Audit and Log Management, Computer Object Management, and Distribution Group related events to the Security module. 15217

  • Add experimental event log reader implementation that should be faster in most cases. 6585 16849

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat