Skip to content

Commit

Permalink
Update go-modules with master (#16274)
Browse files Browse the repository at this point in the history
* [Filebeat] move create-[module,fileset,fields] to mage (#15836)

- move create-[module,fileset,fields] to mage
- make mage create commands available in x-pack/filebeat
- change Makefile to use mage for create commands

* Elasticsearch index must be lowercase (#16081)

* Index names must be lowercase

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

* update kafka/redis/LS output to guarantee lowercase index
* add godoc

* Regenerate expected files after changes in date parsing (#16139)

Elasticsearch has modified the behaviour on date parsing when the date
doesn't include timezone data. Regenerate a couple of golden files that
are affected by this change.

* Add autodiscover for aws_ec2 (#14823)

* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

* Fix a connection error in httpjson input (#16123)

* Fix a connection error in httpjson input

* Include document_id in decode_json_fields allowed fields (#16156)

* ci: run test on Windows (#15570)

* feat: run test on Windows

* chore: parameter to enable/disable windows test

* deleteDir before of the checkout

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Apply suggestions from code review

* feat: apply dependency hierarchies

* fit: use isChanged for all matches, and add the libbeat match where it is needed

* feat: add x-pack/winlogbeat windows unit tests

* fix: duplicate when condition

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <[email protected]>

Co-authored-by: Andrew Kroh <[email protected]>

* improve kubernetes.pod.cpu.usage.limit.pct field description (#16128)

* upgrade github.com/gogo/protobuf/... to v1.3.1 (#16138)

* [Filebeat] Add ECS tls & categorization fields to apache module (#16121)

* Add ECS tls & categorization fields to apache module

- tls.cipher (access)
- tls.protocol (access)
- tls.protocol_version (access)
- event.kind (access)
- event.category (access)
- event.outcome (access)
- lowercase http.request.method for ECS compliance (access)
- event.kind (error)
- event.category (error)
- event.type (error)

Closes #16032

* [Metricbeat] Add Overview dashboard to Tomcat module

* [Metricbeat] Fix PostgreSQL Dashboard (#16132)

* [Metricbeat] Fix PostgreSQL Dashboard

* Update version

* Fix: imports order (#16207)

* [Metricbeat]kube-state-metrics: add storage class support (#16145)

* add ksm storage class support

* [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (#16116)

* Improve parsing of syslog.pid in journalbeat to strip the username in pid when present.

* Add entry to changelog with pull ID.

* Improve the comment on the username strip.

* [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019)

* Add a sha256 pin for the CA Certificate

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validates the chains exposer by the server.
This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a lets a certificate in the chains match the provided
sha256.

Usage example for the Elasticsearch output.

```
output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha1>
```

You can generate the pin using the **openssl** binary with the
following command:

```
openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
```

OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html)

You will need to start Elasticsearch with the following options

```yaml
xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"
```

This pull request also include a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

* [docs] Add 7.6 breaking changes and release highlights (#16202)

* [docs] Add early draft of Elastic Log Driver docs (#15799)

* Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (#16124) (#16225)

Minor update to be more explicit on the index template loading requirement.

Co-authored-by: romain-chanu <[email protected]>

* Remove spaces in prometheus commented out option (#16233)

* Fix: don't miss address scheme (#16205)

* Fix: don't miss address scheme

* Add unit test

* Adjust source after code review

* Add comment to method

* Freeze virtualenv version until issue with CI is resolved (#16235)

* [docs] Fix install command to match instructions on docker hub (#16249)

* [docs] Add link to observability release blog (#16246)

* ci(jenkins): enable fix-permissions to be executed without running make too (#16130)

* ci(jenkins): enable fix-permissions to be executed without running make too

* ci(jenkins): go modules are stored in the HOME path

* ci(jenkins): fix permissions should run only if docker is enabled

* Upgrade go-ucfg to version 0.8.2 (#16199)

* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

* Add ingress nginx controller fileset (#16197)

* update notice

Co-authored-by: Lee Hinman <[email protected]>
Co-authored-by: Steffen Siering <[email protected]>
Co-authored-by: Jaime Soriano Pastor <[email protected]>
Co-authored-by: kaiyan-sheng <[email protected]>
Co-authored-by: Lei Qiu <[email protected]>
Co-authored-by: Fae Charlton <[email protected]>
Co-authored-by: Ivan Fernandez Calvo <[email protected]>
Co-authored-by: Andrew Kroh <[email protected]>
Co-authored-by: Chris Mark <[email protected]>
Co-authored-by: Gil Raphaelli <[email protected]>
Co-authored-by: Mario Castro <[email protected]>
Co-authored-by: Dimitri Mazmanov <[email protected]>
Co-authored-by: Marcin Tojek <[email protected]>
Co-authored-by: Pablo Mercado <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: Pier-Hugues Pellerin <[email protected]>
Co-authored-by: DeDe Morton <[email protected]>
Co-authored-by: romain-chanu <[email protected]>
Co-authored-by: Michal Pristas <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
  • Loading branch information
21 people authored Feb 12, 2020
1 parent cb07176 commit 31551b1
Show file tree
Hide file tree
Showing 154 changed files with 7,036 additions and 874 deletions.
16 changes: 16 additions & 0 deletions .ci/scripts/install-tools.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
set GOPATH=%WORKSPACE%
set MAGEFILE_CACHE=%WORKSPACE%\.magefile
set PATH=%WORKSPACE%\bin;C:\ProgramData\chocolatey\bin;%PATH%

where /q curl
IF ERRORLEVEL 1 (
choco install curl -y --no-progress --skipdownloadcache
)
mkdir %WORKSPACE%\bin
where /q gvm
IF ERRORLEVEL 1 (
curl -sL -o %WORKSPACE%\bin\gvm.exe https://github.com/andrewkroh/gvm/releases/download/v0.2.1/gvm-windows-amd64.exe
)
FOR /f "tokens=*" %%i IN ('"gvm.exe" use %GO_VERSION% --format=batch') DO %%i

go install github.com/elastic/beats/vendor/github.com/magefile/mage
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ before_install:
- curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
- chmod +x docker-compose
- sudo mv docker-compose /usr/local/bin
- if [ $TRAVIS_OS_NAME = osx ]; then pip install virtualenv; fi
- if [ $TRAVIS_OS_NAME = osx ]; then pip install virtualenv==16.7.9; fi


# Skips installations step
Expand Down
15 changes: 13 additions & 2 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d

*Journalbeat*

- Improve parsing of syslog.pid in journalbeat to strip the username when present {pull}16116[16116]


*Metricbeat*

- Make use of secure port when accessing Kubelet API {pull}16063[16063]
- Add Tomcat overview dashboard {pull}14026[14026]

*Packetbeat*

Expand All @@ -51,7 +54,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Update replicaset group to apps/v1 {pull}15854[15802]
- Fix issue where default go logger is not discarded when either * or stdout is selected. {issue}10251[10251] {pull}15708[15708]
- Upgrade go-ucfg to latest v0.8.1. {pull}15937{15937}
- Remove superfluous use of number_of_routing_shards setting from the default template. {pull}16038[16038]
- Fix index names for indexing not always guaranteed to be lower case. {pull}16081[16081]
- Add `ssl.ca_sha256` option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. {issue}15717[15717]

*Auditbeat*

Expand All @@ -66,6 +70,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Improve `elasticsearch/audit` fileset to handle timestamps correctly. {pull}15942[15942]
- Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the `elasticsearch` module. {issue}15840[15840] {pull}15900[15900]
- Fix mapping error for cloudtrail additionalEventData field {pull}16088[16088]
- Fix a connection error in httpjson input. {pull}16123[16123]

*Heartbeat*

Expand All @@ -82,6 +87,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Change lookup_fields from metricset.host to service.address {pull}15883[15883]
- Add dedot for cloudwatch metric name. {issue}15916[15916] {pull}15917[15917]
- Fixed issue `logstash-xpack` module suddenly ceasing to monitor Logstash. {issue}15974[15974] {pull}16044[16044]
- Fix skipping protocol scheme by light modules. {pull}16205[pull]

*Packetbeat*

Expand All @@ -98,17 +104,21 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
*Affecting all Beats*

- Add document_id setting to decode_json_fields processor. {pull}15859[15859]

- Add `aws_ec2` provider for autodiscover. {issue}12518[12518] {pull}14823[14823]

*Auditbeat*


*Filebeat*

- Set event.outcome field based on googlecloud audit log output. {pull}15731[15731]
- Add dashboard for AWS ELB fileset. {pull}15804[15804]
- Add dashboard for AWS vpcflow fileset. {pull}16007[16007]
- Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb {issue}15757[15757] {pull}15935[15936]
- Add custom string mapping to CEF module to support Forcepoint NGFW {issue}14663[14663] {pull}15910[15910]
- Add ingress nginx controller fileset {pull}16197[16197]
- move create-[module,fileset,fields] to mage and enable in x-pack/filebeat {pull}15836[15836]
- Add ECS tls and categorization fields to apache module. {issue}16032[16032] {pull}16121[16121]

*Heartbeat*

Expand Down Expand Up @@ -141,6 +151,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add collecting AuroraDB metrics in rds metricset. {issue}14142[14142] {pull}16004[16004]
- Reuse connections in SQL module. {pull}16001[16001]
- Improve the `logstash` module (when `xpack.enabled` is set to `true`) to use the override `cluster_uuid` returned by Logstash APIs. {issue}15772[15772] {pull}15795[15795]
- Add kubernetes storage class support via kube-state-metrics. {pull}16145[16145]

*Packetbeat*

Expand Down
Loading

0 comments on commit 31551b1

Please sign in to comment.