[Filebeat] Add Zeek NTP Fileset #24224

legoguy1000 · 2021-02-25T02:26:39Z

What does this PR do?

Add the NTP fileset to the Zeek Module

Why is it important?

One of the remaining Zeek log files that isn't parsed already by Filebeat.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2021-02-25T02:31:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: andrewkroh commented: jenkins, run tests
Start Time: 2021-03-17T21:24:56.356+0000
Duration: 47 min 4 sec
Commit: abed2c7

Test stats 🧪

Test	Results
Failed	0
Passed	13161
Skipped	2243
Total	15404

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13161
Skipped	2243
Total	15404

legoguy1000 · 2021-02-25T15:35:15Z

I'm trying to generate the expected log files but getting a weird timeout. Have anyone seen this before??

>> python test: Integration Testing
Found Elastic Beats dir at /go/src/github.com/elastic/beats
exec: /go/src/github.com/elastic/beats/build/ve/docker/bin/pytest --timeout=90 --durations=20 -v --junit-xml=build/TEST-python-integration.xml tests/system/test_filebeat_xpack.py tests/system/test_http_endpoint.py tests/system/test_xpack_modules.py
======================================================================== test session starts =========================================================================
platform linux -- Python 3.7.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /go/src/github.com/elastic/beats/build/ve/docker/bin/python3
cachedir: .pytest_cache
rootdir: /go/src/github.com/elastic/beats, configfile: pytest.ini
plugins: rerunfailures-9.0, timeout-1.3.4
timeout: 90.0s
timeout method: signal
timeout func_only: True
collected 15 items / 14 deselected / 1 selected                                                                                                                      

tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_zeek FAILED                                                                                 [100%]

============================================================================== FAILURES ==============================================================================
_________________________________________________________________ XPackTest.test_fileset_file_0_zeek _________________________________________________________________

a = (<test_xpack_modules.XPackTest testMethod=test_fileset_file_0_zeek>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../filebeat/tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
../../filebeat/tests/system/test_modules.py:150: in run_on_file
    bufsize=0).wait()
/usr/lib/python3.7/subprocess.py:990: in wait
    return self._wait(timeout=timeout)
/usr/lib/python3.7/subprocess.py:1624: in _wait
    (pid, sts) = self._try_wait(0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x7f27237e7dd8>, wait_flags = 0

    def _try_wait(self, wait_flags):
        """All callers to this function MUST hold self._waitpid_lock."""
        try:
>           (pid, sts) = os.waitpid(self.pid, wait_flags)
E           Failed: Timeout >90.0s

/usr/lib/python3.7/subprocess.py:1582: Failed
------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------
Using elasticsearch: http://elasticsearch:9200
Testing zeek/ntp on /go/src/github.com/elastic/beats/x-pack/filebeat/module/zeek/ntp/test/ntp-json.log
------------------------------- generated xml file: /go/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-python-integration.xml -------------------------------
======================================================================== slowest 20 durations ========================================================================
90.00s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_zeek

(2 durations < 0.005s hidden.  Use -vv to show these durations.)
====================================================================== short test summary info =======================================================================
FAILED tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_zeek - Failed: Timeout >90.0s
============================================================ 1 failed, 14 deselected in 117.97s (0:01:57) ============================================================
>> python test: Integration Testing Complete
>>> Fixing file ownership issues from Docker at path=.
chown took: 34.8968867s, changed 0 files
>>> Fixing file ownership issues from Docker at path=/go/src/github.com/elastic/beats/build
chown took: 2m19.3409614s, changed 0 files
Error: running "/go/src/github.com/elastic/beats/build/ve/docker/bin/pytest --timeout=90 --durations=20 -v --junit-xml=build/TEST-python-integration.xml tests/system/test_filebeat_xpack.py tests/system/test_http_endpoint.py tests/system/test_xpack_modules.py" failed with exit code 1
ERROR: 1
exec: docker-compose -p filebeat_8_0_0_5356f8245a-snapshot rm --stop --force
Stopping filebeat_8_0_0_5356f8245a-snapshot_elasticsearch_1 ... done
Removing filebeat_8_0_0_5356f8245a-snapshot_beat_run_11032c46723d ... done
Removing filebeat_8_0_0_5356f8245a-snapshot_proxy_dep_1           ... done
Removing filebeat_8_0_0_5356f8245a-snapshot_elasticsearch_1       ... done
>> Done running testing inside of docker...
Teardown mage...
Error: running "docker-compose -p filebeat_8_0_0_5356f8245a-snapshot run -e DOCKER_COMPOSE_PROJECT_NAME=filebeat_8_0_0_5356f8245a-snapshot -e BEAT_STRICT_PERMS=false -e STACK_ENVIRONMENT=snapshot -e TESTING_ENVIRONMENT=snapshot -e GOCACHE=/go/src/github.com/elastic/beats/build/docker-gocache -v /home/alex/go/pkg/mod/cache/download:/gocache:ro -e GOPROXY=file:///gocache,direct -e EXEC_UID=1000 -e EXEC_GID=1000 -e GENERATE=true -e MAGEFILE_VERBOSE=1 -e BEATS_INSIDE_INTEGRATION_TEST_ENV=true -e GOFLAGS=-mod=readonly -e TESTING_FILEBEAT_FILESETS=ntp -e PYTEST_ADDOPTS=-k zeek beat /go/src/github.com/elastic/beats/x-pack/filebeat/build/mage-linux-amd64 pythonIntegTest" failed with exit code 1

legoguy1000 · 2021-02-25T16:03:06Z

I'm trying to generate the expected log files but getting a weird timeout. Have anyone seen this before??

>> python test: Integration Testing
Found Elastic Beats dir at /go/src/github.com/elastic/beats
exec: /go/src/github.com/elastic/beats/build/ve/docker/bin/pytest --timeout=90 --durations=20 -v --junit-xml=build/TEST-python-integration.xml tests/system/test_filebeat_xpack.py tests/system/test_http_endpoint.py tests/system/test_xpack_modules.py
======================================================================== test session starts =========================================================================
platform linux -- Python 3.7.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /go/src/github.com/elastic/beats/build/ve/docker/bin/python3
cachedir: .pytest_cache
rootdir: /go/src/github.com/elastic/beats, configfile: pytest.ini
plugins: rerunfailures-9.0, timeout-1.3.4
timeout: 90.0s
timeout method: signal
timeout func_only: True
collected 15 items / 14 deselected / 1 selected                                                                                                                      

tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_zeek FAILED                                                                                 [100%]

============================================================================== FAILURES ==============================================================================
_________________________________________________________________ XPackTest.test_fileset_file_0_zeek _________________________________________________________________

a = (<test_xpack_modules.XPackTest testMethod=test_fileset_file_0_zeek>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../filebeat/tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
../../filebeat/tests/system/test_modules.py:150: in run_on_file
    bufsize=0).wait()
/usr/lib/python3.7/subprocess.py:990: in wait
    return self._wait(timeout=timeout)
/usr/lib/python3.7/subprocess.py:1624: in _wait
    (pid, sts) = self._try_wait(0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x7f27237e7dd8>, wait_flags = 0

    def _try_wait(self, wait_flags):
        """All callers to this function MUST hold self._waitpid_lock."""
        try:
>           (pid, sts) = os.waitpid(self.pid, wait_flags)
E           Failed: Timeout >90.0s

/usr/lib/python3.7/subprocess.py:1582: Failed
------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------
Using elasticsearch: http://elasticsearch:9200
Testing zeek/ntp on /go/src/github.com/elastic/beats/x-pack/filebeat/module/zeek/ntp/test/ntp-json.log
------------------------------- generated xml file: /go/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-python-integration.xml -------------------------------
======================================================================== slowest 20 durations ========================================================================
90.00s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_zeek

(2 durations < 0.005s hidden.  Use -vv to show these durations.)
====================================================================== short test summary info =======================================================================
FAILED tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_zeek - Failed: Timeout >90.0s
============================================================ 1 failed, 14 deselected in 117.97s (0:01:57) ============================================================
>> python test: Integration Testing Complete
>>> Fixing file ownership issues from Docker at path=.
chown took: 34.8968867s, changed 0 files
>>> Fixing file ownership issues from Docker at path=/go/src/github.com/elastic/beats/build
chown took: 2m19.3409614s, changed 0 files
Error: running "/go/src/github.com/elastic/beats/build/ve/docker/bin/pytest --timeout=90 --durations=20 -v --junit-xml=build/TEST-python-integration.xml tests/system/test_filebeat_xpack.py tests/system/test_http_endpoint.py tests/system/test_xpack_modules.py" failed with exit code 1
ERROR: 1
exec: docker-compose -p filebeat_8_0_0_5356f8245a-snapshot rm --stop --force
Stopping filebeat_8_0_0_5356f8245a-snapshot_elasticsearch_1 ... done
Removing filebeat_8_0_0_5356f8245a-snapshot_beat_run_11032c46723d ... done
Removing filebeat_8_0_0_5356f8245a-snapshot_proxy_dep_1           ... done
Removing filebeat_8_0_0_5356f8245a-snapshot_elasticsearch_1       ... done
>> Done running testing inside of docker...
Teardown mage...
Error: running "docker-compose -p filebeat_8_0_0_5356f8245a-snapshot run -e DOCKER_COMPOSE_PROJECT_NAME=filebeat_8_0_0_5356f8245a-snapshot -e BEAT_STRICT_PERMS=false -e STACK_ENVIRONMENT=snapshot -e TESTING_ENVIRONMENT=snapshot -e GOCACHE=/go/src/github.com/elastic/beats/build/docker-gocache -v /home/alex/go/pkg/mod/cache/download:/gocache:ro -e GOPROXY=file:///gocache,direct -e EXEC_UID=1000 -e EXEC_GID=1000 -e GENERATE=true -e MAGEFILE_VERBOSE=1 -e BEATS_INSIDE_INTEGRATION_TEST_ENV=true -e GOFLAGS=-mod=readonly -e TESTING_FILEBEAT_FILESETS=ntp -e PYTEST_ADDOPTS=-k zeek beat /go/src/github.com/elastic/beats/x-pack/filebeat/build/mage-linux-amd64 pythonIntegTest" failed with exit code 1

I'm running GENERATE=true PYTEST_ADDOPTS="-k zeek" TESTING_FILEBEAT_FILESETS=ntp mage -v pythonIntegTest from beats/x-pack/filebeat

elasticmachine · 2021-02-25T16:03:46Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

leehinman · 2021-03-01T16:28:14Z

Thanks this looks awesome.

The easiest way to generate the expected golden files is to let the mage target handle it. In the x-pack/filebeat directory run:

GENERATE=true TESTING_FILEBEAT_MODULES=zeek TESTING_FILEBEAT_FILESETS=ntp mage -v pythonIntegTest

That will generate the golden files, limiting it to just the zeek module and the ntp fileset.

legoguy1000 · 2021-03-01T17:18:50Z

Thanks this looks awesome.

The easiest way to generate the expected golden files is to let the mage target handle it. In the x-pack/filebeat directory run:
GENERATE=true TESTING_FILEBEAT_MODULES=zeek TESTING_FILEBEAT_FILESETS=ntp mage -v pythonIntegTest
That will generate the golden files, limiting it to just the zeek module and the ntp fileset.

Tried this too, still getting a timeout error. I think its just me trying to run this on a WSL 2 VM. THough didn't have this issue with my previous PR. Going to try on a dedicated system.

legoguy1000 · 2021-03-02T00:47:17Z

Its still timing out as shown in the post above. I can't seem to figure out what's causing it.

leehinman · 2021-03-03T19:04:32Z

Its still timing out as shown in the post above. I can't seem to figure out what's causing it.

turns out in manifest.yml you have:

ingest_pipeline: ingest/pipeline.json

it should be ingest/pipeline.yml after fixing that the mage target worked for me, although it did find a field that wasn't documented.

legoguy1000 · 2021-03-03T20:10:30Z

Its still timing out as shown in the post above. I can't seem to figure out what's causing it.

turns out in manifest.yml you have:
ingest_pipeline: ingest/pipeline.json
it should be ingest/pipeline.yml after fixing that the mage target worked for me, although it did find a field that wasn't documented.

🤦‍♂️ 🤦‍♂️ I have updated the code and it should be ready to run the pipelines

legoguy1000 · 2021-03-04T17:33:39Z

@leehinman @andrewkroh can 1 of u run the jenkins pipeline?

andrewkroh · 2021-03-17T13:29:39Z

jenkins, run tests

andrewkroh

LGTM, and tests are passing. I just have minor comments to improve the docs.

x-pack/filebeat/module/zeek/ntp/_meta/fields.yml

CHANGELOG.next.asciidoc

andrewkroh

LGTM. I updated the docs.

andrewkroh · 2021-03-17T21:24:32Z

jenkins, run tests

Add the NTP fileset to the Zeek Module. Co-authored-by: Andrew Kroh <[email protected]> (cherry picked from commit ef48945)

Add the NTP fileset to the Zeek Module. Co-authored-by: Andrew Kroh <[email protected]> (cherry picked from commit ef48945) Co-authored-by: Alex Resnick <[email protected]>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 25, 2021

andresrc added the Team:Security-External Integrations label Feb 25, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 25, 2021

legoguy1000 changed the title ~~Add Zeek NTP Fileset~~ [Filebeat] Add Zeek NTP Fileset Feb 25, 2021

legoguy1000 force-pushed the zeek-filesets branch from 3636ebd to 5744a87 Compare February 25, 2021 16:03

legoguy1000 marked this pull request as ready for review February 25, 2021 16:03

andrewkroh added the enhancement label Feb 25, 2021

leehinman self-assigned this Mar 1, 2021

legoguy1000 force-pushed the zeek-filesets branch from 5744a87 to 1b73d29 Compare March 1, 2021 16:41

legoguy1000 force-pushed the zeek-filesets branch from 1b73d29 to 9499306 Compare March 2, 2021 01:21

legoguy1000 force-pushed the zeek-filesets branch 2 times, most recently from ece6da4 to 8b9ba9f Compare March 3, 2021 20:09

legoguy1000 force-pushed the zeek-filesets branch from 8b9ba9f to 332c103 Compare March 4, 2021 16:43

legoguy1000 force-pushed the zeek-filesets branch from 332c103 to f7a01ec Compare March 10, 2021 19:12

andrewkroh added the Filebeat Filebeat label Mar 16, 2021

andrewkroh reviewed Mar 17, 2021

View reviewed changes

x-pack/filebeat/module/zeek/ntp/_meta/fields.yml Outdated Show resolved Hide resolved

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

legoguy1000 added 3 commits March 17, 2021 17:22

Add Zeek NTP Fileset

87045f9

update PR number

d0e0e31

update files

bafd309

legoguy1000 and others added 2 commits March 17, 2021 17:22

fixed typos

f779915

Update docs and changelog

abed2c7

andrewkroh force-pushed the zeek-filesets branch from f7a01ec to abed2c7 Compare March 17, 2021 21:22

andrewkroh approved these changes Mar 17, 2021

View reviewed changes

andrewkroh merged commit ef48945 into elastic:master Mar 17, 2021

andrewkroh pushed a commit to andrewkroh/beats that referenced this pull request Mar 17, 2021

[Filebeat] Add Zeek NTP Fileset (elastic#24224)

0b1a2ac

Add the NTP fileset to the Zeek Module. Co-authored-by: Andrew Kroh <[email protected]> (cherry picked from commit ef48945)

andrewkroh mentioned this pull request Mar 17, 2021

Cherry-pick #24224 to 7.x: [Filebeat] Add Zeek NTP Fileset #24619

Merged

6 tasks

andrewkroh added the v7.13.0 label Mar 17, 2021

legoguy1000 deleted the zeek-filesets branch March 17, 2021 22:16

111andre111 mentioned this pull request May 6, 2022

Additional zeek module log files elastic/integrations#3288

Closed

28 tasks

jamiehynds mentioned this pull request Apr 19, 2021

Add additional log types to the Filebeat Zeek Module #12724

Closed

25 tasks

This was referenced Aug 9, 2021

Zeek Integration missing elastic/integrations#1461

Closed

[Zeek] Add NTP and Signature data streams elastic/integrations#1515

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Add Zeek NTP Fileset #24224

[Filebeat] Add Zeek NTP Fileset #24224

legoguy1000 commented Feb 25, 2021 •

edited by andrewkroh

Loading

elasticmachine commented Feb 25, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

legoguy1000 commented Feb 25, 2021

legoguy1000 commented Feb 25, 2021

elasticmachine commented Feb 25, 2021

leehinman commented Mar 1, 2021

legoguy1000 commented Mar 1, 2021

legoguy1000 commented Mar 2, 2021

leehinman commented Mar 3, 2021

legoguy1000 commented Mar 3, 2021

legoguy1000 commented Mar 4, 2021

andrewkroh commented Mar 17, 2021

andrewkroh left a comment

andrewkroh left a comment

andrewkroh commented Mar 17, 2021

[Filebeat] Add Zeek NTP Fileset #24224

[Filebeat] Add Zeek NTP Fileset #24224

Conversation

legoguy1000 commented Feb 25, 2021 • edited by andrewkroh Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Feb 25, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

legoguy1000 commented Feb 25, 2021

legoguy1000 commented Feb 25, 2021

elasticmachine commented Feb 25, 2021

leehinman commented Mar 1, 2021

legoguy1000 commented Mar 1, 2021

legoguy1000 commented Mar 2, 2021

leehinman commented Mar 3, 2021

legoguy1000 commented Mar 3, 2021

legoguy1000 commented Mar 4, 2021

andrewkroh commented Mar 17, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh commented Mar 17, 2021

legoguy1000 commented Feb 25, 2021 •

edited by andrewkroh

Loading

elasticmachine commented Feb 25, 2021 •

edited by jenkins-beats-ci bot

Loading