Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize Journalbeat #8277

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
485ae50
vendor sdjournal and its dependencies
kvch Sep 10, 2018
9abf838
add E2E tests
kvch Sep 10, 2018
fca5ac8
init minimal journalbeat
kvch Sep 10, 2018
c4600b0
fix make check
kvch Sep 10, 2018
f559751
add missing notices
kvch Sep 11, 2018
4e5f157
purge generated README
kvch Sep 11, 2018
ba0ffc3
fix travis.yml
kvch Sep 11, 2018
9a0e7e7
add comments to types
kvch Sep 11, 2018
f54fdd7
refactoring of e2e tests
kvch Sep 11, 2018
328457a
add e2e test seeking to cursor and reading
kvch Sep 11, 2018
0797f16
rm debug log
kvch Sep 11, 2018
fc29e7e
clean up configuration file
kvch Sep 11, 2018
ff94b5d
add missing comment
kvch Sep 11, 2018
e5308ed
fix pep8 errors
kvch Sep 11, 2018
61aa201
wait, so cpu is not consumed
kvch Sep 11, 2018
4d4f79d
add missing package to travis && sort list
kvch Sep 14, 2018
7a01f63
cleanup && fix epic tyop
kvch Sep 17, 2018
c9e33db
add more docs to config
kvch Sep 17, 2018
cd1c0d6
address minor problems
kvch Sep 19, 2018
457c5db
merge channels from goroutines
kvch Sep 20, 2018
8d36860
propagate constructor errors
kvch Sep 20, 2018
1fc7a59
add scoped logging for inputs and readers
kvch Sep 24, 2018
905f9fd
address comments of Ruflin
kvch Sep 25, 2018
0f1f364
make `publishAll` blocking
kvch Sep 26, 2018
28c7916
close channel from defer
kvch Sep 26, 2018
171bd48
better crosscompiling, still requires work
kvch Sep 27, 2018
650e113
use range to avoid reading from closed channel
kvch Sep 27, 2018
88e4a1f
update description of path option
kvch Sep 27, 2018
88f039f
remove one for add a new if
kvch Sep 28, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,12 @@ jobs:
go: $GO_VERSION
stage: test

# Journalbeat
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a follow up issue or meta issue to make sure we add a Job in Jenkins for it in the near future?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added one more element to the TODO list in the meta issue: #8323

- os: linux
env: TARGETS="-C journalbeat testsuite"
go: $GO_VERSION
stage: test

# Generators
- os: linux
env: TARGETS="-C generator/metricbeat test"
Expand Down Expand Up @@ -146,10 +152,12 @@ addons:
apt:
update: true
packages:
- python-virtualenv
- libc6-dev-i386
- libpcap-dev
- xsltproc
- libsystemd-journal-dev
- libxml2-utils
- python-virtualenv
- xsltproc

before_install:
- python --version
Expand Down
30 changes: 30 additions & 0 deletions NOTICE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,36 @@ COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

--------------------------------------------------------------------
Dependency: github.com/coreos/go-systemd
Revision: eee3db372b31153ca0b90702e165948699803fd0
License type (autodetected): Apache-2.0
./vendor/github.com/coreos/go-systemd/LICENSE:
--------------------------------------------------------------------
Apache License 2.0

-------NOTICE-----
CoreOS Project
Copyright 2018 CoreOS, Inc

This product includes software developed at CoreOS, Inc.
(http://www.coreos.com/).

--------------------------------------------------------------------
Dependency: github.com/coreos/pkg
Revision: 97fdf19511ea361ae1c100dd393cc47f8dcfa1e1
License type (autodetected): Apache-2.0
./vendor/github.com/coreos/pkg/LICENSE:
--------------------------------------------------------------------
Apache License 2.0

-------NOTICE-----
CoreOS Project
Copyright 2014 CoreOS, Inc

This product includes software developed at CoreOS, Inc.
(http://www.coreos.com/).

--------------------------------------------------------------------
Dependency: github.com/davecgh/go-spew
Version: v1.1.0
Expand Down
9 changes: 9 additions & 0 deletions journalbeat/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
/.idea
/build
.DS_Store
.journalbeat_position
/journalbeat
/journalbeat.test
*.pyc
data/meta.json
/*.journal
17 changes: 17 additions & 0 deletions journalbeat/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
BEAT_NAME=journalbeat
BEAT_TITLE=Journalbeat
SYSTEM_TESTS=false
TEST_ENVIRONMENT=false
ES_BEATS?=..
GOX_FLAGS=-cgo
GOX_OS=linux

# Path to the libbeat Makefile
-include $(ES_BEATS)/libbeat/scripts/Makefile

.PHONY: before-build
before-build:

# Collects all dependencies and then calls update
.PHONY: collect
collect:
5 changes: 5 additions & 0 deletions journalbeat/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Journalbeat

Journalbeat is an open source data collector to read and forward journal entries from Linuxes with systemd.

## Getting started
45 changes: 45 additions & 0 deletions journalbeat/_meta/beat.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
###################### Journalbeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The journalbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/journalbeat/index.html

# For more available modules and options, please see the journalbeat.reference.yml sample
# configuration file.

#=========================== Journalbeat inputs =============================

journalbeat.inputs:
# Paths that should be crawled and fetched. Possible values files and directories.
# When setting a directory, all journals under it are merged.
# When empty starts to read from local journal.
- paths: []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make it clear that path must be a directly, maybe adding validation to make sure its a directory and not files?

Coming from filebeat I thought it was a wildcard for files and I've used something like this:

/var/log/journal/xxxxxxxxx/*.journal
===


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the YAML file, it's says glob are supported so I've tried with the following:

/var/log/journal/*/

but I've received an error:

 stat /var/log/journal/*/: no such file or directory


# The number of seconds to wait before trying to read again from journals.
#backoff: 1s
# Multiplier of backoff value.
#backoff_factor: 2
# The maximum number of seconds to wait before attempting to read again from journals.
#max_backoff: 60s

# Position to start reading from journal. Valid values: head, tail, cursor
seek: tail

#========================= Journalbeat global options ============================
#journalbeat:
# Name of the registry file. If a relative path is used, it is considered relative to the
# data path.
#registry_file: registry

# The number of seconds to wait before trying to read again from journals.
#backoff: 1s
# Multiplier of backoff value.
#backoff_factor: 2
# The maximum number of seconds to wait before attempting to read again from journals.
#max_backoff: 60s

# Position to start reading from all journal. Possible values: head, tail, cursor
#seek: head
Loading