Journalbeat matches support && minor additions #8324

kvch · 2018-09-17T10:40:09Z

Matching support

From now on it's possible to match for journal entry fields in Journalbeat using the new option matches. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries.

Example configuration which returns NGINX and dhclient entries from the journal:

include_matches:
  - "systemd.unit=nginx"
  - "process.name=dhclient"

Docker fields

Added docker fields from: https://docs.docker.com/config/containers/logging/journald/

container.id
container.id_truncated
container.name
container.image.tag
container.partial

Parse timestamp of entries

Journalbeat parses the timestamp of the entry and adds it to the event as @timestamp. The time of reading by Journalbeat is saved in read_timestamp.

Save custom fields

Custom fields by various sources are stored under custom. Field names are normalized, meaning "_" prefix is removed and every letter is lowercase.

Fields && processors

From now on it is possible to configure processors and fields, etc on input level.

Metrics

The size of each open reader is reporting in bytes:

{
  "journalbeat": {
    "journals": {
      "journal_1": {
         "path": "system.journal",
         "size_in_bytes": 123124214,
       }
    }
}

journalbeat/checkpoint/file_windows.go

ph · 2018-09-26T12:31:38Z

@kvch I will wait for the other to get merged, I don't know whats new :)

kvch · 2018-09-26T16:01:01Z

@ph I am waiting for you to approve my other PR, so I can merge it ;)

ruflin

Could we name it include_matches to have it close to include_fields and include_lines?

I understand why you went with - "systemd.unit=nginx" as - systemd.unit: nginx does not work because of the dot. I'm now trying to figure out if it's a trick we should also use in the copy processor as an example or rather have it similar to the copy processor. This one here is definitively shorter. Thoughts?

ruflin · 2018-09-30T18:28:58Z

journalbeat/Dockerfile

@@ -0,0 +1,17 @@
+FROM golang:1.10.3


I think this docker file will become useful but I wonder if it is used in the tests or if it was missing before?

It was missing before. I has every dependency installed, so everyone can build and run journalbeat.

kvch · 2018-10-01T09:31:57Z

I renamed matches to include_matches. It's much better name, thanks for the suggestion.

One more reason why I went with "systemd.unit=nginx" is that Journalbeat does not parse these expressions. It simply passes it to the journal reader which does the matching for us.
Processors are different, because the Beat does the processing for you, not an outside lib. So I don't think it's required to configure include_matches the same as processors. However, if it leads to a better cleaner configuration I am all for changing it.

ph

I think we should add unit tests for the added methods.

ph · 2018-09-26T12:31:00Z

journalbeat/.gitignore

@@ -0,0 +1,10 @@
+/.idea
+/build
+


ph · 2018-10-01T19:21:35Z

journalbeat/reader/journal.go

+			return fmt.Errorf("error adding match to journal %v", err)
+		}
+
+		err = j.AddDisjunction()


Oh this is an interesting API, this is to add a logical or for multiple matches.
Maybe in a future version we could allow users to configure OR / AND.

ph · 2018-10-01T19:25:03Z

journalbeat/tests/system/test_base.py

+
+        exit_code = journalbeat_proc.kill_and_wait()
+        assert exit_code == 0
+



Thanks for the integration test!

ph · 2018-10-01T19:25:51Z

journalbeat/reader/journal.go

+		} else {
+			if isKept(kk) {
+				fields.Put(kk, v)
+			}


We could probably add a unit test for that to make sure the normalization is correctly done?

ph · 2018-10-01T19:26:11Z

journalbeat/reader/journal.go

+	}
+	return nil
+}
+


houndci-bot · 2018-10-02T11:55:29Z

journalbeat/cmd/instance/metrics.go

+	journals[path] = journal
+}
+
+func StopMonitoringJournal(path string) {


exported function StopMonitoringJournal should have comment or be unexported

houndci-bot · 2018-10-02T11:55:30Z

journalbeat/cmd/instance/metrics.go

+	monitoring.NewFunc(metrics, "journals", reportJournalSizes, monitoring.Report)
+}
+
+func AddJournalToMonitor(path string, journal *sdjournal.Journal) {


exported function AddJournalToMonitor should have comment or be unexported

houndci-bot · 2018-10-02T11:55:30Z

journalbeat/cmd/instance/metrics.go

+	journals map[string]*sdjournal.Journal
+)
+
+func SetupJournalMetrics() {


exported function SetupJournalMetrics should have comment or be unexported

ruflin · 2018-10-03T06:16:18Z

I'm good with keeping these config options. I would probably rather change the other ones to make them shorter :-)

ph · 2018-10-03T12:50:44Z

@kvch Changes LGTM, I will test it today with the filtering.

ph · 2018-10-03T12:51:15Z

@kvch Also can you address Hound's comments?

kvch · 2018-10-03T13:47:40Z

@ph I have added comments, but it does not seem to be satisfied: ad7ddcd

kvch · 2018-10-04T15:10:38Z

Failing tests are unrelated. If something is missing, I am adding it in a follow up PR.

### Matching support From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` ### Docker fields Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` ### Parse timestamp of entries Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. ### Save custom fields Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. ### Fields && processors From now on it is possible to configure `processors` and `fields`, etc on `input` level. ### Metrics The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ```

* Initialize Journalbeat (#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (#8324) ### Matching support From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` ### Docker fields Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` ### Parse timestamp of entries Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. ### Save custom fields Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. ### Fields && processors From now on it is possible to configure `processors` and `fields`, etc on `input` level. ### Metrics The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (#8618) * Packaging of journalbeat (#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (#8735) * Add journalbeat docs

* Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

* Add Journalbeat (#8703) * Initialize Journalbeat (#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (#8618) * Packaging of journalbeat (#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

* Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) ### Matching support From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` ### Docker fields Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` ### Parse timestamp of entries Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. ### Save custom fields Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. ### Fields && processors From now on it is possible to configure `processors` and `fields`, etc on `input` level. ### Metrics The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs

* Add Journalbeat (elastic#8703) * Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

### Matching support From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` ### Docker fields Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` ### Parse timestamp of entries Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. ### Save custom fields Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. ### Fields && processors From now on it is possible to configure `processors` and `fields`, etc on `input` level. ### Metrics The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ```

kvch added in progress Pull request is currently in progress. review blocked labels Sep 17, 2018

kvch requested a review from ph September 17, 2018 10:40

houndci-bot reviewed Sep 17, 2018

View reviewed changes

journalbeat/checkpoint/file_windows.go Outdated Show resolved Hide resolved

kvch mentioned this pull request Sep 17, 2018

Journalbeat #8323

Closed

23 tasks

kvch changed the title ~~Journalbeat matches support~~ Journalbeat matches support && minor additions Sep 17, 2018

kvch added 6 commits September 29, 2018 11:36

add initial support for filtering on journal reader side

ee766e1

add journald fields of docker

79c2ae3

add unknown fields to "custom"

33f7fb7

add read_timestamp and parse timestamp of entry

df02467

fix @timestamp

6347351

add Dockerfile

b99005e

kvch force-pushed the feature-journalbeat-filters-and-friends branch from 29f8507 to b99005e Compare September 29, 2018 09:37

rm extra neline

3c2f283

kvch removed the blocked label Sep 29, 2018

ruflin reviewed Sep 30, 2018

View reviewed changes

kvch added 3 commits October 1, 2018 09:16

add missing fields

cc90e1e

support Eventmetadata on input level

2a8d113

rename matches to include_matches

8a916d8

kvch added 2 commits October 1, 2018 13:53

report size of the journal file

1f50262

mark as beta

4401d9b

ph suggested changes Oct 1, 2018

View reviewed changes

kvch added 3 commits October 2, 2018 09:28

prefix field names with "journald."

80aa2e4

add missing metrics file

f23a8a1

add unit tests for setupMatches and toEvent

ddf8598

houndci-bot reviewed Oct 2, 2018

View reviewed changes

kvch added 2 commits October 3, 2018 12:39

feed the hound

ad7ddcd

mark journalbeat experimental

528cba7

kvch removed the in progress Pull request is currently in progress. label Oct 3, 2018

ph approved these changes Oct 4, 2018

View reviewed changes

ph added the new beat label Oct 4, 2018

kvch merged commit c66a084 into elastic:feature-journalbeat Oct 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Journalbeat matches support && minor additions #8324

Journalbeat matches support && minor additions #8324

kvch commented Sep 17, 2018 •

edited

Loading

ph commented Sep 26, 2018

kvch commented Sep 26, 2018

ruflin left a comment

ruflin Sep 30, 2018

kvch Oct 1, 2018

kvch commented Oct 1, 2018 •

edited

Loading

ph left a comment

ph Sep 26, 2018

ph Oct 1, 2018

ph Oct 1, 2018

ph Oct 1, 2018

ph Oct 1, 2018

houndci-bot Oct 2, 2018

houndci-bot Oct 2, 2018

houndci-bot Oct 2, 2018

ruflin commented Oct 3, 2018

ph commented Oct 3, 2018

ph commented Oct 3, 2018

kvch commented Oct 3, 2018

kvch commented Oct 4, 2018 •

edited

Loading


		exit_code = journalbeat_proc.kill_and_wait()
		assert exit_code == 0

		@@ -0,0 +1,10 @@
		/.idea
		/build

Journalbeat matches support && minor additions #8324

Journalbeat matches support && minor additions #8324

Conversation

kvch commented Sep 17, 2018 • edited Loading

Matching support

Docker fields

Parse timestamp of entries

Save custom fields

Fields && processors

Metrics

ph commented Sep 26, 2018

kvch commented Sep 26, 2018

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Oct 1, 2018 • edited Loading

ph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Oct 3, 2018

ph commented Oct 3, 2018

ph commented Oct 3, 2018

kvch commented Oct 3, 2018

kvch commented Oct 4, 2018 • edited Loading

kvch commented Sep 17, 2018 •

edited

Loading

kvch commented Oct 1, 2018 •

edited

Loading

kvch commented Oct 4, 2018 •

edited

Loading