Initialize Journalbeat #8277

kvch · 2018-09-10T21:37:56Z

This is the first PR to initialize Journalbeat with minimal functionality.

The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (backoff, backoff_factor, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of paths. The readers are not going to implement the interface Harverster until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code.

Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated.

Example configuration to read from the beginning of the local journal

journalbeat.inputs:
- paths: []
  seek: head

Features

read from local journal, journal file and directory
position tracking by using checkpointing as it's done in Winlogbeat
seek to "tail", "head", "cursor"
minimal E2E tests
fields.yml and documentation

Vendored:

github.com/coreos/go-systemd/sdjournal

Questions to answer in the future by me

Are readers need to be stopped manually when journals deleted/rotated?
Is the saved cursor valid after rotation? Yes.

houndci-bot · 2018-09-10T21:38:34Z

journalbeat/reader/journal.go

+	}
+}
+
+func (r *Reader) Close() {


exported method Reader.Close should have comment or be unexported

houndci-bot · 2018-09-10T21:38:34Z

journalbeat/reader/journal.go

+	"github.com/elastic/beats/libbeat/logp"
+)
+
+type Config struct {


exported type Config should have comment or be unexported

houndci-bot · 2018-09-10T21:38:34Z

journalbeat/input/input.go

+func (i *Input) Stop() {
+}
+
+func (i *Input) Wait() {


exported method Input.Wait should have comment or be unexported

houndci-bot · 2018-09-10T21:38:34Z

journalbeat/input/input.go

+
+}
+
+func (i *Input) Stop() {


exported method Input.Stop should have comment or be unexported

houndci-bot · 2018-09-10T21:38:34Z

journalbeat/input/input.go

+	}
+}
+
+func (i *Input) Run() {


exported method Input.Run should have comment or be unexported

houndci-bot · 2018-09-10T21:38:36Z

journalbeat/config/config.go

+	"github.com/elastic/beats/libbeat/common"
+)
+
+type Config struct {


exported type Config should have comment or be unexported

houndci-bot · 2018-09-10T21:38:36Z

journalbeat/checkpoint/file_windows.go

+)
+
+const (
+	_FILE_FLAG_WRITE_THROUGH = 0x80000000


don't use ALL_CAPS in Go names; use CamelCase

houndci-bot · 2018-09-10T21:38:36Z

journalbeat/beater/journalbeat.go

+	i.Run()
+}
+
+func (bt *Journalbeat) Stop() {


exported method Journalbeat.Stop should have comment or be unexported

houndci-bot · 2018-09-10T21:38:36Z

journalbeat/beater/journalbeat.go

+	return bt, nil
+}
+
+func (bt *Journalbeat) Run(b *beat.Beat) error {


exported method Journalbeat.Run should have comment or be unexported

houndci-bot · 2018-09-10T21:38:36Z

journalbeat/beater/journalbeat.go

+	checkpoint *checkpoint.Checkpoint // Persists event log state to disk.
+}
+
+func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {


exported function New should have comment or be unexported

kvch · 2018-09-10T21:42:09Z

jenkins test this

houndci-bot · 2018-09-11T06:28:07Z

journalbeat/beater/journalbeat.go

+	"github.com/elastic/beats/journalbeat/config"
+)
+
+type Journalbeat struct {


exported type Journalbeat should have comment or be unexported

houndci-bot · 2018-09-11T06:28:08Z

journalbeat/beater/journalbeat.go

+	"github.com/elastic/beats/journalbeat/config"
+)
+
+type Journalbeat struct {


exported type Journalbeat should have comment or be unexported

jsoriano · 2018-09-11T08:01:38Z

.travis.yml

@@ -99,6 +99,12 @@ jobs:
      go: $GO_VERSION
      stage: test

+    # Journalbeat
+    - os: linux
+      env: TARGETS="-C filebeat testsuite"


s/filebeat/journalbeat/ ? 🙂

houndci-bot · 2018-09-11T11:05:04Z

journalbeat/reader/journal.go

+)
+
+const (
+	LocalSystemJournalID = "LOCAL_SYSTEM_JOURNAL"


exported const LocalSystemJournalID should have comment (or a comment on this block) or be unexported

kvch · 2018-09-17T10:26:33Z

I have created a meta issue to track matters related to Journalbeat: #8323

ph · 2018-09-18T13:09:52Z

journalbeat/.gitignore

@@ -0,0 +1,10 @@
+/.idea
+/build
+


we can remove this line?

ph · 2018-09-18T13:38:11Z

journalbeat/_meta/fields.common.yml

+          required: false
+          example: 123
+          description: >
+            The line number of the code which generated the log message.


Should it be also in log.source?

What do you mean?

Looking at ECS isn't this field should be named log.source instead? Similar to what we do with Filebeat.

This is different. log.source is path to the log file the line is read from. code.line is the number of the code line which generated the message.

ph · 2018-09-18T13:39:40Z

journalbeat/beater/journalbeat.go

+	}
+
+	wg.Wait()
+	bt.checkpoint.Shutdown()


bt.checkpoint.Shutdown() look like a good case for defer that we could set up after the ack handler?

ph · 2018-09-18T13:43:49Z

journalbeat/checkpoint/checkpoint.go

+	dir := filepath.Dir(c.file)
+	logp.Info("Creating %s if it does not exist.", dir)
+	return os.MkdirAll(dir, os.FileMode(0750))
+}


NOTES: I only did a quick sanity check on them, because theses library are coming from winlogbeat.

ph · 2018-09-18T13:44:10Z

journalbeat/checkpoint/file_unix.go

+
+func create(path string) (*os.File, error) {
+	return os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC|os.O_SYNC, 0600)
+}


NOTES: I only did a quick sanity check on them, because theses library are coming from winlogbeat.

ph · 2018-09-18T14:06:16Z

journalbeat/input/input.go

+	if err != nil {
+		logp.Err("Error connecting: %v", err)
+		return
+	}


Previous comment on returning an error.

What do you mean? It's not a constructor. Also, I tried mimicing Input of Filebeat. There is logging on every level in each non-constructor. So I consider this acceptable.

ph · 2018-09-18T14:08:03Z

journalbeat/magefile.go

+)
+
+func init() {
+	mage.BeatDescription = "Winlogbeat ships Windows event logs to Elasticsearch or Logstash."


Winlogbeat?

ph · 2018-09-18T14:37:29Z

journalbeat/input/input.go

+				}
+			}
+		}
+	}


I am not sure the above is doing what you are expecting. Adding the Follow implementation in the comment, Note I need to read more on the Reader class but at first look we could have an issue.

func (r *Reader) Follow() <-chan *beat.Event { out := make(chan *beat.Event) go r.readEntriesFromJournal(out) return out }

The above return a channel that we consume with range, this means we will consume all the events until events are available or the channel is closed, looking at the reader implementation we only close the channel when we received a done signal.

Since the publish to the queue is done in a single goroutine, it is possible that a specific reader in the list make all the other readers starves for resource and the possibility to push publish events?

I think we could fix it in two differents way:

Provide a call back on new events and that callback will do the actual push to the queue.

Merge N goroutines into a single goroutine and read from that. Similar to this link

Now I tested it again for multiple runs. It seems like sometimes harvesters are starved. I haven't noticed it until you pointed it out. Thanks!

ph · 2018-09-18T14:46:21Z

journalbeat/reader/journal.go

+	done    chan struct{}
+}
+
+// New creates a new journal reader and moves the FP to the configured position.


I think this type would benefit from having a scoped logger, it will be useful to know which input created that logger when we are debugging users errors.

journalbeat/reader/journal.go

kvch · 2018-09-24T13:18:39Z

Failing tests are unrelated.

ph · 2018-09-24T14:17:46Z

journalbeat/input/input.go

+		}
+	}
+
+}


Instead of doing a for loop here and check for i.done and calling i.publishAll we should just call i.publishAll. This would simplify the code logic.

ph · 2018-09-24T14:36:12Z

journalbeat/input/input.go

+	for e := range out {
+		client.Publish(*e)
+	}
+}


The way the code is structured, publishAll should be a blocking call and we should do the check if is.Done is called here.

Maybe something like the following will work.

func (i *Input) publishAll(client beat.Client) { out := make(chan *beat.Event) defer close(out) var wg sync.WaitGroup merge := func(in chan *beat.Event) { wg.Add(1) go func(in chan *beat.Event, out chan *beat.Event) { for { select { case <-i.Done: return case out <- c: } } }(in) } for _, r := range i.readers { c := r.Follow() merge(c) } for { select { case <-is.Done: break case e <- out: client.Publish(*e) } } wg.Wait() }

ruflin · 2018-09-25T07:06:13Z

.travis.yml

@@ -99,6 +99,12 @@ jobs:
      go: $GO_VERSION
      stage: test

+    # Journalbeat


Can we have a follow up issue or meta issue to make sure we add a Job in Jenkins for it in the near future?

I added one more element to the TODO list in the meta issue: #8323

ruflin · 2018-09-25T07:08:42Z

journalbeat/_meta/fields.common.yml

+  description: >
+    Contains common fields available in all event types.
+  fields:
+    - name: coredump


Do you expect all these fields like coredump, kernel to be on the top level?

Yes, because I specify the field names here: https://github.com/elastic/beats/pull/8277/files#diff-e6feda99a2ce30c6c943d4a15664abd4R24

I am OK with moving them to lower levels to fit better into ECS. I haven't found any namespace which could contain these fields. Do you have any suggestions?

Are they all related to journald? If yes, I wonder if we should prefix them with journald?

It would be nice to have some example json events in the future somewhere which we can use for the docs but should also make it much clearer how the events will look like.

Yes, these are all related to journald. Here is the full list of fields provided by systemd journal: https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html
But some of the fields are not coming from the journal, but an application appends some fields. For example coredump is provided by systemd-coredump kernel helper. So prefixing with journald might be confusing.

Are all the fields part of systemd? In general I would suggest to first prefix them with "something" and make them more global over time. At the same time not a blocker.

Ok. I will prefix fields which does not belong anywhere in ECS with journald in a follow up.

Ok, lets have a quick chat before you put all the work into prefixing it.

It's too late. :D 80aa2e4

haha, great. Going form prefix to non prefix is always easy (as we have alias), otherway around is hard.

ruflin · 2018-09-25T07:09:44Z

journalbeat/beater/journalbeat.go

+func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
+	config := config.DefaultConfig
+	if err := cfg.Unpack(&config); err != nil {
+		return nil, fmt.Errorf("Error reading config file: %v", err)


nit: Error should be lower case.

ruflin · 2018-09-25T07:15:07Z

journalbeat/checkpoint/checkpoint.go

+			break loop
+		case s := <-c.save:
+			c.lock.Lock()
+			c.states[s.Path] = s


Instead of using locks around c.states could we use some methods like setState that have the lock inside. Like this we make sure we never forget.

I just simply copied checkpointing from Winlogbeat and haven't touched it apart from adjusting the structs which are stored. I would rather not make further changes, because we are going to port everything to the new registry soon.

Got it. @andrewkroh FYI

ruflin · 2018-09-25T07:16:36Z

journalbeat/checkpoint/checkpoint.go

+	file, err := create(tempFile)
+	if os.IsNotExist(err) {
+		// Try to create directory if it does not exist.
+		if createDirErr := c.createDir(); createDirErr == nil {


Is this something we should already try on startup? I "think" we do it in Filebeat to warn the user that he might miss permissions to update / create the registry file as in the past we had the issue that FB was running but states could not be written.

ruflin · 2018-09-25T07:21:14Z

journalbeat/config/config.go

+	Backoff:       1 * time.Second,
+	BackoffFactor: 2,
+	MaxBackoff:    60 * time.Second,
+	Seek:          "cursor",


Should the default be the same as in the config file?

ruflin · 2018-09-25T07:22:36Z

journalbeat/input/config.go

+		Backoff:       1 * time.Second,
+		BackoffFactor: 2,
+		MaxBackoff:    60 * time.Second,
+		Seek:          "tail",


The global and local default should be the same?

Yes, I missed changing the global. Thanks for catching it.

ruflin · 2018-09-25T07:23:55Z

journalbeat/input/input.go

+
+// Run connects to the output, collects entries from the readers
+// and then publishes the events.
+func (i *Input) Run() {


Should run return an error?

I tried following the interface defined in https://github.com/elastic/beats/blob/master/filebeat/input/input.go#L43, so when we are moving it to Filebeat, less refactoring is required. Thus, I would not return an error from this function.

We should probably introduce an error at one stage in Filebeat :-)

ruflin · 2018-09-25T07:25:10Z

journalbeat/reader/journal.go

+
+// Reader reads entries from journal(s).
+type Reader struct {
+	j       *sdjournal.Journal


can we call it journal or something more descriptive? I prefer to have single char variables only for the struct inside methods or for loops in most cases.

ruflin · 2018-09-25T07:29:06Z

journalbeat/tests/system/input/test.registry

@@ -0,0 +1,6 @@
+update_time: 2018-09-11T10:06:50.895829905Z
+journal_entries:
+- path: /home/n/go/src/github.com/elastic/beats/journalbeat/tests/system/input/test.journal


Great to have such examples in.

ph · 2018-09-26T12:35:08Z

journalbeat/input/input.go

 		}(in)
 	}
+
+	// close output channel after all input channels are merged or beats is stopped
 	go func() {
 		wg.Wait()
 		close(out)
 	}()


@kvch There is no need to create a goroutine to wait on the WaitGroup, instead it could be done in a defer.

If I close using defer journalbeat panics with "send on a closed channel". I am not sure how exactly your solution supposed to work. Could you give more context?
I mean OFC I understand defer is called at the end of the function. But how would you rewrite the code so it could work?

Using defer sometimes works correctly. But most of the time journalbeat panics, either because of a null pointer dereference or sending on a closed channel. So closing the channel after wait is necessary to provide stable behaviour.

ph · 2018-09-26T17:02:04Z

because I am weird I've tried to build it directly undermacos, since this beats require linux and some journald headers files, could we report something better when you are building it with an inferior os :)

go test -i github.com/elastic/beats/journalbeat github.com/elastic/beats/journalbeat/beater github.com/elastic/beats/journalbeat/checkpoint github.com/elastic/beats/journalbeat/cmd github.com/elastic/beats/journalbeat/config github.com/elastic/beats/journalbeat/include github.com/elastic/beats/journalbeat/input github.com/elastic/beats/journalbeat/reader
# github.com/elastic/beats/vendor/github.com/coreos/go-systemd/sdjournal
../vendor/github.com/coreos/go-systemd/sdjournal/journal.go:27:11: fatal error: 'systemd/sd-journal.h' file not found
 #include <systemd/sd-journal.h>
          ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [unit] Error 2

ph · 2018-09-26T17:47:16Z

After bootstrapping a linux vm and installing the libsystemd-dev package I was able to build it.

Testing:

Problems when stopping it with CTRL-C

I haven't look at all the code, but I was not able to exit journalbeat, even with multiple CTRL-C

2018-09-26T17:42:00.460Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":5}},"total":{"ticks":170,"time":{"ms":9},"value":170},"user":{"ticks":10,"time":{"ms":4}}},"fd":{"limit":{"hard":1048576,"soft":1024},"open":7},"info":{"ephemeral_id":"9ed707ac-09db-4165-b7a8-2269c6749d4f","uptime":{"ms":150175}},"memstats":{"gc_next":4194304,"memory_alloc":1606832,"memory_total":4686992}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"load":{"1":0,"15":0.03,"5":0.01,"norm":{"1":0,"15":0.015,"5":0.005}}}}}}
^C2018-09-26T17:42:05.669Z      DEBUG   [service]       service/service.go:51   Received sigterm/sigint, stopping
C2018-09-26T17:44:34.897Z      INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":135}},"total":{"ticks":140,"time":{"ms":152},"value":140},"user":{"ticks":10,"time":{"ms":17}}},"fd":{"limit":{"hard":1048576,"soft":1024},"open":5},"info":{"ephemeral_id":"3e49e629-e01c-4d2b-8825-d69edf4389c4","uptime":{"ms":30150}},"memstats":{"gc_next":4194304,"memory_alloc":1818584,"memory_total":3577192,"rss":14954496}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1,"batches":1,"total":1},"type":"console","write":{"bytes":937}},"pipeline":{"clients":1,"events":{"active":0,"published":1,"total":1},"queue":{"acked":1}}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0.01,"5":0,"norm":{"1":0,"15":0.005,"5":0}}}}}}
2018-09-26T17:45:04.888Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":5}},"total":{"ticks":150,"time":{"ms":6},"value":150},"user":{"ticks":10,"time":{"ms":1}}},"fd":{"limit":{"hard":1048576,"soft":1024},"open":5},"info":{"ephemeral_id":"3e49e629-e01c-4d2b-8825-d69edf4389c4","uptime":{"ms":60144}},"memstats":{"gc_next":4194304,"memory_alloc":2181176,"memory_total":3939784}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"load":{"1":0,"15":0.01,"5":0,"norm":{"1":0,"15":0.005,"5":0}}}}}}

ph

@kvch Good work! I've tested it with the knowledge I have about journald, it appear to work correctly with events. We still have a few things that we can do in this PR and other in followups PR.

What we should addressed in this PR.

I was never able to quit journalbeat other than kill -9 the process.
The path/globbing need some work, I was confused by the behavior.

What can be addressed in another PR.

We need to have a better way to build it, maybe a special docker image with all the correct dependencies similar to packetbeat.
Adding Fields support
Adding processors support for each input.

ph · 2018-09-26T17:41:56Z

journalbeat/_meta/beat.yml

+journalbeat.inputs:
+  # Paths that should be crawled and fetched. Glob based paths.
+  # When empty starts to read from local journal.
+- paths: []


Can we make it clear that path must be a directly, maybe adding validation to make sure its a directory and not files?

Coming from filebeat I thought it was a wildcard for files and I've used something like this:

/var/log/journal/xxxxxxxxx/*.journal ===

In the YAML file, it's says glob are supported so I've tried with the following:

/var/log/journal/*/

but I've received an error:

stat /var/log/journal/*/: no such file or directory

ph · 2018-09-26T17:54:03Z

journalbeat/input/input.go

+		Processor:     nil,
+		ACKCount: func(n int) {
+			i.logger.Infof("journalbeat successfully published %d events", n)
+		},


I think we are missing two behavior that Filebeat has.

Adding custom fields to the events with EventMetadata

The support of processor per input.

The current code only support processors defined globally.

The above can be done in another PR.

ph · 2018-09-27T20:14:49Z

journalbeat/input/input.go

-				case v := <-c:
+				default:
+				}
+				for v := range c {


our discussion this morning made me think and look at the code.
Instead of using the default route you can do the following, which replace 2 for loop with 1.

select { case <-i.done: return case v, ok := <-c: // ok is true if fetch is succesful. if !ok { return } out <- v }

ph · 2018-09-27T21:19:09Z

I've tested the new PR and we can now correctly interrupt the execution of journaldbeat.

@ruflin I think we can merge that in the feature branch and iterate WDYT?

kvch · 2018-09-28T08:59:21Z

Rebased the branch on top of master, so tests should be green.

ph · 2018-09-28T12:22:59Z

Great work @kvch!

ruflin · 2018-09-30T18:45:20Z

vendor/vendor.json

@@ -126,6 +126,24 @@
 			"revision": "af9db2027c98c61ecd8e17caa5bd265792b9b9a2",
 			"revisionTime": "2018-03-18T00:15:26Z"
 		},
+		{
+			"checksumSHA1": "cEszpxh1szqTb440ze4hm/Vfm40=",
+			"path": "github.com/coreos/go-systemd/sdjournal",


Seeing that there are release for go-systemd, I wonder if we should tie it to one?

This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal

* Initialize Journalbeat (#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (#8324) ### Matching support From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` ### Docker fields Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` ### Parse timestamp of entries Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. ### Save custom fields Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. ### Fields && processors From now on it is possible to configure `processors` and `fields`, etc on `input` level. ### Metrics The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (#8618) * Packaging of journalbeat (#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (#8735) * Add journalbeat docs

* Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

* Add Journalbeat (#8703) * Initialize Journalbeat (#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (#8618) * Packaging of journalbeat (#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

joelika · 2018-12-12T15:04:47Z

@kvch

Are readers need to be stopped manually when journals deleted/rotated?

I've been evaluating journalbeaet on a RHEL 7 setup, and I noticed that journalbeat stops sending logs to logstash daily. I've been debugging, and it appears as if it happens when journald rotates it's journal.

Is there a way to detect this in journalbeat and restart it? Right now, I have to go in and manually restart journalbeat each day.

I'm using the systemd configuration included in the rpm of journalbeat to manage the service.

ph · 2018-12-12T20:08:02Z

@joelika This is indeed a bug and it should be handled transparently by journalbeat, would you mind creating an issue with it.

joelika · 2018-12-13T18:41:44Z

@ph thanks! I opened #9533

* Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) ### Matching support From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` ### Docker fields Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` ### Parse timestamp of entries Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. ### Save custom fields Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. ### Fields && processors From now on it is possible to configure `processors` and `fields`, etc on `input` level. ### Metrics The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs

* Add Journalbeat (elastic#8703) * Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal

kvch added review new beat labels Sep 10, 2018

houndci-bot reviewed Sep 10, 2018

View reviewed changes

houndci-bot reviewed Sep 11, 2018

View reviewed changes

kvch added the in progress Pull request is currently in progress. label Sep 11, 2018

jsoriano reviewed Sep 11, 2018

View reviewed changes

houndci-bot reviewed Sep 11, 2018

View reviewed changes

ph self-assigned this Sep 11, 2018

kvch removed the in progress Pull request is currently in progress. label Sep 13, 2018

kvch force-pushed the feature-journalbeat-init-multiple-inputs branch 3 times, most recently from 92c014a to acf063e Compare September 14, 2018 14:36

kvch mentioned this pull request Sep 17, 2018

Journalbeat #8323

Closed

23 tasks

kvch mentioned this pull request Sep 17, 2018

Journalbeat matches support && minor additions #8324

Merged

ph suggested changes Sep 18, 2018

View reviewed changes

ph suggested changes Sep 24, 2018

View reviewed changes

ruflin reviewed Sep 25, 2018

View reviewed changes

ph reviewed Sep 26, 2018

View reviewed changes

ph suggested changes Sep 26, 2018

View reviewed changes

ph reviewed Sep 27, 2018

View reviewed changes

kvch added 3 commits September 28, 2018 10:58

vendor sdjournal and its dependencies

485ae50

add E2E tests

9abf838

init minimal journalbeat

fca5ac8

kvch added 12 commits September 28, 2018 10:58

add more docs to config

c9e33db

address minor problems

cd1c0d6

merge channels from goroutines

457c5db

propagate constructor errors

8d36860

add scoped logging for inputs and readers

1fc7a59

address comments of Ruflin

905f9fd

make publishAll blocking

0f1f364

close channel from defer

28c7916

better crosscompiling, still requires work

171bd48

use range to avoid reading from closed channel

650e113

update description of path option

88e4a1f

remove one for add a new if

88f039f

kvch force-pushed the feature-journalbeat-init-multiple-inputs branch from 6056350 to 88f039f Compare September 28, 2018 08:58

ph approved these changes Sep 28, 2018

View reviewed changes

kvch merged commit 09e8e88 into elastic:feature-journalbeat Sep 28, 2018

ruflin reviewed Sep 30, 2018

View reviewed changes

		@@ -0,0 +1,10 @@
		/.idea
		/build

Initialize Journalbeat #8277

Initialize Journalbeat #8277

Conversation

kvch commented Sep 10, 2018 • edited Loading

Example configuration to read from the beginning of the local journal

Features

Vendored:

Questions to answer in the future by me

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Sep 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Sep 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Sep 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Sep 26, 2018

ph commented Sep 26, 2018

Testing:

Problems when stopping it with CTRL-C

ph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Sep 10, 2018 •

edited

Loading

ph Sep 27, 2018 •

edited

Loading