Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Journalbeat matches support && minor additions #8324

Merged
merged 17 commits into from
Oct 4, 2018
Merged

Journalbeat matches support && minor additions #8324

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
ee766e1
add initial support for filtering on journal reader side
kvch Sep 17, 2018
79c2ae3
add journald fields of docker
kvch Sep 17, 2018
33f7fb7
add unknown fields to "custom"
kvch Sep 21, 2018
df02467
add read_timestamp and parse timestamp of entry
kvch Sep 21, 2018
6347351
fix @timestamp
kvch Sep 21, 2018
b99005e
add Dockerfile
kvch Sep 28, 2018
3c2f283
rm extra neline
kvch Sep 29, 2018
cc90e1e
add missing fields
kvch Oct 1, 2018
2a8d113
support Eventmetadata on input level
kvch Oct 1, 2018
8a916d8
rename matches to include_matches
kvch Oct 1, 2018
1f50262
report size of the journal file
kvch Oct 1, 2018
4401d9b
mark as beta
kvch Oct 1, 2018
80aa2e4
prefix field names with "journald."
kvch Oct 2, 2018
f23a8a1
add missing metrics file
kvch Oct 2, 2018
ddf8598
add unit tests for setupMatches and toEvent
kvch Oct 2, 2018
ad7ddcd
feed the hound
kvch Oct 3, 2018
528cba7
mark journalbeat experimental
kvch Oct 3, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions journalbeat/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
FROM golang:1.10.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this docker file will become useful but I wonder if it is used in the tests or if it was missing before?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was missing before. I has every dependency installed, so everyone can build and run journalbeat.

MAINTAINER Noémi Ványi <[email protected]>

RUN set -x && \
apt-get update && \
apt-get install -y --no-install-recommends \
python-pip virtualenv libsystemd-dev libc6-dev-i386 gcc-arm-linux-gnueabi && \
apt-get clean

RUN pip install --upgrade setuptools

# Setup work environment
ENV JOURNALBEAT_PATH /go/src/github.com/elastic/beats/journalbeat

RUN mkdir -p $JOURNALBEAT_PATH/build/coverage
WORKDIR $JOURNALBEAT_PATH
HEALTHCHECK CMD exit 0
15 changes: 15 additions & 0 deletions journalbeat/_meta/beat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,17 @@ journalbeat.inputs:
# Position to start reading from journal. Valid values: head, tail, cursor
seek: tail

# Exact matching for field values of events.
# Matching for nginx entries: "systemd.unit=nginx"
#include_matches: []

# Optional fields that you can specify to add additional information to the
# output. Fields can be scalar values, arrays, dictionaries, or any nested
# combination of these.
#fields:
# env: staging


#========================= Journalbeat global options ============================
#journalbeat:
# Name of the registry file. If a relative path is used, it is considered relative to the
Expand All @@ -43,3 +54,7 @@ journalbeat.inputs:

# Position to start reading from all journal. Possible values: head, tail, cursor
#seek: head

# Exact matching for field values of events.
# Matching for nginx entries: "systemd.unit=nginx"
#matches: []
235 changes: 126 additions & 109 deletions journalbeat/_meta/fields.common.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
description: >
Contains common fields available in all event types.
fields:
- name: read_timestamp
description: >
The time when Journalbeat read the journal entry.
- name: coredump
type: group
description: >
Expand All @@ -16,117 +19,145 @@
type: keyword
description: >
Annotations of messages containing coredumps from user units.
- name: object
- name: journald
type: group
description: >
Fields to log on behalf of a different program.
Fields provided by journald.
fields:
- name: audit
- name: object
type: group
description: >
Audit fields of event.
Fields to log on behalf of a different program.
fields:
- name: loginuid
- name: audit
type: group
description: >
Audit fields of event.
fields:
- name: loginuid
type: long
required: false
example: 1000
description: >
The login UID of the object process.
- name: session
type: long
required: false
example: 3
description: >
The audit session of the object process.
- name: cmd
type: keyword
required: false
example: "/lib/systemd/systemd --user"
description: >
The command line of the process.
- name: name
type: keyword
required: false
example: "/lib/systemd/systemd"
description: >
Name of the executable.
- name: executable
type: keyword
required: false
description: >
Path to the the executable.
example: "/lib/systemd/systemd"
- name: uid
type: long
required: false
example: 1000
description: >
The login UID of the object process.
- name: session
UID of the object process.
- name: gid
type: long
required: false
example: 3
description: >
The audit session of the object process.
- name: cmd
type: keyword
required: false
example: "/lib/systemd/systemd --user"
description: >
The command line of the process.
- name: name
type: keyword
required: false
example: "/lib/systemd/systemd"
description: >
Name of the executable.
- name: executable
type: keyword
required: false
description: >
Path to the the executable.
example: "/lib/systemd/systemd"
- name: uid
type: long
required: false
description: >
UID of the object process.
- name: gid
type: long
required: false
description: >
GID of the object process.
- name: pid
type: long
required: false
description: >
PID of the object process.
- name: systemd
GID of the object process.
- name: pid
type: long
required: false
description: >
PID of the object process.
- name: systemd
type: group
description: >
Systemd fields of event.
fields:
- name: owner_uid
type: long
required: false
description: >
The UID of the owner.
- name: session
type: keyword
required: false
description: >
The ID of the systemd session.
- name: unit
type: keyword
required: false
description: >
The name of the systemd unit.
- name: user_unit
type: keyword
required: false
description: >
The name of the systemd user unit.
- name: kernel
type: group
description: >
Systemd fields of event.
Fields to log on behalf of a different program.
fields:
- name: owner_uid
type: long
- name: device
type: keyword
required: false
description: >
The UID of the owner.
- name: session
The kernel device name.
- name: subsystem
type: keyword
required: false
description: >
The ID of the systemd session.
- name: unit
type: keyword
The kernel subsystem name.
- name: device_symlinks
type: text
required: false
description: >
The name of the systemd unit.
- name: user_unit
type: keyword
Additional symlink names pointing to the device node in /dev.
- name: device_node_path
type: text
required: false
description: >
The name of the systemd user unit.
- name: kernel
type: group
description: >
Fields to log on behalf of a different program.
fields:
- name: device
type: keyword
required: false
description: >
The kernel device name.
- name: subsystem
type: keyword
required: false
description: >
The kernel subsystem name.
- name: device_symlinks
type: text
required: false
description: >
Additional symlink names pointing to the device node in /dev.
- name: device_node_path
type: text
required: false
description: >
The device node path of this device in /dev.
- name: device_name
type: text
required: false
The device node path of this device in /dev.
- name: device_name
type: text
required: false
description: >
The kernel device name as it shows up in the device tree below /sys.
- name: code
type: group
description: >
The kernel device name as it shows up in the device tree below /sys.
- name: process
Fields of the code generating the event.
fields:
- name: file
type: text
required: false
example: "../src/core/manager.c"
description: >
The name of the source file where the log is generated.
- name: function
type: text
required: false
example: "job_log_status_message"
description: >
The name of the function which generated the log message.
- name: line
type: long
required: false
example: 123
description: >
The line number of the code which generated the log message.
- name: process
type: group
description: >
Fields to log on behalf of a different program.
Expand Down Expand Up @@ -184,6 +215,10 @@
example: 1
description: >
The ID of the user which runs the process.
- name: capabilites
required: false
description: >
The effective capabilites of the process.
- name: systemd
type: group
description: >
Expand Down Expand Up @@ -217,6 +252,11 @@
example: "user-1234.slice"
description: >
The systemd slice unit.
- name: user_slice
type: keyword
required: false
description: >
The systemd user slice unit.
- name: unit
type: keyword
required: false
Expand Down Expand Up @@ -246,29 +286,6 @@
example: "dd8c974asdf01dbe2ef26d7fasdf264c9"
description: >
The boot ID for the boot the log was generated in.
- name: code
type: group
description: >
Fields of the code generating the event.
fields:
- name: file
type: text
required: false
example: "../src/core/manager.c"
description: >
The name of the source file where the log is generated.
- name: function
type: text
required: false
example: "job_log_status_message"
description: >
The name of the function which generated the log message.
- name: line
type: long
required: false
example: 123
description: >
The line number of the code which generated the log message.
- name: syslog
type: group
description: >
Expand Down
6 changes: 6 additions & 0 deletions journalbeat/beater/journalbeat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,11 @@ import (
"time"

"github.com/elastic/beats/journalbeat/checkpoint"
"github.com/elastic/beats/journalbeat/cmd/instance"
"github.com/elastic/beats/journalbeat/input"
"github.com/elastic/beats/libbeat/beat"
"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/common/cfgwarn"
"github.com/elastic/beats/libbeat/logp"

"github.com/elastic/beats/journalbeat/config"
Expand All @@ -44,6 +46,8 @@ type Journalbeat struct {

// New returns a new Journalbeat instance
func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
cfgwarn.Experimental("Journalbeat is experimental.")

config := config.DefaultConfig
if err := cfg.Unpack(&config); err != nil {
return nil, fmt.Errorf("error reading config file: %v", err)
Expand All @@ -55,6 +59,8 @@ func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
return nil, err
}

instance.SetupJournalMetrics()

var inputs []*input.Input
for _, c := range config.Inputs {
i, err := input.New(c, b.Publisher, done, cp.States())
Expand Down
Loading