Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log.origin fields #563

Merged
merged 7 commits into from
Sep 19, 2019
Merged

Add log.origin fields #563

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
89c1ac8
Add log.source fields
felixbarny Sep 16, 2019
3d0c917
Update PR number in changelog
felixbarny Sep 16, 2019
60b1ee1
Rename source to origin
felixbarny Sep 17, 2019
ea47d83
Reuse existing file schema for log.origin
felixbarny Sep 18, 2019
75c32b4
Adjust changelog
felixbarny Sep 19, 2019
f2a19d3
Make fields extended
felixbarny Sep 19, 2019
1ec53dc
Merge remote-tracking branch 'origin/master' into log-source
felixbarny Sep 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Thanks, you're awesome :-) -->
### Bugfixes

### Added
* Added `log.source.file`, `log.source.function` and `log.source.line` fields. #563

### Improvements

Expand Down
9 changes: 9 additions & 0 deletions code/go/ecs/log.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 33 additions & 0 deletions docs/field-details.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2113,6 +2113,39 @@ example: `Sep 19 08:26:10 localhost My log`

// ===============================================================

| log.source.file
| The name of the source file which originated the log event.

type: keyword

example: `Bootstrap.java`

| core

// ===============================================================

| log.source.function
| The name of the function or method which originated the log event.

type: keyword

example: `init`

| core

// ===============================================================

| log.source.line
| The line number of the file which originated the log event.

type: integer

example: `42`

| core

// ===============================================================

|=====

[[ecs-network]]
Expand Down
17 changes: 17 additions & 0 deletions generated/beats/fields.ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1577,6 +1577,23 @@
This field is not indexed and doc_values are disabled so it can''t be queried
but the value can be retrieved from `_source`.'
example: Sep 19 08:26:10 localhost My log
- name: source.file
level: core
type: keyword
ignore_above: 1024
description: The name of the source file which originated the log event.
example: Bootstrap.java
- name: source.function
level: core
type: keyword
ignore_above: 1024
description: The name of the function or method which originated the log event.
example: init
- name: source.line
level: core
type: integer
description: The line number of the file which originated the log event.
example: 42
- name: network
title: Network
group: 2
Expand Down
3 changes: 3 additions & 0 deletions generated/csv/fields.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,9 @@ http.version,keyword,extended,1.1,1.2.0-dev
log.level,keyword,core,err,1.2.0-dev
log.logger,keyword,core,org.elasticsearch.bootstrap.Bootstrap,1.2.0-dev
log.original,keyword,core,Sep 19 08:26:10 localhost My log,1.2.0-dev
log.source.file,keyword,core,Bootstrap.java,1.2.0-dev
log.source.function,keyword,core,init,1.2.0-dev
log.source.line,integer,core,42,1.2.0-dev
network.application,keyword,extended,aim,1.2.0-dev
network.bytes,long,core,368,1.2.0-dev
network.community_id,keyword,extended,1:hO+sN4H+MG5MY/8hIrXPqc4ZQz0=,1.2.0-dev
Expand Down
29 changes: 29 additions & 0 deletions generated/ecs/ecs_flat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2237,6 +2237,35 @@ log.original:
order: 1
short: Original log message with light interpretation only (encoding, newlines).
type: keyword
log.source.file:
description: The name of the source file which originated the log event.
example: Bootstrap.java
flat_name: log.source.file
ignore_above: 1024
level: core
name: source.file
order: 3
short: The file which originated the log event.
type: keyword
log.source.function:
description: The name of the function or method which originated the log event.
example: init
flat_name: log.source.function
ignore_above: 1024
level: core
name: source.function
order: 4
short: The function which originated the log event.
type: keyword
log.source.line:
description: The line number of the file which originated the log event.
example: 42
flat_name: log.source.line
level: core
name: source.line
order: 5
short: The line number of the file which originated the log event.
type: integer
message:
description: 'For log events the message field contains the log message, optimized
for viewing in a log viewer.
Expand Down
29 changes: 29 additions & 0 deletions generated/ecs/ecs_nested.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2542,6 +2542,35 @@ log:
order: 1
short: Original log message with light interpretation only (encoding, newlines).
type: keyword
source.file:
description: The name of the source file which originated the log event.
example: Bootstrap.java
flat_name: log.source.file
ignore_above: 1024
level: core
name: source.file
order: 3
short: The file which originated the log event.
type: keyword
source.function:
description: The name of the function or method which originated the log event.
example: init
flat_name: log.source.function
ignore_above: 1024
level: core
name: source.function
order: 4
short: The function which originated the log event.
type: keyword
source.line:
description: The line number of the file which originated the log event.
example: 42
flat_name: log.source.line
level: core
name: source.line
order: 5
short: The line number of the file which originated the log event.
type: integer
group: 2
name: log
prefix: log.
Expand Down
15 changes: 15 additions & 0 deletions generated/elasticsearch/6/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -945,6 +945,21 @@
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"source": {
"properties": {
"file": {
"ignore_above": 1024,
"type": "keyword"
},
"function": {
"ignore_above": 1024,
"type": "keyword"
},
"line": {
"type": "integer"
}
}
}
}
},
Expand Down
15 changes: 15 additions & 0 deletions generated/elasticsearch/7/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -944,6 +944,21 @@
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"source": {
"properties": {
"file": {
"ignore_above": 1024,
"type": "keyword"
},
"function": {
"ignore_above": 1024,
"type": "keyword"
},
"line": {
"type": "integer"
}
}
}
}
},
Expand Down
15 changes: 15 additions & 0 deletions generated/legacy/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -643,6 +643,21 @@
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"source": {
"properties": {
"file": {
"ignore_above": 1024,
"type": "keyword"
},
"function": {
"ignore_above": 1024,
"type": "keyword"
},
"line": {
"type": "long"
}
}
}
}
},
Expand Down
30 changes: 30 additions & 0 deletions schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1534,6 +1534,36 @@
"name": "log.original",
"required": false,
"type": "(not indexed)"
},
"log.source.file": {
"description": "The name of the source file which originated the log event.",
"example": "Bootstrap.java",
"footnote": "",
"group": 2,
"level": "core",
"name": "log.source.file",
"required": false,
"type": "keyword"
},
"log.source.function": {
"description": "The name of the function or method which originated the log event.",
"example": "init",
"footnote": "",
"group": 2,
"level": "core",
"name": "log.source.function",
"required": false,
"type": "keyword"
},
"log.source.line": {
"description": "The line number of the file which originated the log event.",
"example": "42",
"footnote": "",
"group": 2,
"level": "core",
"name": "log.source.line",
"required": false,
"type": "integer"
}
},
"group": 2,
Expand Down
25 changes: 25 additions & 0 deletions schemas/log.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,28 @@
short: Name of the logger.
description: >
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.

- name: source.file
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry if what we call source here will conflict with what we have in source under https://github.com/elastic/ecs/blob/master/schemas/source.yml At the same time I think some argument could be it is similar?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the same time I think some argument could be it is similar?

I'd say it's actually not really similar to source. We could also rename to log.origin.* to avoid ambiguities.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on origin.

level: core
felixbarny marked this conversation as resolved.
Show resolved Hide resolved
type: keyword
example: Bootstrap.java
short: The file which originated the log event.
description: >
The name of the source file which originated the log event.

- name: source.function
level: core
type: keyword
example: init
short: The function which originated the log event.
description: >
The name of the function or method which originated the log event.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity: If it's a method, would we expect it to also indicate what class it belongs to? Something like Foo.bar?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it would just be the bar in this case. We could, however, add an optional origin.class for languages which support that concept.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, SGTM. Lets only add it if we already have a use case for it. But

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually do have a use case for logging. For java logging libraries, we usually get both the file name and the class name of the origin. See also https://docs.oracle.com/javase/7/docs/api/java/lang/StackTraceElement.html#getClassName().

I didn't add it at first because not all languages have the concept of a class. But when it's optional, why not?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I could name quite a few that have a similar concept ;-) In the case of Golang I would also use .class even though that is not 100% correct. @axw Hope you agree :-)

+1 on adding class directly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:) I'm fine with class as a name, but I would have made it a qualified name like Foo.bar as you described. @felixbarny what was the reason behind excluding the class name? Is it useful to search on them independently?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason why the class name should be included log.origin.function, am I missing something?

We can definitely have an additional field for the class (log.origin.class) which contains the fully qualified class name like org.elasticsearch.bootstrap.Bootstrap. But we currently don't have that field in the APM schema where we have module instead, which would translate to a Java package.

I'd like to defer these potentially controversial discussions and get the lowest common denominator in which, I think, is log.origin.file.name, log.origin.file.line and log.origin.function.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the description it mentions, it can be a function or method. So if it is a method, would it be wrong to put here Bootstrap.foo? What I'm getting at, it would be nice if the shortcut to this discussion is that both options are fine. Note: I left out the package name on purpose, was not even thinking of it before @felixbarny brought up the example.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would seem a bit weird to me as the method name is foo, not Bootstrap.foo. https://docs.oracle.com/javase/7/docs/api/java/lang/StackTraceElement.html#getMethodName() would also just return the method name.

What would be the benefit of concatenating the simple class name and the method name?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felixbarny I'm not too fussed if this is deferred. I don't like to sound like a broken record, but the use case I have in mind is once again profiling, and I was hoping we could find a common definition of fields.

I'd say the benefit of having them combined is having a simple display name, so that UIs don't need to know how to recombine the class and method name. e.g. it's "Class::Method" in C++.


- name: source.line
level: core
type: integer
example: 42
short: The line number of the file which originated the log event.
description: >
The line number of the file which originated the log event.