Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log.origin fields #563

Merged
merged 7 commits into from
Sep 19, 2019
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Thanks, you're awesome :-) -->
### Bugfixes

### Added
* Added `log.origin.file`, `log.origin.function` and `log.origin.line` fields. #563
felixbarny marked this conversation as resolved.
Show resolved Hide resolved

### Improvements

Expand Down
9 changes: 9 additions & 0 deletions code/go/ecs/log.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 33 additions & 0 deletions docs/field-details.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2098,6 +2098,39 @@ example: `org.elasticsearch.bootstrap.Bootstrap`

// ===============================================================

| log.origin.file
| The name of the source file which originated the log event.

type: keyword

example: `Bootstrap.java`

| core

// ===============================================================

| log.origin.function
| The name of the function or method which originated the log event.

type: keyword

example: `init`

| core

// ===============================================================

| log.origin.line
| The line number of the file which originated the log event.

type: integer

example: `42`

| core

// ===============================================================

| log.original
| This is the original log message and contains the full log message before splitting it up in multiple parts.

Expand Down
17 changes: 17 additions & 0 deletions generated/beats/fields.ecs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1562,6 +1562,23 @@
description: The name of the logger inside an application. This is usually the
name of the class which initialized the logger, or can be a custom name.
example: org.elasticsearch.bootstrap.Bootstrap
- name: origin.file
level: core
type: keyword
ignore_above: 1024
description: The name of the source file which originated the log event.
example: Bootstrap.java
- name: origin.function
level: core
type: keyword
ignore_above: 1024
description: The name of the function or method which originated the log event.
example: init
- name: origin.line
level: core
type: integer
description: The line number of the file which originated the log event.
example: 42
- name: original
level: core
type: keyword
Expand Down
3 changes: 3 additions & 0 deletions generated/csv/fields.csv
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,9 @@ http.response.status_code,long,extended,404,1.2.0-dev
http.version,keyword,extended,1.1,1.2.0-dev
log.level,keyword,core,err,1.2.0-dev
log.logger,keyword,core,org.elasticsearch.bootstrap.Bootstrap,1.2.0-dev
log.origin.file,keyword,core,Bootstrap.java,1.2.0-dev
log.origin.function,keyword,core,init,1.2.0-dev
log.origin.line,integer,core,42,1.2.0-dev
log.original,keyword,core,Sep 19 08:26:10 localhost My log,1.2.0-dev
network.application,keyword,extended,aim,1.2.0-dev
network.bytes,long,core,368,1.2.0-dev
Expand Down
29 changes: 29 additions & 0 deletions generated/ecs/ecs_flat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2216,6 +2216,35 @@ log.logger:
order: 2
short: Name of the logger.
type: keyword
log.origin.file:
description: The name of the source file which originated the log event.
example: Bootstrap.java
flat_name: log.origin.file
ignore_above: 1024
level: core
name: origin.file
order: 3
short: The file which originated the log event.
type: keyword
log.origin.function:
description: The name of the function or method which originated the log event.
example: init
flat_name: log.origin.function
ignore_above: 1024
level: core
name: origin.function
order: 4
short: The function which originated the log event.
type: keyword
log.origin.line:
description: The line number of the file which originated the log event.
example: 42
flat_name: log.origin.line
level: core
name: origin.line
order: 5
short: The line number of the file which originated the log event.
type: integer
log.original:
description: 'This is the original log message and contains the full log message
before splitting it up in multiple parts.
Expand Down
29 changes: 29 additions & 0 deletions generated/ecs/ecs_nested.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2521,6 +2521,35 @@ log:
order: 2
short: Name of the logger.
type: keyword
origin.file:
description: The name of the source file which originated the log event.
example: Bootstrap.java
flat_name: log.origin.file
ignore_above: 1024
level: core
name: origin.file
order: 3
short: The file which originated the log event.
type: keyword
origin.function:
description: The name of the function or method which originated the log event.
example: init
flat_name: log.origin.function
ignore_above: 1024
level: core
name: origin.function
order: 4
short: The function which originated the log event.
type: keyword
origin.line:
description: The line number of the file which originated the log event.
example: 42
flat_name: log.origin.line
level: core
name: origin.line
order: 5
short: The line number of the file which originated the log event.
type: integer
original:
description: 'This is the original log message and contains the full log message
before splitting it up in multiple parts.
Expand Down
15 changes: 15 additions & 0 deletions generated/elasticsearch/6/template.json
Original file line number Diff line number Diff line change
Expand Up @@ -940,6 +940,21 @@
"ignore_above": 1024,
"type": "keyword"
},
"origin": {
"properties": {
"file": {
"ignore_above": 1024,
"type": "keyword"
},
"function": {
"ignore_above": 1024,
"type": "keyword"
},
"line": {
"type": "integer"
}
}
},
"original": {
"doc_values": false,
"ignore_above": 1024,
Expand Down
15 changes: 15 additions & 0 deletions generated/elasticsearch/7/template.json
Original file line number Diff line number Diff line change
Expand Up @@ -939,6 +939,21 @@
"ignore_above": 1024,
"type": "keyword"
},
"origin": {
"properties": {
"file": {
"ignore_above": 1024,
"type": "keyword"
},
"function": {
"ignore_above": 1024,
"type": "keyword"
},
"line": {
"type": "integer"
}
}
},
"original": {
"doc_values": false,
"ignore_above": 1024,
Expand Down
15 changes: 15 additions & 0 deletions generated/legacy/template.json
Original file line number Diff line number Diff line change
Expand Up @@ -638,6 +638,21 @@
"ignore_above": 1024,
"type": "keyword"
},
"origin": {
"properties": {
"file": {
"ignore_above": 1024,
"type": "keyword"
},
"function": {
"ignore_above": 1024,
"type": "keyword"
},
"line": {
"type": "long"
}
}
},
"original": {
"doc_values": false,
"ignore_above": 1024,
Expand Down
30 changes: 30 additions & 0 deletions schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -1525,6 +1525,36 @@
"required": false,
"type": "keyword"
},
"log.origin.file": {
"description": "The name of the source file which originated the log event.",
"example": "Bootstrap.java",
"footnote": "",
"group": 2,
"level": "core",
"name": "log.origin.file",
"required": false,
"type": "keyword"
},
"log.origin.function": {
"description": "The name of the function or method which originated the log event.",
"example": "init",
"footnote": "",
"group": 2,
"level": "core",
"name": "log.origin.function",
"required": false,
"type": "keyword"
},
"log.origin.line": {
"description": "The line number of the file which originated the log event.",
"example": "42",
"footnote": "",
"group": 2,
"level": "core",
"name": "log.origin.line",
"required": false,
"type": "integer"
},
"log.original": {
"description": "This is the original log message and contains the full log message before splitting it up in multiple parts.\nIn contrast to the `message` field which can contain an extracted part of the log message, this field contains the original, full log message. It can have already some modifications applied like encoding or new lines removed to clean up the log message.\nThis field is not indexed and doc_values are disabled so it can't be queried but the value can be retrieved from `_source`.",
"example": "Sep 19 08:26:10 localhost My log",
Expand Down
25 changes: 25 additions & 0 deletions schemas/log.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,28 @@
short: Name of the logger.
description: >
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.

- name: origin.file
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's reuse the file object here: https://github.com/elastic/ecs/blob/master/schemas/file.yml#L15

So this becomes log.origin.file.path I guess?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call! It would be log.origin.file.name though. Should we then also do log.origin.file.line instead of log.origin.line?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would make sense I guess. I wonder if we "must" extend file for this or we just use it here? I'm good with just adding it here but I'm sure at @webmat will have an opinion on this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with directly adding 2-3 specific fields for now 👍

We're going for something like this for package.* as well

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Using the file.* object here would require a change to ECS, since it is not currently defined as a reusable (nestable) object. I'm not saying this is wrong, as the idea has come up before to capture events like file-copy events, but we should consider this as we work through this PR, addressing where else file.* could be nested. For now, I would suggest we don't re-use the file.* object here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not using the full file object here but re-uses the same structure. Coming up with slightly different names just not to re-use the file object structure doesn't feel quite right. But I think I don't quite understand what you have in mind here. Which names would you suggest?

level: core
felixbarny marked this conversation as resolved.
Show resolved Hide resolved
type: keyword
example: Bootstrap.java
short: The file which originated the log event.
description: >
The name of the source file which originated the log event.

- name: origin.function
level: core
type: keyword
example: init
short: The function which originated the log event.
description: >
The name of the function or method which originated the log event.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity: If it's a method, would we expect it to also indicate what class it belongs to? Something like Foo.bar?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it would just be the bar in this case. We could, however, add an optional origin.class for languages which support that concept.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, SGTM. Lets only add it if we already have a use case for it. But

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually do have a use case for logging. For java logging libraries, we usually get both the file name and the class name of the origin. See also https://docs.oracle.com/javase/7/docs/api/java/lang/StackTraceElement.html#getClassName().

I didn't add it at first because not all languages have the concept of a class. But when it's optional, why not?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I could name quite a few that have a similar concept ;-) In the case of Golang I would also use .class even though that is not 100% correct. @axw Hope you agree :-)

+1 on adding class directly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:) I'm fine with class as a name, but I would have made it a qualified name like Foo.bar as you described. @felixbarny what was the reason behind excluding the class name? Is it useful to search on them independently?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason why the class name should be included log.origin.function, am I missing something?

We can definitely have an additional field for the class (log.origin.class) which contains the fully qualified class name like org.elasticsearch.bootstrap.Bootstrap. But we currently don't have that field in the APM schema where we have module instead, which would translate to a Java package.

I'd like to defer these potentially controversial discussions and get the lowest common denominator in which, I think, is log.origin.file.name, log.origin.file.line and log.origin.function.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the description it mentions, it can be a function or method. So if it is a method, would it be wrong to put here Bootstrap.foo? What I'm getting at, it would be nice if the shortcut to this discussion is that both options are fine. Note: I left out the package name on purpose, was not even thinking of it before @felixbarny brought up the example.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would seem a bit weird to me as the method name is foo, not Bootstrap.foo. https://docs.oracle.com/javase/7/docs/api/java/lang/StackTraceElement.html#getMethodName() would also just return the method name.

What would be the benefit of concatenating the simple class name and the method name?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felixbarny I'm not too fussed if this is deferred. I don't like to sound like a broken record, but the use case I have in mind is once again profiling, and I was hoping we could find a common definition of fields.

I'd say the benefit of having them combined is having a simple display name, so that UIs don't need to know how to recombine the class and method name. e.g. it's "Class::Method" in C++.


- name: origin.line
level: core
type: integer
example: 42
short: The line number of the file which originated the log event.
description: >
The line number of the file which originated the log event.