-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Elasticsearch json logging #36833
Merged
pgomulka
merged 75 commits into
elastic:master
from
pgomulka:feature/logging-structured
Jan 29, 2019
Merged
Elasticsearch json logging #36833
Changes from 72 commits
Commits
Show all changes
75 commits
Select commit
Hold shift + click to select a range
3ad2a57
NodeId available on all log lines
pgomulka 3ea4695
revert back changes due to use of json layout
pgomulka 76b9a45
Merge branch 'master' into feature/logging-structured
pgomulka 5593c46
remove unused improts
pgomulka aeb686e
cleanup in plugin
pgomulka fe87cc5
adding new option with clustestatelistener, pattern converter and thr…
pgomulka 73ce302
exception handling
pgomulka 3f838c7
json message and exception works
pgomulka e699d27
common wrapper layout class
pgomulka af54281
removed debugging code
pgomulka 3bb69eb
passing dignity test
pgomulka 5e1f125
fixing build
pgomulka 18908ed
Merge branch 'master' into feature/logging-structured
pgomulka 2c2653c
fix failing test
pgomulka 97d2cec
fix failing test
pgomulka fb2f531
fix import
pgomulka 20ee653
extending logs test
pgomulka 55dc6eb
fix parsing and exception formatting
pgomulka b7ad650
fix failing test
pgomulka 39a1ef7
fix checkstyle
pgomulka 372207f
Merge branch 'master' into feature/logging-structured
pgomulka dec2024
small cleanup
pgomulka 1da7c97
json logs cleanup
pgomulka 4acd89d
Merge branch 'master' into feature/logging-structured
pgomulka 28c20c1
test cleanup
pgomulka 0686fee
sometimes HttpServerTransport is logging first, and then the server d…
pgomulka 669e9ec
additional json tests
pgomulka de17fc1
docker log4j config cleanup
pgomulka 147ca9c
incorrect docker appender ref
pgomulka 7a2b537
the right order of reading values from clusterListener
pgomulka 6a01097
add missing marker in a pattern
pgomulka 0af53c0
empty lines cleanup
pgomulka 4f8cdae
Merge branch 'master' into feature/logging-structured
pgomulka 4aa84d7
addressing Nik's comments
pgomulka 1d0d66a
follow up after Daniel's comments
pgomulka 0e84d02
failing test
pgomulka 490b56d
unused imports
pgomulka 1119f5e
failing tests
pgomulka 66b1420
rename test log name
pgomulka 1f91bad
method rename
pgomulka 7bde657
Merge branch 'master' into feature/logging-structured
pgomulka bcf5f85
rename name to server
pgomulka c7bd58a
rename revert and level corrected
pgomulka 66c1942
wrong assertion
pgomulka b84cf9a
rename log name files in package tests
pgomulka 12677cf
addressing Daniels' second round of comments
pgomulka 5d78edf
javadocs
pgomulka 4cbea2b
additional test verifing old config
pgomulka f780f74
unused import
pgomulka 8c3c766
empty unused test
pgomulka c3ebfc0
small fixes after review
pgomulka 18aca44
comment cleanup after review
pgomulka a4d9336
documentation and licence fix
pgomulka 6bc7d1c
typo printted -> printed
pgomulka 7c208c8
setOnce argument ordering
pgomulka a6e81fa
javadoc typo
pgomulka f01a4ff
refactor cluster state listeners to use setOnce
pgomulka 53ead59
removed empty line
pgomulka 88d1368
methods rename and cleanup
pgomulka 72bd776
javadoc typo
pgomulka ee3322c
Merge pull request #7 from pgomulka/fix/observer-logging
pgomulka c1a4206
keep the old appenders and let the nodeIDlistener start earlier
pgomulka cbe83bc
Merge branch 'master' into feature/logging-structured
pgomulka b12f8ee
improved documentation and more robust test
pgomulka 254d32b
Merge branch 'master' into feature/logging-structured
pgomulka 1951e2a
split logging config in 2 for docs
pgomulka 1502105
enable log print out for this test
pgomulka 7eaaada
rename logs to .json
pgomulka 1291ead
migration logging
pgomulka 30d9675
old log rename and documentation update
pgomulka faa81fd
making test more stable
pgomulka 5503ad5
doc changes after review
pgomulka b563423
fix doc
pgomulka 42481b3
Merge branch 'master' into feature/logging-structured
pgomulka 8aa18ec
Merge branch 'master' into feature/logging-structured
pgomulka File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,9 +1,43 @@ | ||
status = error | ||
|
||
appender.console.type = Console | ||
appender.console.name = console | ||
appender.console.layout.type = PatternLayout | ||
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n | ||
# log action execution errors for easier debugging | ||
logger.action.name = org.elasticsearch.action | ||
logger.action.level = debug | ||
|
||
appender.rolling.type = Console | ||
appender.rolling.name = rolling | ||
appender.rolling.layout.type = ESJsonLayout | ||
appender.rolling.layout.type_name = server | ||
|
||
rootLogger.level = info | ||
rootLogger.appenderRef.console.ref = console | ||
rootLogger.appenderRef.rolling.ref = rolling | ||
|
||
appender.deprecation_rolling.type = Console | ||
appender.deprecation_rolling.name = deprecation_rolling | ||
appender.deprecation_rolling.layout.type = ESJsonLayout | ||
appender.deprecation_rolling.layout.type_name = deprecation | ||
|
||
logger.deprecation.name = org.elasticsearch.deprecation | ||
logger.deprecation.level = warn | ||
logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling | ||
logger.deprecation.additivity = false | ||
|
||
appender.index_search_slowlog_rolling.type = Console | ||
appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling | ||
appender.index_search_slowlog_rolling.layout.type = ESJsonLayout | ||
appender.index_search_slowlog_rolling.layout.type_name = index_search_slowlog | ||
|
||
logger.index_search_slowlog_rolling.name = index.search.slowlog | ||
logger.index_search_slowlog_rolling.level = trace | ||
logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling | ||
logger.index_search_slowlog_rolling.additivity = false | ||
|
||
appender.index_indexing_slowlog_rolling.type = Console | ||
appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling | ||
appender.index_indexing_slowlog_rolling.layout.type = ESJsonLayout | ||
appender.index_indexing_slowlog_rolling.layout.type_name = index_indexing_slowlog | ||
|
||
logger.index_indexing_slowlog.name = index.indexing.slowlog.index | ||
logger.index_indexing_slowlog.level = trace | ||
logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling | ||
logger.index_indexing_slowlog.additivity = false |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
[float] | ||
[[breaking_70_logging_changes]] | ||
=== Logging changes | ||
|
||
[float] | ||
==== New JSON format log files in `log` directory | ||
|
||
Elasticsearch now will produce additional log files in JSON format. They will be stored in `*.json` suffix files. | ||
Following files should be expected now in log directory: | ||
* ${cluster_name}_server.json | ||
* ${cluster_name}_deprecation.json | ||
* ${cluster_name}_index_search_slowlog.json | ||
* ${cluster_name}_index_indexing_slowlog.json | ||
* ${cluster_name}.log | ||
* ${cluster_name}_deprecation.log | ||
* ${cluster_name}_index_search_slowlog.log | ||
* ${cluster_name}_index_indexing_slowlog.log | ||
* ${cluster_name}_audit.json | ||
* gc.log | ||
|
||
Note: You can configure which of these files are written by editing `log4j2.properties`. | ||
|
||
[float] | ||
==== Log files ending with `*.log` deprecated | ||
Log files with the `.log` file extension using the old pattern layout format | ||
are now considered deprecated and the newly added JSON log file format with | ||
the .json file extension should be used instead. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: formatting: .json should be enclosed in backticks. |
||
Note: GC logs which are written to the file gc.log will not be changed. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: formatting: gc.log should be enclosed in backticks? |
||
|
||
[float] | ||
==== Docker output in JSON format | ||
|
||
All Docker console logs are now in JSON format. You can distinguish logs streams with the `type` field. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd say it is ok that we keep the old format as is but now the new format contains different information (e.g. the cluster UUID is only present in the JSON format).