Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manually prepare mysql package (alpha) #323

Merged
merged 20 commits into from
Apr 21, 2020
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
input: log
paths:
{{#each paths}}
- {{this}}
{{/each}}
exclude_files: [".gz$"]
multiline:
# Consider lines without timestamp part of the previous message
pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}|[0-9]{6})'
negate: true
match: after
processors:
- add_locale: ~
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"description": "Pipeline for parsing MySQL error logs",
"processors": [{
"grok": {
"field": "message",
"patterns": [
"%{MYSQLDATETIME}%{SPACE}(%{NUMBER:mysql.thread_id:long}%{SPACE})?(\\[%{DATA:log.level}\\]%{SPACE})?%{GREEDYMULTILINE:message}",
"%{GREEDYDATA:message}"
],
"ignore_missing": true,
"pattern_definitions": {
"LOCALDATETIME": "(?:%{YEAR}-%{MONTHNUM}-%{MONTHDAY}|%{NUMBER})%{SPACE}%{TIME}",
"MYSQLDATETIME": "(?:%{LOCALDATETIME:_tmp.local_timestamp}|%{TIMESTAMP_ISO8601:_tmp.timestamp})",
"GREEDYMULTILINE": "(.|\n)+"
}
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "event.created"
}
}, {
"date": {
"if": "ctx._tmp?.local_timestamp != null && ctx.event?.timezone == null",
"field": "_tmp.local_timestamp",
"formats": [
"yyMMdd H:m:s",
"yyMMdd H:m:s",
"yyyy-MM-dd H:m:s",
"yyyy-MM-dd H:m:s"
]
}
}, {
"date": {
"if": "ctx._tmp?.local_timestamp != null && ctx.event?.timezone != null",
"field": "_tmp.local_timestamp",
"timezone": "{{ event.timezone }}",
"formats": [
"yyMMdd H:m:s",
"yyMMdd H:m:s",
"yyyy-MM-dd H:m:s",
"yyyy-MM-dd H:m:s"
]
}
}, {
"date": {
"if": "ctx._tmp?.timestamp != null",
"field": "_tmp.timestamp",
"formats": [
"ISO8601"
]
}
}, {
"remove":{
"field": "_tmp",
"ignore_missing": true
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
17 changes: 17 additions & 0 deletions dev/packages/alpha/mysql-0.0.2/dataset/error/fields/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
- name: mysql.error
type: group
description: >
Contains fields from the MySQL error logs.
fields:
- name: thread_id
type: alias
path: mysql.thread_id
migration: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the migration field be removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm not sure. If so, then probably that field should disappear at all from all entries?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I meant here thread_id, the full block. I assume mysql.error.thread_id does not exist anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so this change will affect fields.yml and exported fields in README. I will work on this then, but it may take some time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- name: level
type: alias
path: log.level
migration: true
- name: message
type: alias
path: message
migration: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
- name: mysql
type: group
description: >
Fields from the MySQL log files.
fields:
- name: thread_id
type: long
description: >
The connection or thread ID for the query.
23 changes: 23 additions & 0 deletions dev/packages/alpha/mysql-0.0.2/dataset/error/manifest.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
title: MySQL error logs
release: ga
type: logs
streams:
- input: logs
vars:
- name: paths
type: text
title: Error log paths
multi: true
required: true
show_user: true
default:
- /var/log/mysql/error.log*
- /var/log/mysqld.log*
os.darwin:
default:
- /usr/local/var/mysql/localhost.localdomain.err*
os.windows:
default:
- c:/programdata/MySQL/MySQL Server*/error.log*
title: MySQL error logs
description: Collect MySQL error logs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
input: mysql/metrics
metricsets: ["galera_status"]
hosts: {{hosts}}
{{#if password}}
password: {{password}}
{{/if}}
period: {{period}}
{{#if raw}}
raw: {{raw}}
{{/if}}
{{#if username}}
username: {{username}}
{{/if}}
Loading