Manually prepare mysql package (alpha) #323

mtojek · 2020-04-01T11:14:29Z

I'm opening this as draft to present all manual changes that were applied.

Changes:

copied content of mysql-0.0.1
refactored vars according to: MySQL integration: additional var properties for UI #318
extended generator to use "alpha" packages

Action items:

Wait for fixing [Ingest] Fix stream generation from package kibana#62330
Sync with master
Wait for fixing [EPM] Support pipelines in both json and yaml format kibana#63251
Wait for fixing Internal server error: Compressor detection can only be called ... kibana#63726
Copy to docs real events in agent mode

Closes: #317
Closes: #318
Closes: #319
Closes: #320

Screenshot:

ruflin · 2020-04-01T12:34:17Z

dev/packages/alpha/mysql-0.0.2/manifest.yml

+  description: Collect logs and metrics from MySQL instances
+  inputs:
+  - type: logs
+    description: Collecting MySQL logs


I think it would be nice to also have a title for the logs and metrics input

ruflin · 2020-04-01T12:35:12Z

dev/packages/alpha/mysql-0.0.2/dataset/status/manifest.yml

+    default: 10s
+  - name: raw
+    type: bool
+    title: Raw (all fields from "SHOW /*!50002 GLOBAL */ STATUS;")


Should we put this into the description?

ruflin · 2020-04-01T12:35:54Z

dev/packages/alpha/mysql-0.0.2/dataset/slowlog/manifest.yml

+    default:
+    - /var/log/mysql/mysql-slow.log*
+    - /var/lib/mysql/{{.builtin.hostname}}-slow.log
+    os.darwin:


The platform flags will not work yet but please keep them in as we plan to support them later.

I adjusted all config properties with path variables.

ruflin · 2020-04-01T12:36:57Z

dev/packages/alpha/mysql-0.0.2/dataset/slowlog/manifest.yml

+    show_user: true
+    default:
+    - /var/log/mysql/mysql-slow.log*
+    - /var/lib/mysql/{{.builtin.hostname}}-slow.log


Not supported yet, but we plan to add it: elastic/beats#17340 As soon as we add it we need to adjust this the way we need it. Best keep it as is for now.

I adjusted all config properties with path variables.

mtojek · 2020-04-01T13:49:26Z

Also, screenshot updated.

mtojek · 2020-04-02T09:51:57Z

@ruflin

After fixing the issue with merging vars, I managed to install the package flawlessly.

health status index                                  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logs-mysql.error-default               PGiI_XKlRoumrhcBHRtldw   1   1         12            0     17.8kb         17.8kb
green  open   .apm-custom-link                       1hlCqV90QxSGfaahQos9_Q   1   0          0            0       208b           208b
yellow open   metrics-system.users-default           tVWFKAdyQgWxsvslCXJHIw   1   1       1510            0      1.1mb          1.1mb
green  open   .kibana_task_manager_1                 rotyxHtLTTC13l-e-IOpgA   1   0          5            7     32.3kb         32.3kb
yellow open   metrics-system.fsstat-default          NZBTRa11SKKO4kQWAb-XhQ   1   1        304            0      266kb          266kb
yellow open   metrics-system.raid-default            iH5Q93BPTZynPjE-PcgUlw   1   1       1242            0    986.8kb        986.8kb
yellow open   metrics-system.diskio-default          KcxO0FkBSk6ypXkHBDEdRQ   1   1       1244            0    980.2kb        980.2kb
yellow open   metrics-system.core-default            e0rlMsp1SHauqtNJbK69BA   1   1       1242            0   1008.8kb       1008.8kb
yellow open   metrics-system.network_summary-default byWDiHpVS9CqlROJIl8F7w   1   1       1764            0      1.2mb          1.2mb
yellow open   metrics-system.filesystem-default      wH30U3GlQ_q2THzXeUaITg   1   1        313            0    269.3kb        269.3kb
yellow open   metrics-system.cpu-default             4YAfxdPmRUqd_--J_UqVnA   1   1       1236            0    985.2kb        985.2kb
green  open   .apm-agent-configuration               JUeER1i-QSGOVp2_r4aNsg   1   0          0            0       208b           208b
yellow open   metrics-system.entropy-default         XeR7q0LARgu5nv7brBcsZw   1   1       1246            0   1020.1kb       1020.1kb
yellow open   metrics-system.service-default         iiyTJB2ASOmjP2UVJWdnOw   1   1       1252            0    965.2kb        965.2kb
yellow open   metrics-system.process_summary-default aLDlD2-NTzaX-1rFbjKfcg   1   1       1768            0      1.2mb          1.2mb
green  open   .kibana_1                              yxriaA3yRzy1kamcY_IQIg   1   0        375          207    509.5kb        509.5kb
yellow open   metrics-mysql.status-default           9fYhAPjER2eZwr0vyAhE7A   1   1         21            0     97.7kb         97.7kb
green  open   .security-7                            3O5VlAZwSlWxkF3bqPifAg   1   0         49            0    162.8kb        162.8kb
yellow open   metrics-system.memory-default          u74S8se-T8iD788aY0elgA   1   1       1776            0      1.2mb          1.2mb
yellow open   metrics-system.uptime-default          6AosWZ-kSTO_SVXO5KCzsg   1   1        306            0    289.3kb        289.3kb
yellow open   metrics-system.process-default         yiRugaNkQleo4R1thgj-PQ   1   1       1766            0      1.2mb          1.2mb
yellow open   metrics-system.network-default         ayaXEXp1Rb2VKfbl2w9KHQ   1   1       1243            0        1mb            1mb
yellow open   metrics-mysql.galera_status-default    4hay0weLQui4g4VqohoMCg   1   1         21            0     97.7kb         97.7kb
yellow open   metrics-system.load-default            UPR0fTo_QkK1V9lF8xo93A   1   1       1252            0      980kb          980kb
yellow open   metrics-system.socket_summary-default  SXAXQb8tRMKdHIWpzIexhA   1   1       1234            0    972.8kb        972.8kb

Dashboard:

ruflin · 2020-04-02T10:31:30Z

dev/packages/alpha/mysql-0.0.2/dataset/error/fields/fields.yml

+    - name: thread_id
+      type: alias
+      path: mysql.thread_id
+      migration: true


Should the migration field be removed?

Hmm, I'm not sure. If so, then probably that field should disappear at all from all entries?

Sorry, I meant here thread_id, the full block. I assume mysql.error.thread_id does not exist anymore.

Ok, so this change will affect fields.yml and exported fields in README. I will work on this then, but it may take some time.

mtojek · 2020-04-06T08:55:05Z

There was an issue with stream generation found last week: elastic/kibana#62330 .

Once it's fixed, I will re-verify the integration and re-generate events (leaving it here for visibility).

exekias · 2020-04-07T10:44:40Z

Hi @mtojek just checking in, is this still blocked? is there anything else pending to have MySQL package ready?

mtojek · 2020-04-07T10:48:45Z

Hi @mtojek just checking in, is this still blocked? is there anything else pending to have MySQL package ready?

Once [Ingest] Fix stream generation from package kibana#62330 is closed, I can reverify the integration and regenerate real events for docs purposes.
PR: Remove migrated not-ECS migrated fields #327 (you're marked also as reviewer).

In the mean time, I've prepared a tutorial how to build a package (#330).

mtojek · 2020-04-20T12:49:36Z

@ruflin I verified all bugfixes, merged changes from the master branch, installed the package in the testing environment. It works fine.

Switching to a ready PR.

ruflin · 2020-04-21T11:16:35Z

I look at this in the UI and created the config. The part I didn't test was if it ingests data after shipping it to the agent but the config looks good.

One thing I stumbled over is we should have status and galera status both enabled by default?

mtojek · 2020-04-21T11:31:28Z

One thing I stumbled over is we should have status and galera status both enabled by default?

I like this one. The dataset galera status has been disabled.

ruflin · 2020-04-21T11:48:03Z

Tried this locally and it seems it is still enabled in the UI. If you can reproduce it on your end, this sounds more like a bug in Kibana.

The other part I stumbled over is that I think quite a few fields in error are not documented.

But lets move forward here and get this and follow up with iterations.

Great job!

ruflin · 2020-04-21T11:48:22Z

Haha, already merged. Seems like we agree :-D

mtojek requested a review from ruflin April 1, 2020 11:14

mtojek self-assigned this Apr 1, 2020

ruflin reviewed Apr 1, 2020

View reviewed changes

mtojek added 5 commits April 1, 2020 15:05

Manually create mysql package (alpha)

dc7d138

Fix errors

b3f7222

Fix username, password

baa94e5

Update ingest pipeline

2d9c7c5

Address PR comments

b4ffdfb

mtojek force-pushed the package-mysql-0.0.2 branch from 335e9c8 to b4ffdfb Compare April 1, 2020 13:21

mtojek requested a review from ruflin April 1, 2020 13:49

ruflin reviewed Apr 2, 2020

View reviewed changes

mtojek mentioned this pull request Apr 2, 2020

Remove migrated not-ECS migrated fields #327

Merged

mtojek added 2 commits April 6, 2020 14:51

Merge branch 'master' into package-mysql-0.0.2

ba1101e

Fix: stream configuration

1278b08

mtojek added 11 commits April 7, 2020 15:34

Fix: indent

a080802

Merge branch 'master' into package-mysql-0.0.2

b8cc5f1

Update ingest pipeline

9e37828

Merge branch 'master' into package-mysql-0.0.2

a8f23a4

Refresh imported

a3b4bdf

Remove stale files

c2c91c4

Update package content

8e8bf18

Merge branch 'master' into package-mysql-0.0.2

2dc52f5

Merge branch 'master' into package-mysql-0.0.2

ce4bdf0

Resolve conflict

549b3d0

Fix latest issues

460d8a3

Update docs with logged events

c28c77c

mtojek requested a review from ruflin April 20, 2020 12:49

mtojek marked this pull request as ready for review April 20, 2020 12:49

ruflin approved these changes Apr 21, 2020

View reviewed changes

Disable galera_status by default

06867eb

mtojek merged commit 579d05d into elastic:master Apr 21, 2020

mtojek mentioned this pull request Feb 16, 2021

Adding package changelogs elastic/integrations#675

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manually prepare mysql package (alpha) #323

Manually prepare mysql package (alpha) #323

mtojek commented Apr 1, 2020 •

edited

Loading

ruflin Apr 1, 2020

mtojek Apr 1, 2020

ruflin Apr 1, 2020

mtojek Apr 1, 2020

ruflin Apr 1, 2020

mtojek Apr 1, 2020

ruflin Apr 1, 2020

mtojek Apr 1, 2020

mtojek commented Apr 1, 2020

mtojek commented Apr 2, 2020

ruflin Apr 2, 2020

mtojek Apr 2, 2020

ruflin Apr 2, 2020

mtojek Apr 2, 2020

mtojek Apr 2, 2020

mtojek commented Apr 6, 2020

exekias commented Apr 7, 2020

mtojek commented Apr 7, 2020 •

edited

Loading

mtojek commented Apr 20, 2020

ruflin commented Apr 21, 2020

mtojek commented Apr 21, 2020 •

edited

Loading

ruflin commented Apr 21, 2020

ruflin commented Apr 21, 2020

Manually prepare mysql package (alpha) #323

Manually prepare mysql package (alpha) #323

Conversation

mtojek commented Apr 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek commented Apr 1, 2020

mtojek commented Apr 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek commented Apr 6, 2020

exekias commented Apr 7, 2020

mtojek commented Apr 7, 2020 • edited Loading

mtojek commented Apr 20, 2020

ruflin commented Apr 21, 2020

mtojek commented Apr 21, 2020 • edited Loading

ruflin commented Apr 21, 2020

ruflin commented Apr 21, 2020

mtojek commented Apr 1, 2020 •

edited

Loading

mtojek commented Apr 7, 2020 •

edited

Loading

mtojek commented Apr 21, 2020 •

edited

Loading