-
Notifications
You must be signed in to change notification settings - Fork 458
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spring boot package [Memory - data stream] #2979
Merged
mtojek
merged 18 commits into
elastic:main
from
sunny-elastic:package_spring_boot_memory
Apr 25, 2022
Merged
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
3f34ad5
Spring boot package [Memory - data stream]
sunny-elastic 27cd6f4
Merge branch 'elastic:main' into package_spring_boot_memory
yug-rajani 742c82b
addressed review comments
sunny-elastic d91df9d
add descriptions and update CODEOWNERS
sunny-elastic 4f95dc2
Merge branch 'elastic:main' into package_spring_boot_memory
sunny-elastic f2a32a0
update manifest
sunny-elastic cf5263c
update manifest
sunny-elastic bc65d60
Merge branch 'elastic:main' into package_spring_boot_memory
sunny-elastic c09baff
resolved CI/CD issue
sunny-elastic 2cabd9d
update the nesting structure
sunny-elastic 8d6a0c4
resolve merge conflicts
sunny-elastic 0e5ca56
update manifest
sunny-elastic 4d53a3e
update stream.yml.hbs file
sunny-elastic 2669ed7
resolve CI/CD issue
sunny-elastic e4e2e3b
removed extra ecs fields
sunny-elastic b2675bb
update README.md
sunny-elastic a93de61
update ecs field
sunny-elastic 1d7c202
update description
sunny-elastic File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 changes: 6 additions & 0 deletions
6
packages/spring_boot/data_stream/memory/_dev/test/system/test-default-config.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| vars: | ||
| hosts: http://springboot:8090/actuator/jolokia | ||
| path: /jolokia/?ignoreErrors=true&canonicalNaming=false | ||
| input: jolokia/metrics | ||
| data_stream: | ||
| vars: ~ |
47 changes: 47 additions & 0 deletions
47
packages/spring_boot/data_stream/memory/agent/stream/stream.yml.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| metricsets: ["jmx"] | ||
| namespace: "metrics" | ||
| hosts: {{hosts}} | ||
| http_method: "GET" | ||
| path: {{path}} | ||
| username: {{username}} | ||
| password: {{password}} | ||
| period: {{period}} | ||
| {{#if ssl}} | ||
| ssl: {{ssl}} | ||
| {{/if}} | ||
| jmx.mappings: | ||
| - mbean: 'java.lang:type=Memory' | ||
| attributes: | ||
| - attr: HeapMemoryUsage | ||
| field: heap | ||
| - attr: NonHeapMemoryUsage | ||
| field: non_heap | ||
| - mbean: 'java.nio:name=direct,type=BufferPool' | ||
| attributes: | ||
| - attr: TotalCapacity | ||
| field: buffer_pool.direct.total_capacity | ||
| - attr: MemoryUsed | ||
| field: buffer_pool.direct.used | ||
| - attr: Count | ||
| field: buffer_pool.direct.count | ||
| - mbean: 'java.nio:name=mapped,type=BufferPool' | ||
| attributes: | ||
| - attr: TotalCapacity | ||
| field: buffer_pool.mapped.total_capacity | ||
| - attr: MemoryUsed | ||
| field: buffer_pool.mapped.used | ||
| - attr: Count | ||
| field: buffer_pool.mapped.count | ||
| - mbean: 'java.lang:name=CodeCacheManager,type=MemoryManager' | ||
| attributes: | ||
| - attr: Valid | ||
| field: manager.code_cache.valid | ||
| - attr: Name | ||
| field: manager.code_cache.name | ||
| - mbean: 'java.lang:name=Metaspace Manager,type=MemoryManager' | ||
| attributes: | ||
| - attr: Valid | ||
| field: manager.metaspace.valid | ||
| - attr: Name | ||
| field: manager.metaspace.name | ||
|
|
38 changes: 38 additions & 0 deletions
38
packages/spring_boot/data_stream/memory/elasticsearch/ingest_pipeline/default.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| --- | ||
| description: Pipeline for parsing Spring Boot Memory metrics. | ||
| processors: | ||
| - set: | ||
| field: ecs.version | ||
| value: "8.1.0" | ||
| - rename: | ||
| field: message | ||
| target_field: event.original | ||
| ignore_missing: true | ||
| - rename: | ||
| field: jolokia.metrics | ||
| target_field: spring_boot.memory | ||
| ignore_missing: true | ||
| ignore_failure: true | ||
| - set: | ||
| field: event.type | ||
| value: info | ||
| - set: | ||
| field: event.kind | ||
| value: metric | ||
| - set: | ||
| field: event.category | ||
| value: database | ||
| - set: | ||
| field: event.module | ||
| value: spring_boot | ||
| - set: | ||
| field: event.dataset | ||
| value: spring_boot.memory | ||
| - remove: | ||
| field: jolokia | ||
| ignore_missing: true | ||
| ignore_failure: true | ||
| on_failure: | ||
| - set: | ||
| field: error.message | ||
| value: "{{ _ingest.on_failure_message }}" |
12 changes: 12 additions & 0 deletions
12
packages/spring_boot/data_stream/memory/fields/base-fields.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| - name: data_stream.type | ||
| type: constant_keyword | ||
| description: Data stream type. | ||
| - name: data_stream.dataset | ||
| type: constant_keyword | ||
| description: Data stream dataset. | ||
| - name: data_stream.namespace | ||
| type: constant_keyword | ||
| description: Data stream namespace. | ||
| - name: '@timestamp' | ||
| type: date | ||
| description: Event timestamp. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| - external: ecs | ||
| name: ecs.version | ||
| - external: ecs | ||
| name: error.message | ||
| - external: ecs | ||
| name: event.category | ||
| - external: ecs | ||
| name: event.dataset | ||
| - external: ecs | ||
| name: event.kind | ||
| - external: ecs | ||
| name: event.module | ||
| - external: ecs | ||
| name: service.address | ||
| - external: ecs | ||
| name: service.type |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| - name: spring_boot | ||
| type: group | ||
| fields: | ||
| - name: memory | ||
| type: group | ||
| fields: | ||
| - name: buffer_pool | ||
| type: group | ||
| fields: | ||
| - name: direct | ||
| type: group | ||
| fields: | ||
| - name: count | ||
| type: long | ||
| description: Count of direct buffer pool memory | ||
| - name: used | ||
| type: long | ||
| description: Used memory of direct buffer pool | ||
| - name: total_capacity | ||
| type: long | ||
| description: Total capacity of direct buffer pool memory | ||
| - name: mapped | ||
| type: group | ||
| fields: | ||
| - name: count | ||
| type: long | ||
| description: Count of mapped buffer pool memory | ||
| - name: used | ||
| type: long | ||
| description: Used memory of mapped buffer pool | ||
| - name: total_capacity | ||
| type: long | ||
| description: Total capacity of mapped buffer pool memory | ||
| - name: heap | ||
| type: group | ||
| fields: | ||
| - name: committed | ||
| type: long | ||
| description: Committed heap memory usage of JVM | ||
| - name: init | ||
| type: long | ||
| description: Init heap memory usage of JVM | ||
| - name: max | ||
| type: long | ||
| description: Max heap memory usage of JVM | ||
| - name: used | ||
| type: long | ||
| description: Used heap memory usage of JVM | ||
| - name: non_heap | ||
| type: group | ||
| fields: | ||
| - name: committed | ||
| type: long | ||
| description: Committed non-heap memory usage of JVM | ||
| - name: init | ||
| type: long | ||
| description: Init non-heap memory usage of JVM | ||
| - name: max | ||
| type: long | ||
| description: Max non-heap memory usage of JVM | ||
| - name: used | ||
| type: long | ||
| description: Used non-heap memory usage of JVM | ||
| - name: manager | ||
| type: group | ||
| fields: | ||
| - name: code_cache | ||
| type: group | ||
| fields: | ||
| - name: name | ||
| type: keyword | ||
| description: Name of the cacheManager to qualify the cache | ||
| - name: valid | ||
| type: boolean | ||
| description: Validation of code cache | ||
| - name: metaspace | ||
| type: group | ||
| fields: | ||
| - name: name | ||
| type: keyword | ||
| description: Name of the Metaspace Manager to qualify the cache | ||
| - name: valid | ||
| type: boolean | ||
| description: Validation of metaspace manager |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| title: Memory Metrics | ||
| type: metrics | ||
| streams: | ||
| - input: jolokia/metrics | ||
| template_path: stream.yml.hbs | ||
| title: Memory metrics | ||
| description: Collect Spring Boot Memory metrics. | ||
| vars: | ||
| - name: period | ||
| type: text | ||
| title: Period | ||
| default: 60s | ||
| - name: processors | ||
| type: yaml | ||
| title: Processors | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| description: > | ||
| Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| { | ||
| "@timestamp": "2022-04-20T13:03:45.533Z", | ||
| "agent": { | ||
| "ephemeral_id": "c9cba3ad-ab82-4f17-95b1-a92257a086b0", | ||
| "id": "f7cd0ea7-4d35-4573-956a-021aa2718bbe", | ||
| "name": "docker-fleet-agent", | ||
| "type": "metricbeat", | ||
| "version": "8.1.0" | ||
| }, | ||
| "data_stream": { | ||
| "dataset": "spring_boot.memory", | ||
| "namespace": "ep", | ||
| "type": "metrics" | ||
| }, | ||
| "ecs": { | ||
| "version": "8.1.0" | ||
| }, | ||
| "elastic_agent": { | ||
| "id": "f7cd0ea7-4d35-4573-956a-021aa2718bbe", | ||
| "snapshot": false, | ||
| "version": "8.1.0" | ||
| }, | ||
| "event": { | ||
| "agent_id_status": "verified", | ||
| "category": "database", | ||
| "dataset": "spring_boot.memory", | ||
| "duration": 498219023, | ||
| "ingested": "2022-04-20T13:03:48Z", | ||
| "kind": "metric", | ||
| "module": "spring_boot", | ||
| "type": "info" | ||
| }, | ||
| "host": { | ||
| "architecture": "x86_64", | ||
| "containerized": true, | ||
| "hostname": "docker-fleet-agent", | ||
| "ip": [ | ||
| "192.168.144.4" | ||
| ], | ||
| "mac": [ | ||
| "02:42:c0:a8:90:04" | ||
| ], | ||
| "name": "docker-fleet-agent", | ||
| "os": { | ||
| "codename": "focal", | ||
| "family": "debian", | ||
| "kernel": "3.10.0-1160.59.1.el7.x86_64", | ||
| "name": "Ubuntu", | ||
| "platform": "ubuntu", | ||
| "type": "linux", | ||
| "version": "20.04.3 LTS (Focal Fossa)" | ||
| } | ||
| }, | ||
| "metricset": { | ||
| "name": "jmx", | ||
| "period": 60000 | ||
| }, | ||
| "service": { | ||
| "address": "http://springboot:8090/actuator/jolokia", | ||
| "type": "jolokia" | ||
| }, | ||
| "spring_boot": { | ||
| "memory": { | ||
| "heap": { | ||
| "committed": 303038464, | ||
| "init": 96468992, | ||
| "max": 1350041600, | ||
| "used": 135078232 | ||
| }, | ||
| "non_heap": { | ||
| "committed": 60882944, | ||
| "init": 2555904, | ||
| "max": -1, | ||
| "used": 55917256 | ||
| } | ||
| } | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you dump an event that collects most of defined metrics?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated.