Sidecar support for tags and collectors with multiple configurations #13433
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gally47
changed the title
…#13349) * Add support for multiple configurations per collector This will need a new sidcar release. Old sidecars are still supported with just a single configuration. * Use Java 9 Lists * Add support for ${sidecar.spoolDir} variable * Fix tests * Revert "Add support for multiple configurations per collector" This reverts commit 0a17743. * Use ${sidecar.spoolDir} for fresh collector templates And add explanation to the UI * make eslint happy * convert tags to Set and fix Autovalue * Support configuration_id in CollectorStatus Co-authored-by: Othello Maurer <[email protected]>
* add assignedFromTags property to assignment * Don't fail for unknown properties on NodeDetails This makes it easier to extend the sidecar in the future * Create sidecar config assignments based on tags This is a first naive approach. Every registration request will rebuild the assignments and this is probably too expensive. * fix test * Fix more Tests * Fine grained EtagService Use three different caches for configs, collectors and assignments. * Add Etag caching for UpdateRegistration results The response to the registration PUT request contains configuration assignments, which can be cached as well. This requires us to invalidate the sidecar etag cache in more situations: - Configration assignments - User triggered collector actions (stop, start, restart) * Cleanup old assignment caching commit * optimize * fix after merge * replace deprecated notempty annotation * cache assignments per sidecar id using just the md5 of an assignment result might leed to sidecars missing out on updates. An assignment is always meant for a single sidecar, caching the entire assignment over all sidecars is wrong. A per sidecar cache entry allows more fine grained cache invalidations. Bump the cacheMaxSize to 5000 entries, which is a more meaningful number, considered that this might have to hold all sidecars. * Refactor sidecar register() call. Only update the tagged assignments when we miss the cache. * change cache naming * more refactoring * improve cache invalidation * use fancy java Co-authored-by: Othello Maurer <[email protected]> * Add more cases for tag invalidation and fix tests * add tags to Config summary * Generate EntityTags with a better hash algorithm As @thll noticed, Object.hashCode() is prone to having collisions * Handle manual assignments via API * rename * Make tags available as config variables. This allows configs to be written with conditionals. E.g.: ``` <#if sidecar.tags.apache??> - /var/log/apache/*.log </#if> ``` * dont lose tags on UI configuration updates Co-authored-by: Othello Maurer <[email protected]>
* Only assign tagged configs that match the OS * rename test * add tests for SidecarService.updateTaggedConfigurationAssignments() * add tests for SidecarService.applyManualAssignments()
* Fix cache invalidation for collector actions The registration cache was using the sidecar "_id", while the ActionService was invalidating the sidecar "node_id". Change the entire cache to using the "node_id" * Rename parameter from sidecarId to sidecarNodeId Co-authored-by: Othello Maurer <[email protected]>
* Invalidate registration cache only for affected sidecars * Use node-id for cache invalidation The cache key has been changed since this PR has been started. * Consider OS for cache invalidation Also use symmetric difference of tags to further reduce unnecessary invalidations * Invalidate less when creating new configs
gally47
force-pushed
the
sidecars-ux-13362
branch
from
99bd9c2 to
4555f99
Compare
…#13349) * Add support for multiple configurations per collector This will need a new sidcar release. Old sidecars are still supported with just a single configuration. * Use Java 9 Lists * Add support for ${sidecar.spoolDir} variable * Fix tests * Revert "Add support for multiple configurations per collector" This reverts commit 0a17743. * Use ${sidecar.spoolDir} for fresh collector templates And add explanation to the UI * make eslint happy * convert tags to Set and fix Autovalue * Support configuration_id in CollectorStatus Co-authored-by: Othello Maurer <[email protected]>
* add assignedFromTags property to assignment * Don't fail for unknown properties on NodeDetails This makes it easier to extend the sidecar in the future * Create sidecar config assignments based on tags This is a first naive approach. Every registration request will rebuild the assignments and this is probably too expensive. * fix test * Fix more Tests * Fine grained EtagService Use three different caches for configs, collectors and assignments. * Add Etag caching for UpdateRegistration results The response to the registration PUT request contains configuration assignments, which can be cached as well. This requires us to invalidate the sidecar etag cache in more situations: - Configration assignments - User triggered collector actions (stop, start, restart) * Cleanup old assignment caching commit * optimize * fix after merge * replace deprecated notempty annotation * cache assignments per sidecar id using just the md5 of an assignment result might leed to sidecars missing out on updates. An assignment is always meant for a single sidecar, caching the entire assignment over all sidecars is wrong. A per sidecar cache entry allows more fine grained cache invalidations. Bump the cacheMaxSize to 5000 entries, which is a more meaningful number, considered that this might have to hold all sidecars. * Refactor sidecar register() call. Only update the tagged assignments when we miss the cache. * change cache naming * more refactoring * improve cache invalidation * use fancy java Co-authored-by: Othello Maurer <[email protected]> * Add more cases for tag invalidation and fix tests * add tags to Config summary * Generate EntityTags with a better hash algorithm As @thll noticed, Object.hashCode() is prone to having collisions * Handle manual assignments via API * rename * Make tags available as config variables. This allows configs to be written with conditionals. E.g.: ``` <#if sidecar.tags.apache??> - /var/log/apache/*.log </#if> ``` * dont lose tags on UI configuration updates Co-authored-by: Othello Maurer <[email protected]>
ousmaneo
requested changes
Oct 18, 2022
graylog2-web-interface/src/components/sidecars/administration/CollectorsAdministration.css
Outdated
Show resolved
Hide resolved
graylog2-web-interface/src/components/sidecars/common/ColorLabel.tsx
Outdated
Show resolved
Hide resolved
ousmaneo
approved these changes
Oct 19, 2022
mpfz0r
changed the title
mpfz0r
approved these changes
Oct 19, 2022
|
Hey guys, great to see this issue is merged; when can we expect an update for the graylog-sidecar binaries ? days ? weeks? thanks ! |
|
@gael-donat we are planning to release a 5.0 beta / Sidecar 1.3 beta next week. |
|
Ok thanks. Since the LDAP group mapping is not a free feature anymore, we still are on graylog3. I assume the configuration will not be available if we stay on graylog3 ? |
|
Yeah, sorry. that's not gonna happen. We are not even actively maintaining 3.x anymore. (Patching Log4Shell was an exception) |
|
that's understandable. Shame we lost the tag feature switch from collector to sidecar tho.. but great to see it added again. |
This was referenced Jan 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds two features to the Sidecar:
Both features need Sidecar 1.3.x or later.
Older sidecars are still supported, but will only pick up the first assigned configuration.
For implementation details see https://github.com/Graylog2/graylog-plugin-enterprise/issues/3956
To improve the UX of Sidecars Configurations Assignment and Tags in this PR we:
Fixes https://github.com/Graylog2/graylog-plugin-enterprise/issues/3956
Fixes #10454