support multiple configurations per collector #441
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This enables us to use the same backend with multiple configurations
Expand each assignment and backend into a unique ID that combines the backendID + configID Most of the stores us a map with this Id that holds everything together.
Multiple configs can still be assigned by using multiple assignments.
E.g.:
[
{ "collector-1": "config-A" },
{ "collector-1": "config-B" }
]
This will be used to create the ${sidecar.spoolDir} config variable.
Older Graylog servers will refuse unknown NodeDetails attributes Also update go module to version 1.19
If we are talking to an older Graylog instance, report the collectors with just the collector ID. Refactor version handling into separate type
Speeds up testing the sidecar
mpfz0r
changed the title
thll
requested changes
Sep 9, 2022
There was a problem hiding this comment.
First round of review. I'd like to look into it some more. Only did a smoke test with a server version <4.4 to see if we are still compatible. Got the following error when un-assigning a configuration from a collector in the UI:
DEBU[0045] Cleaning up backend: filebeat-6246ef72b8dc9c6bba59bf89
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x132c354]
goroutine 19 [running]:
github.com/Graylog2/collector-sidecar/daemon.(*DaemonConfig).SyncWithAssignments(0xc000138240, 0xc00013ea00)
/Users/thll/go/src/github.com/Graylog2/collector-sidecar/daemon/daemon.go:146 +0x5f4
github.com/Graylog2/collector-sidecar/services.StartPeriodicals.func1()
/Users/thll/go/src/github.com/Graylog2/collector-sidecar/services/periodicals.go:93 +0x773
created by github.com/Graylog2/collector-sidecar/services.StartPeriodicals
/Users/thll/go/src/github.com/Graylog2/collector-sidecar/services/periodicals.go:40 +0xae
Exiting.
Debugger finished with the exit code 0
The recent code changes triggered an old exisiting bug, because we would only keep a backend list of actually assigned backends.
thll
requested changes
Sep 13, 2022
Newer Graylog servers can use it to differentiate multiple configs on one collector. Older servers won't see this field for backwards compatibility.
thll
approved these changes
Sep 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains 3 changes.
1. Refactor the sidecar so it can handle more than one configuration per backend.
The sidecar has three different global "stores" where it keeps the current state:
All of these stores are currently tied together with the collector id.
If we want to support multiple instances of one collector, we need to use a unique ID for this.
The obvious choice is to build an ID out of the collector and the configuration ID. (
collector_id-configuration_id)An assignment like:
Will create 3 backends:
1234-aaa,1234-bbb,5678-eee2. Report the
collector_configuration_directoryback to GraylogThis allows Graylog to create a sidecar config variable
${sidecar.spoolDir}would expand to sth like/var/lib/graylog-sidecar/generated/62864e0056caeb4862bf016b/To make this backwards compatible with older Graylog servers that won't accept additional
NodeDetailfields in the API. (:roll_eyes:) we need to add a version check, and only send this to servers that support it.3. Backwards compatible collector status.
The status of each collector was reported back to Graylog by using the collector_id.
Additionally, we will now report this with the assigned
configuration_id.But only to Graylog > 4.3
Refs https://github.com/Graylog2/graylog-plugin-enterprise/issues/3956