Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow multiple configurations for a sidecar #5706

Closed
bitfactory-henno-schooljan opened this issue Feb 21, 2019 · 13 comments
Closed

Allow multiple configurations for a sidecar #5706

bitfactory-henno-schooljan opened this issue Feb 21, 2019 · 13 comments

Comments

@bitfactory-henno-schooljan
Copy link

bitfactory-henno-schooljan commented Feb 21, 2019

In Graylog 2, tags were a very convenient method for classifying and combining configurations for specific sidecars. E.g. a generic 'linux' tag for gathering generic logs from all Linux machines, and more specific tags that would only apply to certain machines (including these Linux machines).
Simply defining the appropriate tags in a sidecar configuration file on a server would cause it to collect the correct logs from this server, using the combined configurations for all these tags.

With the removal of tag-based configuration it looks like it is no longer possible to apply multiple configurations to a sidecar. I need to create a configuration for every single combination of (former) tags I want to use on all my servers, and combine the desired log inputs inside all these configurations using variables if I want to avoid duplication. The number of combinations can grow exponentially when more log inputs are introduced.

How difficult would it be to allow multiple configurations to be applied to a sidecar, effectively allowing the combination of different configurations without having to create a lot of compound configurations every time a new log input is introduced?

@deeshe
Copy link
Contributor

deeshe commented Feb 25, 2019

Please refer to this community thread for a workaround:
https://community.graylog.org/t/graylog-3-0-filebeat-configuration-and-tags/8949/16

@mpfz0r
Copy link
Contributor

mpfz0r commented Mar 7, 2019

One possible way to support this, is that we add a tags list back into the sidecar.yml.
We could expose these tags as runtime variables.
So let's say that your sidecar.yml has tags: [ "apache", "rails" ]

your configuration could render sections based on tags.
For instance

filebeat.inputs:
 - type: log
   paths:
    - /var/log/test*.log
    <#if sidecar.tags.apache?? >
    - /var/log/apache
    </#if>
    <#if sidecar.tags.rails?? >
    - /var/log/rails/application.log
    </#if>

@menai34
Copy link

menai34 commented Mar 20, 2020

Hello,
Unfortunately, this doesn't work for Graylog 3.2 and a new sidecar.

@zez3
Copy link

zez3 commented Mar 20, 2020

Yeah, it would be nice to have this in the future but please focus on supporting ES 7

@bernd bernd changed the title Graylog 3.0: Allow multiple configurations for a sidecar Allow multiple configurations for a sidecar Nov 15, 2020
@mpibpc-mroose
Copy link

The help for the most recent Graylog version implies, that the conditional configuration of sidecars should work:

image

I tried this but never succeeded. Seems that the if statement is somehow processed, but the tags in sidcar.yaml do not "reach" graylog when the configuration is rendered. Is this a documentation issue or a bug? Reading this issue I would say it's not correctly documented.

As this feature makes a lot of sense a +1 for me to re-implement it.

@mpfz0r
Copy link
Contributor

mpfz0r commented Jan 13, 2023

@mpibpc-mroose which version of graylog and sidecar are you using?

@mpibpc-mroose
Copy link

both in the newest versions:

Graylog: 5.0.2
Sidecar: 1.2.0

Discussed that in the graylog forum, too:

https://community.graylog.org/t/use-sidecar-tags-in-filebeat-config/27251

@mpfz0r
Copy link
Contributor

mpfz0r commented Jan 13, 2023

You need at least sidecar 1.3.0 for this.
As of today we also released 1.4.0

@boosty
Copy link
Contributor

boosty commented Jan 13, 2023

Hi everyone, I think we can close this now.

With Graylog 5.0 (and Sidecar 1.3) we released the possibility to automatically assign Sidecar configurations, based on tags.

A list of tags can be assigned in a Sidecar's YAML configuration. Collector configuration matching any of these tags will then automatically be applied to the Sidecar.

Details:

Recently a related change was added to the Sidecar installer on Windows to accept tags at installation time, for convenience:

@boosty boosty closed this as completed Jan 13, 2023
@BenoitPoulet
Copy link

Great but the doc is not clear about this.
I copy/paste the doc with "1.2" written in the commands. perhaps you should edit and replace "1.2" with "$version" or something like that.

The doc about tags and sniplets is minimalist too.

@mpfz0r
Copy link
Contributor

mpfz0r commented Jan 13, 2023

@BenoitPoulet which docs are you referring to specifically?

@BenoitPoulet
Copy link

@mpfz0r
Copy link
Contributor

mpfz0r commented Jan 13, 2023

@BenoitPoulet ah, thanks. I'll get that fixed.
It should be
wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-5_all.deb
and
sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-5.noarch.rpm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants