Remove filebeat from linux packages

[mpfz0r]

Linux users should install their collectors from elastic

[mariussturm]

Should we also remove the nxlog spool directory to be collector independent? Would need to be reflected in the default config as well.

[bernd]

I think it would be great if every collector instance gets it's own working directory that is created by the sidecar before starting the collector. We could also expose this as a sidecar. variable. Otherwise the user has to make sure that certain directories exist before a collector can start.

[mariussturm]

Ok, then we should add working directories for all default collectors or even for all that are listed in collector_binaries_whitelist.

[bernd]

Ok, then we should add working directories for all default collectors or even for all that are listed in collector_binaries_whitelist.

If we do that, I think we need a working directory for every instance. If I want to run two separate instances of filebeat on one sidecar node we cannot really use the same working directory.

But I think the working directory stuff is a new feature and out of scope for 3.0, unfortunately.

[mariussturm]

Sure we can create them dynamically in a future version but what about that one static directory we have now in the package?

[bernd]

Sure we can create them dynamically in a future version but what about that one static directory we have now in the package?

Does the nxlog package create a spool directory already? I am not sure why we created the nxlog spool directory in the sidecar package in the first place. Can you elaborate?

[mariussturm]

The official package creates /var/spool/nxlog. The intention was that we don't wanted to create a collision with Sidecar managed and other running instances. So the main question is if we want to keep that separation or if we want to rely on directories that are provided by the standard collector packages as they are the recommend way of installing collectors anyway now? Same would be true for Filebeat then.

[bernd]

So the main question is if we want to keep that separation or if we want to rely on directories that are provided by the standard collector packages as they are the recommend way of installing collectors anyway now?

My personal opinion is that the sidecar should manage everything that is needed for a collector instance to run (spool, log, cache, ...) in a separate location. (e.g. /var/lib/graylog-sidecar/work/...) Otherwise the user needs to make sure that the filebeat, nxlog and winlogbeat services that get installed by the official packages are disabled and not in use. The user also needs to manually create log and spool directories if she wants to manage multiple instances of filebeat or nxlog on the same sidecar node.

I think having the separation makes it easier for the user to use the sidecar system in general.

What do you think @mariussturm @mpfz0r? 😃

[mariussturm]

I agree, the Sidecar should manage the directories for each collector instance (PID is another example) but till we don't have dynamic creation I would go with a static spool directory for every default collector. It's fairly simple to put a second directory next to it if a user needs it. The collector configuration needs to be adopted anyway in that case.
The default Filebeat config is using this:

path:
  data: /var/cache/graylog-sidecar/filebeat/data"
  logs: /var/log/graylog-sidecar"

So we could create at least the spool dir. Windows needs to be checked as well.

[mpfz0r]

Moving this discussion to: #331