Filebeat service restart every 5 min by juju agent

[wmphang]

Hi,
I have setup the filebeat to ship the log to logstash. Everything is working fine, I am able to view the log in kibana. However checking the syslog on, the filebeat service restarted every 5 min by the juju agent, this could result in missing log during the started. Any reason why the juju agent keep restarting the filebeat service? There are no configuration changes, so there should not be any service restart.

/var/log/syslog
Sep 8 04:46:10 xxxx systemd[1]: Started filebeat.
Sep 8 04:50:58 xxxx systemd[1]: Stopping filebeat...
Sep 8 04:50:58 xxxx systemd[1]: Stopped filebeat.
Sep 8 04:50:58 xxxx systemd[1]: Started filebeat.
Sep 8 04:56:13 xxxx systemd[1]: Stopping filebeat...
Sep 8 04:56:13 xxxx systemd[1]: Stopped filebeat.
Sep 8 04:56:13 xxxx systemd[1]: Started filebeat.

/var/log/juju/unit-filebeat-0.log
2017-09-08 04:56:12 INFO juju-log Reactive main running for hook update-status
2017-09-08 04:56:12 INFO juju-log Initializing Apt Layer
2017-09-08 04:56:12 INFO juju-log Invoking reactive handler: reactive/beats_base.py:9:config_changed
2017-09-08 04:56:12 INFO juju-log Invoking reactive handler: reactive/beats_base.py:14:waiting_messaging
2017-09-08 04:56:13 INFO juju-log Invoking reactive handler: reactive/beats_base.py:19:cache_logstash_data
2017-09-08 04:56:13 INFO juju-log Invoking reactive handler: reactive/apt.py:47:ensure_package_status
2017-09-08 04:56:13 INFO juju-log Invoking reactive handler: reactive/filebeat.py:22:render_filebeat_template
2017-09-08 04:56:13 INFO juju-log Writing file /etc/filebeat/filebeat.yml root:root 444

Thanks

[wmphang]

Hi,
Looking at the reactive/filebeat.py code, it restart the filebeat service when there is elasticsearch.available, logstash.available and config.changed event.
I have modified the code that will only restart the filebeat service when changes in /etc/filebeat/filebeat.yml.
Check if the following modification is ok or not as I am new to juju charm not sure if it may cause any issue or not.

filebeat.py

from charms.reactive import when_file_changed

@when('beat.render')
@when_any('elasticsearch.available', 'logstash.available', 'config.changed')
def render_filebeat_template():
    render_without_context('filebeat.yml', '/etc/filebeat/filebeat.yml')
    remove_state('beat.render')
#    service_restart('filebeat')
    status_set('active', 'Filebeat ready.')

@when_file_changed('/etc/filebeat/filebeat.yml')
def restart_filebeat_service():
    service_restart('filebeat')

Thanks

[wmphang]

Hi,
I notice that the juju agent keep overwriting the /etc/filebeat/filebeat.yml every 5 min even there is no changes on the configuration file.
The @when('beat.render') and @when_any('elasticsearch.available', 'logstash.available', 'config.changed') condition is always true and execute render_filebeat_template every 5 min. Is this a normal ?

Thanks

[lazypower]

Thanks for the report @wmhang - that file shouldn't be overwritten every 5 minutes kicking the service. The behavior is indeed errant.

I haven't been active in this source tree for a bit and would be happy to help vet a contribution. You did propose a change inline above where you decorated the restart method behind a file-crc gate. I'm OK with that change so long as its been validated to correct the errant behavior.

[wmphang]

Hi @chuckbutler
Thanks for the response. I am using the filebeat togather with elasticsearch and logstash. In the logstash is also having similar issue the conf.d/beats.conf is being overwrite periodically during hook update-status. I came across this blog http://blog.ajkavanagh.me/2016/12/30/better-charm-interfaces/ talking about similar issue.

[tbaumann]

I suppose this is fixed with the pull request above

[lazypower]

I haven't closed it yet as I haven't published layer-filebeat in some time. we have a hefty amount of merges to master since the last charm build has been run.

I need to get in touch with @marcoceppi to see if he's cool with lending me some credentials or a stack to test on. I unfortunately don't have a lab to validate these changes (or a laptop with lxd at this time).

[jamesbeedy]

@chuckbutler "I unfortunately don't have a lab to validate these changes (or a laptop with lxd at this time)" - oh how times have changed

Looks like you need to apply for some charming credsssssss

[lazypower]

I applied weeks ago, no response as of yet

[lazypower]

Ok, I have some good news. I scraped together some older hardware and should be able to vet the changes in here and cut a newer release of the beats stack. I'll also look into allocating some resources for executing tests in a LXD environment so we can validate submissions. It will come with the Closet Net CLA where if my power goes out, your test runs will fail - but this is temporary until we can get them in some centrally hosted juju test environment.

I opened #29 to help cut down on future churn like we had this last month with people actually using and submitting patches to filebeat.

[axinojolais]

This is canonical/layer-beats-base#18 by the way

[kwmonroe]

@axinojolais, thanks for finding and fixing this in layer-beats-base. I'm closing this out as it has made its way to the stable channel here (rev 10):

https://jujucharms.com/filebeat/