(maint) Fix resource order for installing splunk forwarders

[briancain]

Prior to this commit, when applying the forwarder class to any agents
the initial run would always fail because the module would attempt to
start the splunk service before it existed. This commit fixes that by
adding a notify param to the provider that sets up that forwarder
service.

Before this commit:

May 17 21:41:28 localhost puppet-agent[9660]: Retrieving pluginfacts
May 17 21:41:28 localhost puppet-agent[9660]: Retrieving plugin
May 17 21:41:30 localhost puppet-agent[9660]: Loading facts
May 17 21:41:37 localhost puppet-agent[9660]: Caching catalog for samurai-dog
May 17 21:41:37 localhost puppet-agent[9660]: Applying configuration version '1463521294'
May 17 21:41:37 localhost puppet-agent[9660]: (/Stage[main]/Staging/File[/opt/staging]/ensure) created
May 17 21:41:38 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/Staging::File[DownloadActivityServlet]/File[/opt/staging/splunk]/ensure) created
May 17 21:41:40 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/Staging::File[DownloadActivityServlet]/Exec[/opt/staging/splunk/DownloadActivityServlet]/returns) executed successfully
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/Package[splunkforwarder]/ensure) created
May 17 21:41:41 localhost puppet-agent[9660]: Could not unmask splunk:
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Virtual/Service[splunk]/ensure) change from stopped to running failed: Could not unmask splunk:
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/inputs.conf]/ensure) created
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/outputs.conf]/ensure) created
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/web.conf]/ensure) created
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/Splunkforwarder_input[default_host]/ensure) created
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/Splunkforwarder_output[tcpout_defaultgroup]/ensure) created
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/Splunkforwarder_output[defaultgroup_server]/ensure) created
May 17 21:41:41 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Forwarder/Splunkforwarder_web[forwarder_splunkd_port]/ensure) created
May 17 21:41:43 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns) executed successfully
May 17 21:41:43 localhost systemd: Reloading.
May 17 21:41:43 localhost systemd: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
May 17 21:41:43 localhost systemd: Reloading.
May 17 21:41:43 localhost systemd: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
May 17 21:41:43 localhost puppet-agent[9660]: (/Stage[main]/Splunk::Platform::Posix/Exec[enable_splunkforwarder]/returns) executed successfully
May 17 21:41:43 localhost puppet-agent[9660]: (/Stage[main]/Profile::Splunkforwarder/Splunkforwarder_input[messages-sourcetype]/ensure) created
May 17 21:41:43 localhost puppet-agent[9660]: (Stage[main]) Unscheduling all events on Stage[main]
May 17 21:41:43 localhost puppet-agent[9660]: Applied catalog in 6.17 seconds

After this commit:

root@badger ~]# puppet agent -t --no-use_cached_catalog
Notice: Local environment: 'production' doesn't match server specified node environment 'splunkitup', switching agent to 'splunkitup'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for badger
Info: Applying configuration version '1463528182'
Notice: /Stage[main]/Staging/File[/opt/staging]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/Staging::File[DownloadActivityServlet]/File[/opt/staging/splunk]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/Staging::File[DownloadActivityServlet]/Exec[/opt/staging/splunk/DownloadActivityServlet]/returns: executed successfully
Notice: /Stage[main]/Splunk::Forwarder/Package[splunkforwarder]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/inputs.conf]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/outputs.conf]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/web.conf]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/Splunkforwarder_input[default_host]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/Splunkforwarder_output[tcpout_defaultgroup]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/Splunkforwarder_output[defaultgroup_server]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/Splunkforwarder_web[forwarder_splunkd_port]/ensure: created
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: executed successfully
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[enable_splunkforwarder]/returns: executed successfully
Info: /Stage[main]/Splunk::Platform::Posix/Exec[enable_splunkforwarder]: Scheduling refresh of Service[splunk]
Notice: /Stage[main]/Splunk::Virtual/Service[splunk]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Splunk::Virtual/Service[splunk]: Unscheduling refresh on Service[splunk]
Notice: /Stage[main]/Profile::Splunkforwarder/Splunkforwarder_input[messages-sourcetype]/ensure: created
Notice: Applied catalog in 7.70 seconds

[angrox]

We run into the same issue. Fix works as designed (for the forwarder)

[briancain]

Would love to see this merged :)

[petems]

This simple simple, safe and sensible 👍