LogSenderService sends too much log after config change

[meiao]

Description

After changing the config file, LogSenderService will get the new value for application_logging.forwarding.max_samples_stored and use that to limit the reservoirs. That configuration option is measured in logs/minute, but logs use real time streaming, so they are gathered every 5 seconds.

So every 5 seconds the agent will harvest and send up to what it should send in 1 minute.

This ignores any restrictions set from the backend. And is not limited to changes on this configuration option, any change in the config file (other than that config option) will make the LogSenderService send 12 times more logs than expected.

Also, TransactionLog always use the value from the config, never the one returned from the backend. This causes TransactionLog to always hold up to 12 times more logs than could ever be sent.

Steps to reproduce

Create an application that logs 20 lines on each request.
Configure the agent turning on log forwarding and set application_logging.forwarding.max_samples_stored to 120.
Start the application and make 1 request.
Check logs, metrics or UI and see that 10 log lines were sent.
Change the configuration and set application_logging.forwarding.max_samples_stored to 20.
Wait until the new settings are loaded.
Make one request. This request should still sends 10 log lines as the reservoirs are not regenerated unless they are harvested.
Make another request. This request will send 20 log lines.

[workato-integration]

https://new-relic.atlassian.net/browse/NR-232381

[jbedell-newrelic]

Resolved by: #1784