[Reporting] Review default intervals for reporting pollers

[kostasb]

The default intervals for the reporting pollers generate tens of requests per minute against .reporting-* indices.

Here's a rough count in one minute from a plain install on v6.4:

40 POST /.reporting-*/esqueue/_search?version=true
18 POST /.reporting-*/_search?filter_path=hits.total%2Caggregations.jobTypes.buckets%2Caggregations.objectTypes.buckets%2Caggregations.layoutTypes.buckets%2Caggregations.statusTypes.buckets

There is 1 server-side poller in reporting:
- xpack.reporting.queue.pollInterval which is the worker's interval to poll the reporting job queue for new reporting jobs. This controls the queries to esqueue, it is configurable and defaults to 3000 (ms).

There are 2 client-side pollers in reporting:
- xpack.reporting.poll.jobsRefresh.interval which is used by the ReportListing component under management / reporting to display a list of all reporting jobs.
- xpack.reporting.poll.jobCompletionNotifier.interval which is used by the job completion notifier. The job completion notifier displays toast notifications when a user's reporting job has finished.

We may want to reconsider the default intervals to avoid generating too many requests, particularly on use cases which do not utilize the reporting features heavily enough to make good use of such brief polling interval.

cc @nreese @timroes

[tsullivan]

We're working on improving the report generation API to have an "immediate" option, at which point increasing the polling interval higher makes a lot of sense. Polling will primarily be a feature that will support scheduled reports.

[tsullivan]

40 POST /.reporting-*/esqueue/_search?version=true

BTW this number is misleadingly high, because we have a bug where Reporting executes 1 search per "export type." Export types are PDF, PNG and CSV, and we want to give ourselves room to add more.

#32839 changes the logic to execute a single search per xpack.reporting.queue.pollInterval, no matter how many export types there are.

[tsullivan]

Hi @kostasb #32839 fixes the amount of server-side polling to be deterministic. If the server-side and front-end interval settings are the same, you should see the amount of polling be the same. It will still be in the "tens" as the server-side alone should be 20 per minute. The browser-side is harder to adjust for, because it depends on how many browsers are open in a particular timeframe.

We may want to reconsider the default intervals to avoid generating too many requests,

The fix doesn't address the issue you raised directly, but I think it reduces the severity quite a bit. And user's should always understand their settings and the default settings, monitor their search count and adjust settings accordingly.

Is OK to close this issue?

[kostasb]

@tsullivan Thank your putting this fix together! It will certainly help reduce the request count.

Before we close this, could you please confirm whether the setting xpack.reporting.queue.pollInterval will still be applicable once your PR #32839 is merged?

I'd like to follow up with a request to whitelist it in the Elasticsearch service kibana settings. Currently this isn't accessible to the users of the service, so Reporting can trigger a significant amount of search activity beyond the user's control.

[tsullivan]

Yes, xpack.reporting.queue.pollInterval is still the config that determines how often Reporting checks for work. The fix is that overall check is now only 1 request to Elasticsearch each interval period. The bug was performance-related in that it used multiple requests per interval period.

It makes sense to whitelist xpack.reporting.queue.pollInterval in user-controlled configs. users should be able to change the query rate depending on their use of Reporting.

[kostasb]

Thank you Tim, I'll go ahead and close this one and follow up about enabling the setting on Elastic Cloud.