Make telemetry task use a schedule instead of scheduling explicitly for midnight

[mikecote]

Fixes #140973
Fixes https://github.com/elastic/kibana-team/issues/563

In this PR, I'm fixing flaky tests that caused extra telemetry runs whenever CI would run them near midnight UTC. The assertion expected two runs while sometimes a 3rd run would happen if the test ran near midnight when the telemetry task was scheduled to run again..

To fix this, I've moved away from the midnight scheduling given telemetry only needs to be reported daily, and moved the task to use a schedule within task manager to make the task run daily (+24hrs from the previous run). This also improves error handling given task manager will now know it's a recurring task and recurring tasks never get marked as failed.

The following verification steps can be done using this query in Dev Tools

GET .kibana_task_manager/_search
{
  "query": {
    "term": {
      "task.taskType": "actions_telemetry"
    }
  }
}

To verify existing tasks migrating to a schedule

1. Using main, setup a fresh Kibana and ES instance
2. Keep Elasticsearch running but shut down Kibana after setup is complete
3. Switch from main to this PR
4. Add await taskManager.runSoon(TASK_ID); after the ensureScheduled call within x-pack/plugins/actions/server/usage/task.ts.
5. Startup Kibana
6. Go in Dev Tools and pull the task information to see a new schedule attribute added

To verify fresh installs

1. Using this PR code, setup a fresh Kibana and ES instance
2. Go in Dev Tools and pull the task information to see a new schedule attribute added

Flaky test runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2017

[mikecote]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

should this be 1d?

[mikecote]

Whops, ended up committing something for testing. Reverted back in 412db8d. That would have been A LOT of telemetry collection 😆

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 412db8d

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
taskManager	39	40	+1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
taskManager	7	6	-1

Unknown metric groups

API count

id	before	after	diff
taskManager	80	82	+2

ESLint disabled line counts

id	before	after	diff
securitySolution	433	436	+3

Total ESLint disabled count

id	before	after	diff
securitySolution	513	516	+3

History

• 💔 Build #114786 failed 33a75ee
• 💚 Build #114616 succeeded 336d6da
• 💔 Build #114433 failed 756e541

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mikecote

[ymao1]

LGTM!