Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete connector usage reporting task in on-prem and cloud #199650

Merged

Conversation

ersin-erdal
Copy link
Contributor

@ersin-erdal ersin-erdal commented Nov 11, 2024

Towards: https://github.com/elastic/response-ops-team/issues/209

connector_usage_reporting task is supposed to work only in serverless.
As there is no check for that, even though it reports nothing, It was working for on-prem and cloud as well.
This PR deletes the connector_usage_reporting task after the first run by checking if plugins.cloud.serverless.projectId is missing.

To verify:

Run Kibana on this branch, then check the reporting task in the console by running the below query.
There shouldn't be any task.

GET .kibana_task_manager_*/_search
{
  "query": {
    "prefix": {
      "task.taskType": {
        "value": "actions:"
      }
    }
  }
}

Then remove or comment out the line (shouldDeleteTask: true) added by this PR and save the file, Kibana will restart and register the task again. You can use the above query again to see that the task is still registered.

@ersin-erdal ersin-erdal added release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Nov 11, 2024
@ersin-erdal ersin-erdal marked this pull request as ready for review November 11, 2024 15:53
@ersin-erdal ersin-erdal requested a review from a team as a code owner November 11, 2024 15:53
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@ersin-erdal ersin-erdal self-assigned this Nov 11, 2024
@mikecote
Copy link
Contributor

Should we / can we also skip starting the connector usage reporting task service on startup? Otherwise we'd be scheduling and running the task once on every Kibana startup, I believe.

@ersin-erdal
Copy link
Contributor Author

Should we / can we also skip starting the connector usage reporting task service on startup? Otherwise we'd be scheduling and running the task once on every Kibana startup, I believe.

Then we have to remove it from the functional test that keeps track of the registered tasks.
I am ok with that too, if we think it is better.

@mikecote
Copy link
Contributor

I took a deeper look, I think it makes sense to keep the task registered in any scenario (stateless or stateful) given we have this out already.

Perhaps we can wrap the ensureScheduled in an if statement that checks for serverless, so moving forward we only schedule the tasks in serverless?

https://github.com/mikecote/kibana/blob/ba9f8ecb0fba97869d913c31fb6cb9e7c6c94ca5/x-pack/plugins/actions/server/usage/connector_usage_reporting_task.ts#L93-L102

@ersin-erdal ersin-erdal added the ci:cloud-redeploy Always create a new Cloud deployment label Nov 22, 2024
@ersin-erdal ersin-erdal added ci:project-deploy-observability Create an Observability project and removed ci:cloud-redeploy Always create a new Cloud deployment labels Nov 22, 2024
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Contributor

elasticmachine commented Nov 25, 2024

💛 Build succeeded, but was flaky

  • Buildkite Build
  • Commit: ff3ea81
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-199650-ff3ea8178daa

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #53 / discover/group4 data view flyout "after all" hook for "update data view with no time field"
  • [job] [logs] FTR Configs #53 / discover/group4 data view flyout create saved data view

Metrics [docs]

✅ unchanged

History

cc @ersin-erdal

@ersin-erdal
Copy link
Contributor Author

I took a deeper look, I think it makes sense to keep the task registered in any scenario (stateless or stateful) given we have this out already.

Perhaps we can wrap the ensureScheduled in an if statement that checks for serverless, so moving forward we only schedule the tasks in serverless?

https://github.com/mikecote/kibana/blob/ba9f8ecb0fba97869d913c31fb6cb9e7c6c94ca5/x-pack/plugins/actions/server/usage/connector_usage_reporting_task.ts#L93-L102

Done

Copy link
Contributor

@ymao1 ymao1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Verified an existing task in a deployment with no project ID gets deleted and that the task is never scheduled for new deployments with no project ID.

@ersin-erdal ersin-erdal merged commit a1e97b1 into elastic:main Nov 26, 2024
25 checks passed
@ersin-erdal ersin-erdal deleted the remove-reporting-task-for-on-prem branch November 26, 2024 10:12
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12028368247

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 26, 2024
…99650)

Towards: elastic/response-ops-team#209

`connector_usage_reporting` task is supposed to work only in serverless.
As there is no check for that, even though it reports nothing, It was
working for on-prem and cloud as well.
This PR deletes the `connector_usage_reporting` task after the first run
by checking if `plugins.cloud.serverless.projectId` is missing.

### To verify:
Run Kibana on this branch, then check the reporting task in the console
by running the below query.
There shouldn't be any task.

```
GET .kibana_task_manager_*/_search
{
  "query": {
    "prefix": {
      "task.taskType": {
        "value": "actions:"
      }
    }
  }
}
```

Then remove or comment out the line (`shouldDeleteTask: true`) added by
this PR and save the file, Kibana will restart and register the task
again. You can use the above query again to see that the task is still
registered.

(cherry picked from commit a1e97b1)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Nov 26, 2024
…9650) (#201741)

# Backport

This will backport the following commits from `main` to `8.x`:
- [Delete connector usage reporting task in on-prem and cloud
(#199650)](#199650)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ersin
Erdal","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-26T10:12:23Z","message":"Delete
connector usage reporting task in on-prem and cloud
(#199650)\n\nTowards:
https://github.com/elastic/response-ops-team/issues/209\r\n\r\n`connector_usage_reporting`
task is supposed to work only in serverless.\r\nAs there is no check for
that, even though it reports nothing, It was\r\nworking for on-prem and
cloud as well.\r\nThis PR deletes the `connector_usage_reporting` task
after the first run\r\nby checking if
`plugins.cloud.serverless.projectId` is missing.\r\n\r\n### To
verify:\r\nRun Kibana on this branch, then check the reporting task in
the console\r\nby running the below query.\r\nThere shouldn't be any
task.\r\n\r\n```\r\nGET .kibana_task_manager_*/_search\r\n{\r\n
\"query\": {\r\n \"prefix\": {\r\n \"task.taskType\": {\r\n \"value\":
\"actions:\"\r\n }\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen remove or comment
out the line (`shouldDeleteTask: true`) added by\r\nthis PR and save the
file, Kibana will restart and register the task\r\nagain. You can use
the above query again to see that the task is
still\r\nregistered.","sha":"a1e97b1a43f2971dbf528201b3b82fb4b178b6bb","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v9.0.0","backport:prev-minor","ci:project-deploy-observability"],"title":"Delete
connector usage reporting task in on-prem and
cloud","number":199650,"url":"https://github.com/elastic/kibana/pull/199650","mergeCommit":{"message":"Delete
connector usage reporting task in on-prem and cloud
(#199650)\n\nTowards:
https://github.com/elastic/response-ops-team/issues/209\r\n\r\n`connector_usage_reporting`
task is supposed to work only in serverless.\r\nAs there is no check for
that, even though it reports nothing, It was\r\nworking for on-prem and
cloud as well.\r\nThis PR deletes the `connector_usage_reporting` task
after the first run\r\nby checking if
`plugins.cloud.serverless.projectId` is missing.\r\n\r\n### To
verify:\r\nRun Kibana on this branch, then check the reporting task in
the console\r\nby running the below query.\r\nThere shouldn't be any
task.\r\n\r\n```\r\nGET .kibana_task_manager_*/_search\r\n{\r\n
\"query\": {\r\n \"prefix\": {\r\n \"task.taskType\": {\r\n \"value\":
\"actions:\"\r\n }\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen remove or comment
out the line (`shouldDeleteTask: true`) added by\r\nthis PR and save the
file, Kibana will restart and register the task\r\nagain. You can use
the above query again to see that the task is
still\r\nregistered.","sha":"a1e97b1a43f2971dbf528201b3b82fb4b178b6bb"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199650","number":199650,"mergeCommit":{"message":"Delete
connector usage reporting task in on-prem and cloud
(#199650)\n\nTowards:
https://github.com/elastic/response-ops-team/issues/209\r\n\r\n`connector_usage_reporting`
task is supposed to work only in serverless.\r\nAs there is no check for
that, even though it reports nothing, It was\r\nworking for on-prem and
cloud as well.\r\nThis PR deletes the `connector_usage_reporting` task
after the first run\r\nby checking if
`plugins.cloud.serverless.projectId` is missing.\r\n\r\n### To
verify:\r\nRun Kibana on this branch, then check the reporting task in
the console\r\nby running the below query.\r\nThere shouldn't be any
task.\r\n\r\n```\r\nGET .kibana_task_manager_*/_search\r\n{\r\n
\"query\": {\r\n \"prefix\": {\r\n \"task.taskType\": {\r\n \"value\":
\"actions:\"\r\n }\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen remove or comment
out the line (`shouldDeleteTask: true`) added by\r\nthis PR and save the
file, Kibana will restart and register the task\r\nagain. You can use
the above query again to see that the task is
still\r\nregistered.","sha":"a1e97b1a43f2971dbf528201b3b82fb4b178b6bb"}}]}]
BACKPORT-->

Co-authored-by: Ersin Erdal <[email protected]>
paulinashakirova pushed a commit to paulinashakirova/kibana that referenced this pull request Nov 26, 2024
…99650)

Towards: elastic/response-ops-team#209

`connector_usage_reporting` task is supposed to work only in serverless.
As there is no check for that, even though it reports nothing, It was
working for on-prem and cloud as well.
This PR deletes the `connector_usage_reporting` task after the first run
by checking if `plugins.cloud.serverless.projectId` is missing.

### To verify:
Run Kibana on this branch, then check the reporting task in the console
by running the below query.
There shouldn't be any task.

```
GET .kibana_task_manager_*/_search
{
  "query": {
    "prefix": {
      "task.taskType": {
        "value": "actions:"
      }
    }
  }
}
```

Then remove or comment out the line (`shouldDeleteTask: true`) added by
this PR and save the file, Kibana will restart and register the task
again. You can use the above query again to see that the task is still
registered.
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
…99650)

Towards: elastic/response-ops-team#209

`connector_usage_reporting` task is supposed to work only in serverless.
As there is no check for that, even though it reports nothing, It was
working for on-prem and cloud as well.
This PR deletes the `connector_usage_reporting` task after the first run
by checking if `plugins.cloud.serverless.projectId` is missing.

### To verify:
Run Kibana on this branch, then check the reporting task in the console
by running the below query.
There shouldn't be any task.

```
GET .kibana_task_manager_*/_search
{
  "query": {
    "prefix": {
      "task.taskType": {
        "value": "actions:"
      }
    }
  }
}
```

Then remove or comment out the line (`shouldDeleteTask: true`) added by
this PR and save the file, Kibana will restart and register the task
again. You can use the above query again to see that the task is still
registered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.18.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants