[Response Ops][Task Manager] Onboard 12.5% of ECH clusters to use mget task claiming
#196317
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ymao1
added
ci:cloud-deploy
ymao1
commented
Oct 15, 2024
ymao1
changed the title
mget task claiming
mikecote
approved these changes
Oct 15, 2024
|
@elasticmachine merge upstream |
ymao1
added
release_note:skip
|
Pinging @elastic/response-ops (Team:ResponseOps) |
|
@elasticmachine merge upstream |
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]Public APIs missing comments
History
cc @ymao1 |
|
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/11356536561 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 16, 2024
…et` task claiming (elastic#196317) Resolves elastic/response-ops-team#239 ## Summary Deployed to cloud: deployment ID was `ab4e88d139f93d43024837d96144e7d4`. Since the deployment ID starts with an `a`, this should start with `mget` and I can see in the logs with the latest push that this is true <img width="2190" alt="Screenshot 2024-10-15 at 2 59 20 PM" src="https://github.com/user-attachments/assets/079bc4d8-365e-4ba6-b7a9-59fe506283d9"> Deployed to serverless: project ID was `d33d22a94ce246d091220eace2c4e4bb`. See in the logs: `Using claim strategy mget as configured for deployment d33d22a94ce246d091220eace2c4e4bb` Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 267efdf)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Oct 16, 2024
…se `mget` task claiming (#196317) (#196460) # Backport This will backport the following commits from `main` to `8.x`: - [[Response Ops][Task Manager] Onboard 12.5% of ECH clusters to use `mget` task claiming (#196317)](#196317) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ying Mao","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-16T00:24:52Z","message":"[Response Ops][Task Manager] Onboard 12.5% of ECH clusters to use `mget` task claiming (#196317)\n\nResolves https://github.com/elastic/response-ops-team/issues/239\r\n\r\n## Summary\r\n\r\nDeployed to cloud: deployment ID was `ab4e88d139f93d43024837d96144e7d4`.\r\nSince the deployment ID starts with an `a`, this should start with\r\n`mget` and I can see in the logs with the latest push that this is true\r\n\r\n<img width=\"2190\" alt=\"Screenshot 2024-10-15 at 2 59 20 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/079bc4d8-365e-4ba6-b7a9-59fe506283d9\">\r\n\r\n\r\nDeployed to serverless: project ID was\r\n`d33d22a94ce246d091220eace2c4e4bb`. See in the logs: `Using claim\r\nstrategy mget as configured for deployment\r\nd33d22a94ce246d091220eace2c4e4bb`\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"267efdf31fe9ae314b0bed99bc23db5452a2aaa3","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Feature:Task Manager","Team:ResponseOps","v9.0.0","backport:prev-minor","ci:cloud-deploy","ci:project-deploy-elasticsearch","v8.16.0"],"title":"[Response Ops][Task Manager] Onboard 12.5% of ECH clusters to use `mget` task claiming","number":196317,"url":"https://github.com/elastic/kibana/pull/196317","mergeCommit":{"message":"[Response Ops][Task Manager] Onboard 12.5% of ECH clusters to use `mget` task claiming (#196317)\n\nResolves https://github.com/elastic/response-ops-team/issues/239\r\n\r\n## Summary\r\n\r\nDeployed to cloud: deployment ID was `ab4e88d139f93d43024837d96144e7d4`.\r\nSince the deployment ID starts with an `a`, this should start with\r\n`mget` and I can see in the logs with the latest push that this is true\r\n\r\n<img width=\"2190\" alt=\"Screenshot 2024-10-15 at 2 59 20 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/079bc4d8-365e-4ba6-b7a9-59fe506283d9\">\r\n\r\n\r\nDeployed to serverless: project ID was\r\n`d33d22a94ce246d091220eace2c4e4bb`. See in the logs: `Using claim\r\nstrategy mget as configured for deployment\r\nd33d22a94ce246d091220eace2c4e4bb`\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"267efdf31fe9ae314b0bed99bc23db5452a2aaa3"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196317","number":196317,"mergeCommit":{"message":"[Response Ops][Task Manager] Onboard 12.5% of ECH clusters to use `mget` task claiming (#196317)\n\nResolves https://github.com/elastic/response-ops-team/issues/239\r\n\r\n## Summary\r\n\r\nDeployed to cloud: deployment ID was `ab4e88d139f93d43024837d96144e7d4`.\r\nSince the deployment ID starts with an `a`, this should start with\r\n`mget` and I can see in the logs with the latest push that this is true\r\n\r\n<img width=\"2190\" alt=\"Screenshot 2024-10-15 at 2 59 20 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/079bc4d8-365e-4ba6-b7a9-59fe506283d9\">\r\n\r\n\r\nDeployed to serverless: project ID was\r\n`d33d22a94ce246d091220eace2c4e4bb`. See in the logs: `Using claim\r\nstrategy mget as configured for deployment\r\nd33d22a94ce246d091220eace2c4e4bb`\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"267efdf31fe9ae314b0bed99bc23db5452a2aaa3"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Ying Mao <[email protected]>
mikecote
added a commit
that referenced
this pull request
Oct 18, 2024
Similar to #196317 In this PR, I'm flipping the mget feature flag to on for all elastic owned ECH clusters. Elastic owned clusters are determined by looking at `plugins.cloud?.isElasticStaffOwned`. ## To verify Observe the PR deployment which doesn't start with `a` or `b` yet is using the mget claim strategy by logging `Using claim strategy mget` on startup.
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 18, 2024
…ic#196757) Similar to elastic#196317 In this PR, I'm flipping the mget feature flag to on for all elastic owned ECH clusters. Elastic owned clusters are determined by looking at `plugins.cloud?.isElasticStaffOwned`. ## To verify Observe the PR deployment which doesn't start with `a` or `b` yet is using the mget claim strategy by logging `Using claim strategy mget` on startup. (cherry picked from commit 97f2a90)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 18, 2024
…ic#196757) Similar to elastic#196317 In this PR, I'm flipping the mget feature flag to on for all elastic owned ECH clusters. Elastic owned clusters are determined by looking at `plugins.cloud?.isElasticStaffOwned`. ## To verify Observe the PR deployment which doesn't start with `a` or `b` yet is using the mget claim strategy by logging `Using claim strategy mget` on startup. (cherry picked from commit 97f2a90)
mikecote
added a commit
to mikecote/kibana
that referenced
this pull request
Oct 21, 2024
… use `mget` task claiming (elastic#196317)" This reverts commit 267efdf.
mikecote
added a commit
that referenced
this pull request
Oct 25, 2024
Resolves #194625 In this PR, I'm setting `mget` as the default task claiming strategy along the following changes: - Given we no longer need the 8.16 specific PRs (#196317 and #196757), I've also reverted them. - Given we now use `met` as the default, I've renamed `task_manager_claimer_mget` to `task_manager_claimer_update_by_query` and made tests in that folder test using the `update_by_query` claim strategy. - Stabilize flaky tests caused by mget + polling for tasks more frequently Flaky test runners: - [[59b71bc](59b71bc)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7197 - [[aea910e](aea910e)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7199 - [[4723ced](4723ced)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7206 - [[d28c8c5](d28c8c5)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7209 - [[dd7773a](dd7773a)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7224 --------- Co-authored-by: kibanamachine <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 25, 2024
Resolves elastic#194625 In this PR, I'm setting `mget` as the default task claiming strategy along the following changes: - Given we no longer need the 8.16 specific PRs (elastic#196317 and elastic#196757), I've also reverted them. - Given we now use `met` as the default, I've renamed `task_manager_claimer_mget` to `task_manager_claimer_update_by_query` and made tests in that folder test using the `update_by_query` claim strategy. - Stabilize flaky tests caused by mget + polling for tasks more frequently Flaky test runners: - [[59b71bc](elastic@59b71bc)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7197 - [[aea910e](elastic@aea910e)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7199 - [[4723ced](elastic@4723ced)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7206 - [[d28c8c5](elastic@d28c8c5)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7209 - [[dd7773a](elastic@dd7773a)] https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7224 --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit c31f11e)
kibanamachine
added a commit
that referenced
this pull request
Oct 25, 2024
# Backport This will backport the following commits from `main` to `8.x`: - [Set mget task claim strategy as the default (#197070)](#197070) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Mike Côté","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-25T12:57:46Z","message":"Set mget task claim strategy as the default (#197070)\n\nResolves https://github.com/elastic/kibana/issues/194625\r\n\r\nIn this PR, I'm setting `mget` as the default task claiming strategy\r\nalong the following changes:\r\n- Given we no longer need the 8.16 specific PRs\r\n(#196317 and\r\nhttps://github.com//pull/196757), I've also reverted them.\r\n- Given we now use `met` as the default, I've renamed\r\n`task_manager_claimer_mget` to `task_manager_claimer_update_by_query`\r\nand made tests in that folder test using the `update_by_query` claim\r\nstrategy.\r\n- Stabilize flaky tests caused by mget + polling for tasks more\r\nfrequently\r\n\r\nFlaky test runners:\r\n-\r\n[[59b71bc](https://github.com/elastic/kibana/pull/197070/commits/59b71bcdbe4d617a6d91131976540b334c9220ff)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7197\r\n-\r\n[[aea910e](https://github.com/elastic/kibana/pull/197070/commits/aea910e36dc71116dee708a7168971df30a18a3d)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7199\r\n-\r\n[[4723ced](https://github.com/elastic/kibana/pull/197070/commits/4723ced751f0e5114a9bc7a2928dcf0cb326472e)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7206\r\n-\r\n[[d28c8c5](https://github.com/elastic/kibana/pull/197070/commits/d28c8c56f67802107c17a627357251b9eff797ba)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7209\r\n-\r\n[[dd7773a](https://github.com/elastic/kibana/pull/197070/commits/dd7773aebad5664e725c9849c0ed9418f9dc68ed)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7224\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"c31f11e7d8c5c586258399c5e702e2247e05d0e4","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Feature:Task Manager","Team:ResponseOps","v9.0.0","backport:prev-minor","v8.17.0"],"title":"Set mget task claim strategy as the default","number":197070,"url":"https://github.com/elastic/kibana/pull/197070","mergeCommit":{"message":"Set mget task claim strategy as the default (#197070)\n\nResolves https://github.com/elastic/kibana/issues/194625\r\n\r\nIn this PR, I'm setting `mget` as the default task claiming strategy\r\nalong the following changes:\r\n- Given we no longer need the 8.16 specific PRs\r\n(#196317 and\r\nhttps://github.com//pull/196757), I've also reverted them.\r\n- Given we now use `met` as the default, I've renamed\r\n`task_manager_claimer_mget` to `task_manager_claimer_update_by_query`\r\nand made tests in that folder test using the `update_by_query` claim\r\nstrategy.\r\n- Stabilize flaky tests caused by mget + polling for tasks more\r\nfrequently\r\n\r\nFlaky test runners:\r\n-\r\n[[59b71bc](https://github.com/elastic/kibana/pull/197070/commits/59b71bcdbe4d617a6d91131976540b334c9220ff)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7197\r\n-\r\n[[aea910e](https://github.com/elastic/kibana/pull/197070/commits/aea910e36dc71116dee708a7168971df30a18a3d)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7199\r\n-\r\n[[4723ced](https://github.com/elastic/kibana/pull/197070/commits/4723ced751f0e5114a9bc7a2928dcf0cb326472e)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7206\r\n-\r\n[[d28c8c5](https://github.com/elastic/kibana/pull/197070/commits/d28c8c56f67802107c17a627357251b9eff797ba)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7209\r\n-\r\n[[dd7773a](https://github.com/elastic/kibana/pull/197070/commits/dd7773aebad5664e725c9849c0ed9418f9dc68ed)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7224\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"c31f11e7d8c5c586258399c5e702e2247e05d0e4"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197070","number":197070,"mergeCommit":{"message":"Set mget task claim strategy as the default (#197070)\n\nResolves https://github.com/elastic/kibana/issues/194625\r\n\r\nIn this PR, I'm setting `mget` as the default task claiming strategy\r\nalong the following changes:\r\n- Given we no longer need the 8.16 specific PRs\r\n(#196317 and\r\nhttps://github.com//pull/196757), I've also reverted them.\r\n- Given we now use `met` as the default, I've renamed\r\n`task_manager_claimer_mget` to `task_manager_claimer_update_by_query`\r\nand made tests in that folder test using the `update_by_query` claim\r\nstrategy.\r\n- Stabilize flaky tests caused by mget + polling for tasks more\r\nfrequently\r\n\r\nFlaky test runners:\r\n-\r\n[[59b71bc](https://github.com/elastic/kibana/pull/197070/commits/59b71bcdbe4d617a6d91131976540b334c9220ff)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7197\r\n-\r\n[[aea910e](https://github.com/elastic/kibana/pull/197070/commits/aea910e36dc71116dee708a7168971df30a18a3d)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7199\r\n-\r\n[[4723ced](https://github.com/elastic/kibana/pull/197070/commits/4723ced751f0e5114a9bc7a2928dcf0cb326472e)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7206\r\n-\r\n[[d28c8c5](https://github.com/elastic/kibana/pull/197070/commits/d28c8c56f67802107c17a627357251b9eff797ba)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7209\r\n-\r\n[[dd7773a](https://github.com/elastic/kibana/pull/197070/commits/dd7773aebad5664e725c9849c0ed9418f9dc68ed)]\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7224\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"c31f11e7d8c5c586258399c5e702e2247e05d0e4"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Mike Côté <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves https://github.com/elastic/response-ops-team/issues/239
Summary
Deployed to cloud: deployment ID was
ab4e88d139f93d43024837d96144e7d4. Since the deployment ID starts with ana, this should start withmgetand I can see in the logs with the latest push that this is trueDeployed to serverless: project ID was
d33d22a94ce246d091220eace2c4e4bb. See in the logs:Using claim strategy mget as configured for deployment d33d22a94ce246d091220eace2c4e4bb