[Fix] flaky bulk disable task test #144739

ersin-erdal · 2022-11-07T19:10:03Z

pmuellr · 2022-11-11T04:55:40Z

I think this might be part of the problem. We throw away the results of the bulk update:

kibana/x-pack/plugins/alerting/server/rules_client/rules_client.ts

Line 2806 in e58e697

await this.taskManager.bulkDisable([attributes.scheduledTaskId]);

If you imagine that one of the updates fails, we'll never know.

We'll need to figure out what we want to return here. Probably a list of the tasks that were attempted to be disabled, and the result of trying to update them.

The reason an individual one might fail is OCC. The task SO doc will get updated when the task runs, so maybe there's some ordering where the task doc gets updated while the disable is taking place? That update will fail, but we don't return the status.

You can see OCC (with the version property) in use, and a refresh: false, here:

kibana/x-pack/plugins/task_manager/server/task_store.ts

Lines 219 to 227 in e58e697

    
           updatedSavedObject = await this.savedObjectsRepository.update<SerializedConcreteTaskInstance>( 
        
             'task', 
        
             doc.id, 
        
             attributes, 
        
             { 
        
               refresh: false, 
        
               version: doc.version, 
        
             } 
        
           );

Once we have a proper result for this API, I think this may need a more elaborate fix, where if we try to disable X rules but only Y end up disabled, then we retry the other ones a few times, like the other OCC wrappers we have. We could still have some failures with some tasks, but at least we'd retry a few times.

…49-flaky-test-aad

pmuellr · 2022-11-16T01:58:42Z

I opened #145316 since I think there's a chance that's what's causing the failures. I think we should probably block this one, till we get that issue resolved.

kibana-ci · 2022-11-17T15:48:05Z

💔 Build Failed

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should have proper callCluster and savedObjectsClient authorization for action type executor when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should have proper callCluster and savedObjectsClient authorization for action type executor when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should have proper callCluster and savedObjectsClient authorization for action type executor when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should have proper callCluster and savedObjectsClient authorization for action type executor when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should have proper callCluster and savedObjectsClient authorization for alert type executor when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should have proper callCluster and savedObjectsClient authorization for alert type executor when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should have proper callCluster and savedObjectsClient authorization for alert type executor when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should have proper callCluster and savedObjectsClient authorization for alert type executor when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should not throttle when changing groups
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should not throttle when changing groups
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should not throttle when changing groups
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should not throttle when changing groups
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should pass updated alert params to executor
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should pass updated alert params to executor
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should pass updated alert params to executor
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should pass updated alert params to executor
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should reset throttle window when not firing
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should reset throttle window when not firing
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should reset throttle window when not firing
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should reset throttle window when not firing
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule actions when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule actions when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule actions when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule actions when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule preconfigured actions when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule preconfigured actions when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule preconfigured actions when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule preconfigured actions when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should throttle alerts when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should throttle alerts when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should throttle alerts when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should throttle alerts when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 shouldn't schedule actions when alert is muted
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 shouldn't schedule actions when alert is muted
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 shouldn't schedule actions when alert is muted
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 shouldn't schedule actions when alert is muted
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should have proper callCluster and savedObjectsClient authorization for action type executor when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should have proper callCluster and savedObjectsClient authorization for action type executor when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should have proper callCluster and savedObjectsClient authorization for action type executor when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should have proper callCluster and savedObjectsClient authorization for action type executor when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should have proper callCluster and savedObjectsClient authorization for alert type executor when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should have proper callCluster and savedObjectsClient authorization for alert type executor when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should have proper callCluster and savedObjectsClient authorization for alert type executor when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should have proper callCluster and savedObjectsClient authorization for alert type executor when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should not throttle when changing groups
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should not throttle when changing groups
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should not throttle when changing groups
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should not throttle when changing groups
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should pass updated alert params to executor
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should pass updated alert params to executor
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should pass updated alert params to executor
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should pass updated alert params to executor
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should reset throttle window when not firing
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should reset throttle window when not firing
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should reset throttle window when not firing
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should reset throttle window when not firing
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule actions when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule actions when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule actions when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule actions when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule preconfigured actions when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule preconfigured actions when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule preconfigured actions when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule preconfigured actions when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should throttle alerts when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should throttle alerts when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should throttle alerts when appropriate
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should throttle alerts when appropriate
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should unmute all instances when unmuting an alert
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should unmute all instances when unmuting an alert
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should unmute all instances when unmuting an alert
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should unmute all instances when unmuting an alert
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 shouldn't schedule actions when alert instance is muted
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 shouldn't schedule actions when alert instance is muted
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 shouldn't schedule actions when alert instance is muted
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 shouldn't schedule actions when alert instance is muted
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 shouldn't schedule actions when alert is muted
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 shouldn't schedule actions when alert is muted
[job] [logs] FTR Configs #7 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 shouldn't schedule actions when alert is muted
[job] [logs] FTR Configs #9 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 shouldn't schedule actions when alert is muted
[job] [logs] FTR Configs #4 / alerting api integration spaces only Alerting scheduled task id for migrated rules - sets scheduled task id to match rule id when rule is disabled then enabled
[job] [logs] FTR Configs #4 / alerting api integration spaces only Alerting scheduled task id for migrated rules - sets scheduled task id to match rule id when rule is disabled then enabled
[job] [logs] Jest Tests #1 / disable() throws when unsecuredSavedObjectsClient update fails
[job] [logs] Jest Tests #1 / rules_client_conflict_retries 1 retry works for method disable
[job] [logs] Jest Tests #1 / rules_client_conflict_retries 3 retries fails with conflict error disable
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series basics Clicking on the chart should create a filter
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series basics Clicking on the chart should create a filter
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series basics Clicking on the chart should create a filter for series with multiple split by terms fields one of which has formatting
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series basics Clicking on the chart should create a filter for series with multiple split by terms fields one of which has formatting
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series Elastic charts Hiding series should hide series by legend item click
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series Elastic charts Hiding series should hide series by legend item click
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series Elastic charts Hiding series should keep series hidden after refresh
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series Elastic charts Hiding series should keep series hidden after refresh
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series Elastic charts should display correct chart data, label names and area colors for min aggregation when split by filters
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series Elastic charts should display correct chart data, label names and area colors for min aggregation when split by filters
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series Elastic charts should display correct chart data, label names and area colors for sum aggregation when split by terms
[job] [logs] test/functional/config.firefox.js / visualize app visual builder Time Series Elastic charts should display correct chart data, label names and area colors for sum aggregation when split by terms

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`osquery`	1	2	+1

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	19	21	+2
`fleet`	59	65	+6
`osquery`	108	113	+5
`securitySolution`	441	447	+6
total			+19

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	20	22	+2
`fleet`	67	73	+6
`osquery`	109	115	+6
`securitySolution`	518	524	+6
total			+20

History

💔 Build #88786 failed 2a5cc7e
💔 Build #88302 failed 5e95f84
💔 Build #88190 failed 52570ea
💔 Build #87974 failed 47cce4a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

ersin-erdal added 2 commits November 7, 2022 19:54

WIP

1cc9b2c

WIP

0fe944b

ersin-erdal added release_note:fix Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.6.0 labels Nov 7, 2022

move disable request in to retry

5344c0e

ersin-erdal changed the title ~~[Fix] bulk disable task flaky test~~ [Fix] flaky bulk disable task test Nov 8, 2022

ersin-erdal added 4 commits November 8, 2022 23:10

add index refresh after task disabling

e51d3f9

add index refresh regexp

1ffb302

move refresh in retry

8122118

test with refresh true

9becb70

ersin-erdal added 10 commits November 13, 2022 18:10

Merge branch 'main' into 141849-flaky-test-aad

2e41471

try with string rule enabled status

d10f718

compare booleans

f73ad04

compare booleans

e982be8

compare booleans

d9718e6

Merge remote-tracking branch 'origin/141849-flaky-test-aad' into 1418…

12d177b

…49-flaky-test-aad

fix test

a452e87

fix getDisableRequest response

2b2f4f7

refresh before getting the task

47cce4a

test task ids

52570ea

pmuellr mentioned this pull request Nov 16, 2022

[ResponseOps] results from taskManager.bulk*() calls ignored in rules client #145316

Closed

ersin-erdal added 4 commits November 16, 2022 07:59

AAD enabled

5e95f84

move rule to try catch

442aa31

move rule to try catch

2a5cc7e

return rule update error

d089198

ersin-erdal closed this Dec 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix] flaky bulk disable task test #144739

[Fix] flaky bulk disable task test #144739

ersin-erdal commented Nov 7, 2022

pmuellr commented Nov 11, 2022

pmuellr commented Nov 16, 2022

kibana-ci commented Nov 17, 2022 •

edited

Loading

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

[Fix] flaky bulk disable task test #144739

[Fix] flaky bulk disable task test #144739

Conversation

ersin-erdal commented Nov 7, 2022

pmuellr commented Nov 11, 2022

pmuellr commented Nov 16, 2022

kibana-ci commented Nov 17, 2022 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

kibana-ci commented Nov 17, 2022 •

edited

Loading