Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group7/exception_operators_data_types/double·ts - detection engine api security and spaces enabled - Group 7 Detection exceptions data types and operators Rule exception operators for data type double "is in list" operator working against string values in the data set will return 3 results if we have a list that includes 1 double #155122

Closed
Tracked by #161531
, #161506
...
kibanamachine opened this issue Apr 18, 2023 · 14 comments
Closed
Tracked by #161531
, #161506
...

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group7/exception_operators_data_types/double·ts - detection engine api security and spaces enabled - Group 7 Detection exceptions data types and operators Rule exception operators for data type double "is in list" operator working against string values in the data set will return 3 results if we have a list that includes 1 double #155122

kibanamachine opened this issue Apr 18, 2023 · 14 comments
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Apr 18, 2023 •
edited
Loading

A test failed on a tracked branch

Error: expected [ '1.1' ] to sort of equal [ '1.1', '1.2', '1.3' ]
    at Assertion.assert (expect.js:100:11)
    at Assertion.eql (expect.js:244:8)
    at Context.<anonymous> (double.ts:513:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.apply (wrap_function.js:73:16) {
  actual: '[\n  "1.1"\n]',
  expected: '[\n  "1.1"\n  "1.2"\n  "1.3"\n]',
  showDiff: true
}

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Apr 18, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Apr 18, 2023
@kibanamachine kibanamachine added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Apr 18, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Apr 18, 2023
@MadameSheema MadameSheema added Team:Detections and Resp Security Detection Response Team Team:Security Solution Platform Security Solution Platform Team labels Apr 18, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@yctercero yctercero removed their assignment May 9, 2023
@yctercero yctercero added Team:Detection Engine Security Solution Detection Engine Area and removed Team:Security Solution Platform Security Solution Platform Team labels May 14, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

mistic added a commit that referenced this issue May 17, 2023
@mistic
skip flaky suite (#155122)
9a84948
@mistic
Copy link
Member

mistic commented May 17, 2023

Skipped.

main: 9a84948

@WafaaNasr WafaaNasr self-assigned this Jul 10, 2023
@WafaaNasr WafaaNasr mentioned this issue Jul 10, 2023
Closed
@yctercero
Copy link
Contributor

Running through flakey test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2597

@yctercero
Copy link
Contributor

Flakey test runner passed all 100 runs. Will push up PR to unskip.

@yctercero yctercero mentioned this issue Jul 11, 2023
Merged
@yctercero
Copy link
Contributor

PR to unskip #161566

@kibanamachine kibanamachine reopened this Jul 19, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.9

@WafaaNasr WafaaNasr removed their assignment Jul 20, 2023
@WafaaNasr WafaaNasr mentioned this issue Jul 26, 2023
Closed
@WafaaNasr WafaaNasr self-assigned this Aug 1, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@dmlemeshko
Copy link
Member

/skip

@kibanamachine
Copy link
Contributor Author

Skipped

main: 50b993e

kibanamachine added a commit that referenced this issue Aug 1, 2023
@kibanamachine
skip failing test suite (#155122)
50b993e
ThomThomson pushed a commit to ThomThomson/kibana that referenced this issue Aug 1, 2023
@kibanamachine
skip failing test suite (elastic#155122)
bbd7b2d
@WafaaNasr
Copy link
Contributor

WafaaNasr commented Aug 2, 2023 •
edited
Loading

While investigating the three failing pipelines and examining the Artifacts, specifically target/test_failures/*.log, I discovered the root cause to be
Did not get an expected 200 "ok" when waiting for a list item (waitForListItem) yet. Retrying until we get a 200 "ok". body: {"message":"list_id: \"list_items.txt\" item of 1.0 does not exist","status_code":404}, status: 404

image

@WafaaNasr WafaaNasr mentioned this issue Aug 2, 2023
Merged
@WafaaNasr
Copy link
Contributor

Upon incorporating additional logs to pinpoint the underlying issue, it was ascertained that the test will persist in its attempts until the preceding error is rectified, and it was determined that the list_items.txt file contains the correct item once located.

The problem seemed to be associated with incomplete indexing of all indices signalsOpen.hits.hits when the test verifies against the anticipated value.

To investigate the matter, two iterations were conducted, totaling 490 attempts. During these iterations, logs, and warnings were deliberately inserted to facilitate the tracing of the issue.

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787

In this Atifact

image

WafaaNasr added a commit that referenced this issue Aug 9, 2023
@WafaaNasr
[Security Solution] [Flaky test] Fix double ftr test flakiness (#162990)
f48d422 
## Summary

- Addresses #155122 


## Investigations


 *First attempt*

> While investigating the three failing pipelines and examining the
Artifacts, specifically target/test_failures/*.log, I discovered the
root cause to be
> Did not get an expected 200 "ok" when waiting for a list item
(waitForListItem) yet. Retrying until we get a 200 "ok". body:
{"message":"list_id: \"list_items.txt\" item of 1.0 does not
exist","status_code":404}, status: 404
> 
>
![image](https://github.com/elastic/kibana/assets/12671903/ad3c4db9-e340-42c0-8a61-63e5387b9b83)
> 

*Second attempt*


> Upon incorporating additional logs to pinpoint the underlying issue,
it was realized that the test will persist in its attempts until the
preceding error is rectified, and it was determined that the
list_items.txt file contains the correct item once located.
> 
> The problem seemed to be associated with incomplete indexing of all
indices signalsOpen.hits.hits when the test verifies against the
anticipated value.
> 
> To investigate the matter, two iterations were conducted, totaling 490
attempts. During these iterations, logs, and warnings were deliberately
inserted to facilitate the tracing of the issue.
> 
>
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52
>
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787
> 
> In this
[Atifact](https://s3.amazonaws.com/buildkiteartifacts.com/e0f3970e-3a75-4621-919f-e6c773e2bb12/fedcad1e-0f36-4d55-bdc7-e1edbecf91fe/0189bb29-495f-4b7e-b34e-558ff6c02ce2/0189bb29-c258-4586-986a-2ac5e128eb52/target/test_failures/0189bb29-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)
> 
>
![image](https://github.com/elastic/kibana/assets/12671903/bc58d389-3586-4343-a9f9-9577331d4876)
> 
> 

## Solution 

By modifying the third parameter of the `await
waitForSignalsToBePresent(supertest, log, 1, [id]);` function to a value
of 3 instead of the initial 1, we signify the number of alerts we need
to await before conducting assertions. This change is made to
accommodate the assertion of three alerts being generated, accounting
for the exclusion of a single 1.0 value.
bryce-b pushed a commit to bryce-b/kibana that referenced this issue Aug 9, 2023
@WafaaNasr @bryce-b
[Security Solution] [Flaky test] Fix double ftr test flakiness (elast…
25a0551 
…ic#162990)

## Summary

- Addresses elastic#155122 


## Investigations


 *First attempt*

> While investigating the three failing pipelines and examining the
Artifacts, specifically target/test_failures/*.log, I discovered the
root cause to be
> Did not get an expected 200 "ok" when waiting for a list item
(waitForListItem) yet. Retrying until we get a 200 "ok". body:
{"message":"list_id: \"list_items.txt\" item of 1.0 does not
exist","status_code":404}, status: 404
> 
>
![image](https://github.com/elastic/kibana/assets/12671903/ad3c4db9-e340-42c0-8a61-63e5387b9b83)
> 

*Second attempt*


> Upon incorporating additional logs to pinpoint the underlying issue,
it was realized that the test will persist in its attempts until the
preceding error is rectified, and it was determined that the
list_items.txt file contains the correct item once located.
> 
> The problem seemed to be associated with incomplete indexing of all
indices signalsOpen.hits.hits when the test verifies against the
anticipated value.
> 
> To investigate the matter, two iterations were conducted, totaling 490
attempts. During these iterations, logs, and warnings were deliberately
inserted to facilitate the tracing of the issue.
> 
>
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52
>
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787
> 
> In this
[Atifact](https://s3.amazonaws.com/buildkiteartifacts.com/e0f3970e-3a75-4621-919f-e6c773e2bb12/fedcad1e-0f36-4d55-bdc7-e1edbecf91fe/0189bb29-495f-4b7e-b34e-558ff6c02ce2/0189bb29-c258-4586-986a-2ac5e128eb52/target/test_failures/0189bb29-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)
> 
>
![image](https://github.com/elastic/kibana/assets/12671903/bc58d389-3586-4343-a9f9-9577331d4876)
> 
> 

## Solution 

By modifying the third parameter of the `await
waitForSignalsToBePresent(supertest, log, 1, [id]);` function to a value
of 3 instead of the initial 1, we signify the number of alerts we need
to await before conducting assertions. This change is made to
accommodate the assertion of three alerts being generated, accounting
for the exclusion of a single 1.0 value.
WafaaNasr added a commit to WafaaNasr/kibana that referenced this issue Aug 23, 2023
@WafaaNasr
[Security Solution] [Flaky test] Fix double ftr test flakiness (elast…
1d30929 
…ic#162990)

## Summary

- Addresses elastic#155122

## Investigations

 *First attempt*

> While investigating the three failing pipelines and examining the
Artifacts, specifically target/test_failures/*.log, I discovered the
root cause to be
> Did not get an expected 200 "ok" when waiting for a list item
(waitForListItem) yet. Retrying until we get a 200 "ok". body:
{"message":"list_id: \"list_items.txt\" item of 1.0 does not
exist","status_code":404}, status: 404
>
>
![image](https://github.com/elastic/kibana/assets/12671903/ad3c4db9-e340-42c0-8a61-63e5387b9b83)
>

*Second attempt*

> Upon incorporating additional logs to pinpoint the underlying issue,
it was realized that the test will persist in its attempts until the
preceding error is rectified, and it was determined that the
list_items.txt file contains the correct item once located.
>
> The problem seemed to be associated with incomplete indexing of all
indices signalsOpen.hits.hits when the test verifies against the
anticipated value.
>
> To investigate the matter, two iterations were conducted, totaling 490
attempts. During these iterations, logs, and warnings were deliberately
inserted to facilitate the tracing of the issue.
>
>
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52
>
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787
>
> In this
[Atifact](https://s3.amazonaws.com/buildkiteartifacts.com/e0f3970e-3a75-4621-919f-e6c773e2bb12/fedcad1e-0f36-4d55-bdc7-e1edbecf91fe/0189bb29-495f-4b7e-b34e-558ff6c02ce2/0189bb29-c258-4586-986a-2ac5e128eb52/target/test_failures/0189bb29-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)
>
>
![image](https://github.com/elastic/kibana/assets/12671903/bc58d389-3586-4343-a9f9-9577331d4876)
>
>

## Solution

By modifying the third parameter of the `await
waitForSignalsToBePresent(supertest, log, 1, [id]);` function to a value
of 3 instead of the initial 1, we signify the number of alerts we need
to await before conducting assertions. This change is made to
accommodate the assertion of three alerts being generated, accounting
for the exclusion of a single 1.0 value.

(cherry picked from commit f48d422)
WafaaNasr added a commit that referenced this issue Aug 24, 2023
@WafaaNasr
[8.9] [Security Solution] [Flaky test] Fix double ftr test flakiness (#…
4c62619 
…162990) (#164616)

# Backport

This will backport the following commits from `main` to `8.9`:
- [[Security Solution] [Flaky test] Fix double ftr test flakiness
(#162990)](#162990)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Wafaa
Nasr","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-08-09T14:40:55Z","message":"[Security
Solution] [Flaky test] Fix double ftr test flakiness (#162990)\n\n##
Summary\r\n\r\n- Addresses
#155122 \r\n\r\n\r\n##
Investigations\r\n\r\n\r\n *First attempt*\r\n\r\n> While investigating
the three failing pipelines and examining the\r\nArtifacts, specifically
target/test_failures/*.log, I discovered the\r\nroot cause to be\r\n>
Did not get an expected 200 \"ok\" when waiting for a list
item\r\n(waitForListItem) yet. Retrying until we get a 200 \"ok\".
body:\r\n{\"message\":\"list_id: \\\"list_items.txt\\\" item of 1.0 does
not\r\nexist\",\"status_code\":404}, status: 404\r\n>
\r\n>\r\n![image](https://github.com/elastic/kibana/assets/12671903/ad3c4db9-e340-42c0-8a61-63e5387b9b83)\r\n>
\r\n\r\n*Second attempt*\r\n\r\n\r\n> Upon incorporating additional logs
to pinpoint the underlying issue,\r\nit was realized that the test will
persist in its attempts until the\r\npreceding error is rectified, and
it was determined that the\r\nlist_items.txt file contains the correct
item once located.\r\n> \r\n> The problem seemed to be associated with
incomplete indexing of all\r\nindices signalsOpen.hits.hits when the
test verifies against the\r\nanticipated value.\r\n> \r\n> To
investigate the matter, two iterations were conducted, totaling
490\r\nattempts. During these iterations, logs, and warnings were
deliberately\r\ninserted to facilitate the tracing of the issue.\r\n>
\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787\r\n>
\r\n> In
this\r\n[Atifact](https://s3.amazonaws.com/buildkiteartifacts.com/e0f3970e-3a75-4621-919f-e6c773e2bb12/fedcad1e-0f36-4d55-bdc7-e1edbecf91fe/0189bb29-495f-4b7e-b34e-558ff6c02ce2/0189bb29-c258-4586-986a-2ac5e128eb52/target/test_failures/0189bb29-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)\r\n>
\r\n>\r\n![image](https://github.com/elastic/kibana/assets/12671903/bc58d389-3586-4343-a9f9-9577331d4876)\r\n>
\r\n> \r\n\r\n## Solution \r\n\r\nBy modifying the third parameter of
the `await\r\nwaitForSignalsToBePresent(supertest, log, 1, [id]);`
function to a value\r\nof 3 instead of the initial 1, we signify the
number of alerts we need\r\nto await before conducting assertions. This
change is made to\r\naccommodate the assertion of three alerts being
generated, accounting\r\nfor the exclusion of a single 1.0
value.","sha":"f48d422a2e5114e899df5697a18de10918beb779","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-failure-flaky","Feature:Rule
Exceptions","v8.9.0","Team:Detection
Engine","v8.10.0","v8.11.0"],"number":162990,"url":"https://github.com/elastic/kibana/pull/162990","mergeCommit":{"message":"[Security
Solution] [Flaky test] Fix double ftr test flakiness (#162990)\n\n##
Summary\r\n\r\n- Addresses
#155122 \r\n\r\n\r\n##
Investigations\r\n\r\n\r\n *First attempt*\r\n\r\n> While investigating
the three failing pipelines and examining the\r\nArtifacts, specifically
target/test_failures/*.log, I discovered the\r\nroot cause to be\r\n>
Did not get an expected 200 \"ok\" when waiting for a list
item\r\n(waitForListItem) yet. Retrying until we get a 200 \"ok\".
body:\r\n{\"message\":\"list_id: \\\"list_items.txt\\\" item of 1.0 does
not\r\nexist\",\"status_code\":404}, status: 404\r\n>
\r\n>\r\n![image](https://github.com/elastic/kibana/assets/12671903/ad3c4db9-e340-42c0-8a61-63e5387b9b83)\r\n>
\r\n\r\n*Second attempt*\r\n\r\n\r\n> Upon incorporating additional logs
to pinpoint the underlying issue,\r\nit was realized that the test will
persist in its attempts until the\r\npreceding error is rectified, and
it was determined that the\r\nlist_items.txt file contains the correct
item once located.\r\n> \r\n> The problem seemed to be associated with
incomplete indexing of all\r\nindices signalsOpen.hits.hits when the
test verifies against the\r\nanticipated value.\r\n> \r\n> To
investigate the matter, two iterations were conducted, totaling
490\r\nattempts. During these iterations, logs, and warnings were
deliberately\r\ninserted to facilitate the tracing of the issue.\r\n>
\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787\r\n>
\r\n> In
this\r\n[Atifact](https://s3.amazonaws.com/buildkiteartifacts.com/e0f3970e-3a75-4621-919f-e6c773e2bb12/fedcad1e-0f36-4d55-bdc7-e1edbecf91fe/0189bb29-495f-4b7e-b34e-558ff6c02ce2/0189bb29-c258-4586-986a-2ac5e128eb52/target/test_failures/0189bb29-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)\r\n>
\r\n>\r\n![image](https://github.com/elastic/kibana/assets/12671903/bc58d389-3586-4343-a9f9-9577331d4876)\r\n>
\r\n> \r\n\r\n## Solution \r\n\r\nBy modifying the third parameter of
the `await\r\nwaitForSignalsToBePresent(supertest, log, 1, [id]);`
function to a value\r\nof 3 instead of the initial 1, we signify the
number of alerts we need\r\nto await before conducting assertions. This
change is made to\r\naccommodate the assertion of three alerts being
generated, accounting\r\nfor the exclusion of a single 1.0
value.","sha":"f48d422a2e5114e899df5697a18de10918beb779"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/162990","number":162990,"mergeCommit":{"message":"[Security
Solution] [Flaky test] Fix double ftr test flakiness (#162990)\n\n##
Summary\r\n\r\n- Addresses
#155122 \r\n\r\n\r\n##
Investigations\r\n\r\n\r\n *First attempt*\r\n\r\n> While investigating
the three failing pipelines and examining the\r\nArtifacts, specifically
target/test_failures/*.log, I discovered the\r\nroot cause to be\r\n>
Did not get an expected 200 \"ok\" when waiting for a list
item\r\n(waitForListItem) yet. Retrying until we get a 200 \"ok\".
body:\r\n{\"message\":\"list_id: \\\"list_items.txt\\\" item of 1.0 does
not\r\nexist\",\"status_code\":404}, status: 404\r\n>
\r\n>\r\n![image](https://github.com/elastic/kibana/assets/12671903/ad3c4db9-e340-42c0-8a61-63e5387b9b83)\r\n>
\r\n\r\n*Second attempt*\r\n\r\n\r\n> Upon incorporating additional logs
to pinpoint the underlying issue,\r\nit was realized that the test will
persist in its attempts until the\r\npreceding error is rectified, and
it was determined that the\r\nlist_items.txt file contains the correct
item once located.\r\n> \r\n> The problem seemed to be associated with
incomplete indexing of all\r\nindices signalsOpen.hits.hits when the
test verifies against the\r\nanticipated value.\r\n> \r\n> To
investigate the matter, two iterations were conducted, totaling
490\r\nattempts. During these iterations, logs, and warnings were
deliberately\r\ninserted to facilitate the tracing of the issue.\r\n>
\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787\r\n>
\r\n> In
this\r\n[Atifact](https://s3.amazonaws.com/buildkiteartifacts.com/e0f3970e-3a75-4621-919f-e6c773e2bb12/fedcad1e-0f36-4d55-bdc7-e1edbecf91fe/0189bb29-495f-4b7e-b34e-558ff6c02ce2/0189bb29-c258-4586-986a-2ac5e128eb52/target/test_failures/0189bb29-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)\r\n>
\r\n>\r\n![image](https://github.com/elastic/kibana/assets/12671903/bc58d389-3586-4343-a9f9-9577331d4876)\r\n>
\r\n> \r\n\r\n## Solution \r\n\r\nBy modifying the third parameter of
the `await\r\nwaitForSignalsToBePresent(supertest, log, 1, [id]);`
function to a value\r\nof 3 instead of the initial 1, we signify the
number of alerts we need\r\nto await before conducting assertions. This
change is made to\r\naccommodate the assertion of three alerts being
generated, accounting\r\nfor the exclusion of a single 1.0
value.","sha":"f48d422a2e5114e899df5697a18de10918beb779"}},{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mistic @yctercero @dmlemeshko @WafaaNasr @elasticmachine @MadameSheema @kibanamachine