Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Security Solution Cypress.x-pack/plugins/security_solution/public/management/cypress/e2e/automated_response_actions/automated_response_actions·cy·ts - Automated Response Actions From alerts "after all" hook for "should have generated endpoint and rule" "after all" hook for "should have generated endpoint and rule" #168427

Closed
kibanamachine opened this issue Oct 10, 2023 · 81 comments · Fixed by #168438, #173589 or #186073
Closed

Failing test: Security Solution Cypress.x-pack/plugins/security_solution/public/management/cypress/e2e/automated_response_actions/automated_response_actions·cy·ts - Automated Response Actions From alerts "after all" hook for "should have generated endpoint and rule" "after all" hook for "should have generated endpoint and rule" #168427

kibanamachine opened this issue Oct 10, 2023 · 81 comments · Fixed by #168438, #173589 or #186073
Assignees
@szwarckonrad
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Defend Workflows “EDR Workflows” sub-team of Security Solution

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 10, 2023 •
edited
Loading

A test failed on a tracked branch

CypressError: `cy.task('deleteIndexedFleetEndpointPolicies')` failed with the following error:

> Request failed with status code 500

https://on.cypress.io/api/task

Because this error occurred during a `after all` hook we are skipping the remaining tests in the current suite: `Automated Response Actions`

Although you have test retries enabled, we do not retry tests when `before all` or `after all` hooks fail
    at <unknown> (http://localhost:5689/__cypress/runner/cypress_runner.js:135296:70)
    at tryCatcher (http://localhost:5689/__cypress/runner/cypress_runner.js:1807:23)
    at Promise._settlePromiseFromHandler (http://localhost:5689/__cypress/runner/cypress_runner.js:1519:31)
    at Promise._settlePromise (http://localhost:5689/__cypress/runner/cypress_runner.js:1576:18)
    at Promise._settlePromise0 (http://localhost:5689/__cypress/runner/cypress_runner.js:1621:10)
    at Promise._settlePromises (http://localhost:5689/__cypress/runner/cypress_runner.js:1697:18)
    at _drainQueueStep (http://localhost:5689/__cypress/runner/cypress_runner.js:2407:12)
    at _drainQueue (http://localhost:5689/__cypress/runner/cypress_runner.js:2400:9)
    at Async._drainQueues (http://localhost:5689/__cypress/runner/cypress_runner.js:2416:5)
    at Async.drainQueues (http://localhost:5689/__cypress/runner/cypress_runner.js:2286:14)
From Your Spec Code:
    at Context.eval (webpack:///./e2e/automated_response_actions/automated_response_actions.cy.ts:53:11)

From Node.js Internals:
  Error: Request failed with status code 500
      at wrapErrorIfNeeded (/opt/local-ssd/buildkite/builds/kb-n2-4-virt-c687cf453f8a221c/elastic/kibana-on-merge/kibana/x-pack/plugins/security_solution/common/endpoint/data_loaders/utils.ts:25:7)
      at wrapErrorAndRejectPromise (/opt/local-ssd/buildkite/builds/kb-n2-4-virt-c687cf453f8a221c/elastic/kibana-on-merge/kibana/x-pack/plugins/security_solution/common/endpoint/data_loaders/utils.ts:28:92)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async deleteIndexedFleetEndpointPolicies (/opt/local-ssd/buildkite/builds/kb-n2-4-virt-c687cf453f8a221c/elastic/kibana-on-merge/kibana/x-pack/plugins/security_solution/common/endpoint/data_loaders/index_fleet_endpoint_policy.ts:166:12)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Oct 10, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Oct 10, 2023
@kibanamachine kibanamachine added the Team:Defend Workflows “EDR Workflows” sub-team of Security Solution label Oct 10, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Oct 10, 2023
@szwarckonrad szwarckonrad mentioned this issue Oct 10, 2023
Merged
@szwarckonrad szwarckonrad self-assigned this Oct 11, 2023
szwarckonrad added a commit that referenced this issue Oct 11, 2023
@szwarckonrad
[EDR Workflows][E2E] Increase the timeout of agent check in (#168438)
91cdbe2 
This pull request extends the agent fleet check timeout from 2 minutes
to 4 minutes. We've identified a number of unreliable tests that fail
during the `beforeAll` stage while executing the `createEndpointHost`
task. The following logs appear before the timeout:

```
info Enrolling Elastic Agent with Fleet
  | Installing service....... DONE
  | Starting service... DONE
  | Enrolling Elastic Agent with Fleet..........Successfully enrolled the Elastic Agent.
  | Elastic Agent has been successfully installed.
  | info Waiting for Agent to check-in with Fleet
```

The error message we encounter is `> Timed out waiting for host
[test-host-4981] to appear in Fleet.`

It appears that all the preceding steps are successful, and only the
final one fails due to either the agent not checking in with the fleet
for 2 minutes or the agent being unhealthy for two minutes. Since I
haven't been able to replicate this behavior locally, and there isn't a
way to inspect what's happening on the agent, I believe the best course
of action at this point is to extend the timeout and monitor the
results.

Reports of this error:
closes #168427
closes #168394
closes #168393
closes #168390
closes #168363
closes #168362
closes #168361
closes #168360
closes #168359

Affected CI runs:
https://buildkite.com/elastic/kibana-on-merge/builds/36483
https://buildkite.com/elastic/kibana-on-merge/builds/36497
https://buildkite.com/elastic/kibana-on-merge/builds/36501
https://buildkite.com/elastic/kibana-on-merge/builds/36526

Another time out happens from time to time when previously set 10
minutes timeout on `createEndpointHost` task is not enough to set up the
environment. Its portrayed below, timeout happens during agent setup
```
  | default: Running: inline script
  | default: Reading package lists...
  | default: Building dependency tree...
  | default: Reading state information...
  | default: Suggested packages:
  | default:   zip
  | default: The following NEW packages will be installed:
  | default:   unzip
  | default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
  | default: Need to get 174 kB of archives.
  | default: After this operation, 385 kB of additional disk space will be used.
  | default: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB]
  | default: dpkg-preconfigure: unable to re-open stdin: No such file or directory
  | default: Fetched 174 kB in 1s (210 kB/s)
  | default: Selecting previously unselected package unzip.
  | (Reading database ... 63961 files and directories currently installed.)
  | default: Preparing to unpack .../unzip_6.0-26ubuntu3.1_amd64.deb ...
  | default: Unpacking unzip (6.0-26ubuntu3.1) ...
  | default: Setting up unzip (6.0-26ubuntu3.1) ...
  | default: Processing triggers for man-db (2.10.2-1) ...
  |  
  | CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`.
```
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 11, 2023
@szwarckonrad
[EDR Workflows][E2E] Increase the timeout of agent check in (elastic#…
de5b076 
…168438)

This pull request extends the agent fleet check timeout from 2 minutes
to 4 minutes. We've identified a number of unreliable tests that fail
during the `beforeAll` stage while executing the `createEndpointHost`
task. The following logs appear before the timeout:

```
info Enrolling Elastic Agent with Fleet
  | Installing service....... DONE
  | Starting service... DONE
  | Enrolling Elastic Agent with Fleet..........Successfully enrolled the Elastic Agent.
  | Elastic Agent has been successfully installed.
  | info Waiting for Agent to check-in with Fleet
```

The error message we encounter is `> Timed out waiting for host
[test-host-4981] to appear in Fleet.`

It appears that all the preceding steps are successful, and only the
final one fails due to either the agent not checking in with the fleet
for 2 minutes or the agent being unhealthy for two minutes. Since I
haven't been able to replicate this behavior locally, and there isn't a
way to inspect what's happening on the agent, I believe the best course
of action at this point is to extend the timeout and monitor the
results.

Reports of this error:
closes elastic#168427
closes elastic#168394
closes elastic#168393
closes elastic#168390
closes elastic#168363
closes elastic#168362
closes elastic#168361
closes elastic#168360
closes elastic#168359

Affected CI runs:
https://buildkite.com/elastic/kibana-on-merge/builds/36483
https://buildkite.com/elastic/kibana-on-merge/builds/36497
https://buildkite.com/elastic/kibana-on-merge/builds/36501
https://buildkite.com/elastic/kibana-on-merge/builds/36526

Another time out happens from time to time when previously set 10
minutes timeout on `createEndpointHost` task is not enough to set up the
environment. Its portrayed below, timeout happens during agent setup
```
  | default: Running: inline script
  | default: Reading package lists...
  | default: Building dependency tree...
  | default: Reading state information...
  | default: Suggested packages:
  | default:   zip
  | default: The following NEW packages will be installed:
  | default:   unzip
  | default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
  | default: Need to get 174 kB of archives.
  | default: After this operation, 385 kB of additional disk space will be used.
  | default: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB]
  | default: dpkg-preconfigure: unable to re-open stdin: No such file or directory
  | default: Fetched 174 kB in 1s (210 kB/s)
  | default: Selecting previously unselected package unzip.
  | (Reading database ... 63961 files and directories currently installed.)
  | default: Preparing to unpack .../unzip_6.0-26ubuntu3.1_amd64.deb ...
  | default: Unpacking unzip (6.0-26ubuntu3.1) ...
  | default: Setting up unzip (6.0-26ubuntu3.1) ...
  | default: Processing triggers for man-db (2.10.2-1) ...
  |  
  | CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`.
```

(cherry picked from commit 91cdbe2)
kibanamachine referenced this issue Oct 11, 2023
@kibanamachine @szwarckonrad
[8.11] [EDR Workflows][E2E] Increase the timeout of agent check in (#…
356349b 
…168438) (#168614)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[EDR Workflows][E2E] Increase the timeout of agent check in
(#168438)](#168438)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Konrad
Szwarc","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-11T14:26:45Z","message":"[EDR
Workflows][E2E] Increase the timeout of agent check in (#168438)\n\nThis
pull request extends the agent fleet check timeout from 2 minutes\r\nto
4 minutes. We've identified a number of unreliable tests that
fail\r\nduring the `beforeAll` stage while executing the
`createEndpointHost`\r\ntask. The following logs appear before the
timeout:\r\n\r\n```\r\ninfo Enrolling Elastic Agent with Fleet\r\n |
Installing service....... DONE\r\n | Starting service... DONE\r\n |
Enrolling Elastic Agent with Fleet..........Successfully enrolled the
Elastic Agent.\r\n | Elastic Agent has been successfully installed.\r\n
| info Waiting for Agent to check-in with Fleet\r\n```\r\n\r\nThe error
message we encounter is `> Timed out waiting for
host\r\n[test-host-4981] to appear in Fleet.`\r\n\r\nIt appears that all
the preceding steps are successful, and only the\r\nfinal one fails due
to either the agent not checking in with the fleet\r\nfor 2 minutes or
the agent being unhealthy for two minutes. Since I\r\nhaven't been able
to replicate this behavior locally, and there isn't a\r\nway to inspect
what's happening on the agent, I believe the best course\r\nof action at
this point is to extend the timeout and monitor
the\r\nresults.\r\n\r\nReports of this error:\r\ncloses
https://github.com/elastic/kibana/issues/168427\r\ncloses
https://github.com/elastic/kibana/issues/168394\r\ncloses
https://github.com/elastic/kibana/issues/168393\r\ncloses
https://github.com/elastic/kibana/issues/168390\r\ncloses
https://github.com/elastic/kibana/issues/168363\r\ncloses
https://github.com/elastic/kibana/issues/168362\r\ncloses
https://github.com/elastic/kibana/issues/168361\r\ncloses
https://github.com/elastic/kibana/issues/168360\r\ncloses
https://github.com/elastic/kibana/issues/168359\r\n\r\nAffected CI
runs:\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36483\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36497\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36501\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36526\r\n\r\nAnother
time out happens from time to time when previously set 10\r\nminutes
timeout on `createEndpointHost` task is not enough to set up
the\r\nenvironment. Its portrayed below, timeout happens during agent
setup\r\n```\r\n  | default: Running: inline script\r\n  | default:
Reading package lists...\r\n  | default: Building dependency
tree...\r\n  | default: Reading state information...\r\n  | default:
Suggested packages:\r\n  | default: zip\r\n  | default: The following
NEW packages will be installed:\r\n  | default: unzip\r\n  | default: 0
upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\r\n  |
default: Need to get 174 kB of archives.\r\n  | default: After this
operation, 385 kB of additional disk space will be used.\r\n  | default:
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip
amd64 6.0-26ubuntu3.1 [174 kB]\r\n  | default: dpkg-preconfigure: unable
to re-open stdin: No such file or directory\r\n  | default: Fetched 174
kB in 1s (210 kB/s)\r\n  | default: Selecting previously unselected
package unzip.\r\n  | (Reading database ... 63961 files and directories
currently installed.)\r\n  | default: Preparing to unpack
.../unzip_6.0-26ubuntu3.1_amd64.deb ...\r\n  | default: Unpacking unzip
(6.0-26ubuntu3.1) ...\r\n  | default: Setting up unzip (6.0-26ubuntu3.1)
...\r\n  | default: Processing triggers for man-db (2.10.2-1) ...\r\n  |
 \r\n  | CypressError: `cy.task('createEndpointHost')` timed out after
waiting
`600000ms`.\r\n```","sha":"91cdbe2d354100683b5d8670de88e0b2cf665ba9","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Defend
Workflows","v8.11.0","v8.12.0"],"number":168438,"url":"https://github.com/elastic/kibana/pull/168438","mergeCommit":{"message":"[EDR
Workflows][E2E] Increase the timeout of agent check in (#168438)\n\nThis
pull request extends the agent fleet check timeout from 2 minutes\r\nto
4 minutes. We've identified a number of unreliable tests that
fail\r\nduring the `beforeAll` stage while executing the
`createEndpointHost`\r\ntask. The following logs appear before the
timeout:\r\n\r\n```\r\ninfo Enrolling Elastic Agent with Fleet\r\n |
Installing service....... DONE\r\n | Starting service... DONE\r\n |
Enrolling Elastic Agent with Fleet..........Successfully enrolled the
Elastic Agent.\r\n | Elastic Agent has been successfully installed.\r\n
| info Waiting for Agent to check-in with Fleet\r\n```\r\n\r\nThe error
message we encounter is `> Timed out waiting for
host\r\n[test-host-4981] to appear in Fleet.`\r\n\r\nIt appears that all
the preceding steps are successful, and only the\r\nfinal one fails due
to either the agent not checking in with the fleet\r\nfor 2 minutes or
the agent being unhealthy for two minutes. Since I\r\nhaven't been able
to replicate this behavior locally, and there isn't a\r\nway to inspect
what's happening on the agent, I believe the best course\r\nof action at
this point is to extend the timeout and monitor
the\r\nresults.\r\n\r\nReports of this error:\r\ncloses
https://github.com/elastic/kibana/issues/168427\r\ncloses
https://github.com/elastic/kibana/issues/168394\r\ncloses
https://github.com/elastic/kibana/issues/168393\r\ncloses
https://github.com/elastic/kibana/issues/168390\r\ncloses
https://github.com/elastic/kibana/issues/168363\r\ncloses
https://github.com/elastic/kibana/issues/168362\r\ncloses
https://github.com/elastic/kibana/issues/168361\r\ncloses
https://github.com/elastic/kibana/issues/168360\r\ncloses
https://github.com/elastic/kibana/issues/168359\r\n\r\nAffected CI
runs:\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36483\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36497\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36501\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36526\r\n\r\nAnother
time out happens from time to time when previously set 10\r\nminutes
timeout on `createEndpointHost` task is not enough to set up
the\r\nenvironment. Its portrayed below, timeout happens during agent
setup\r\n```\r\n  | default: Running: inline script\r\n  | default:
Reading package lists...\r\n  | default: Building dependency
tree...\r\n  | default: Reading state information...\r\n  | default:
Suggested packages:\r\n  | default: zip\r\n  | default: The following
NEW packages will be installed:\r\n  | default: unzip\r\n  | default: 0
upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\r\n  |
default: Need to get 174 kB of archives.\r\n  | default: After this
operation, 385 kB of additional disk space will be used.\r\n  | default:
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip
amd64 6.0-26ubuntu3.1 [174 kB]\r\n  | default: dpkg-preconfigure: unable
to re-open stdin: No such file or directory\r\n  | default: Fetched 174
kB in 1s (210 kB/s)\r\n  | default: Selecting previously unselected
package unzip.\r\n  | (Reading database ... 63961 files and directories
currently installed.)\r\n  | default: Preparing to unpack
.../unzip_6.0-26ubuntu3.1_amd64.deb ...\r\n  | default: Unpacking unzip
(6.0-26ubuntu3.1) ...\r\n  | default: Setting up unzip (6.0-26ubuntu3.1)
...\r\n  | default: Processing triggers for man-db (2.10.2-1) ...\r\n  |
 \r\n  | CypressError: `cy.task('createEndpointHost')` timed out after
waiting
`600000ms`.\r\n```","sha":"91cdbe2d354100683b5d8670de88e0b2cf665ba9"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/168438","number":168438,"mergeCommit":{"message":"[EDR
Workflows][E2E] Increase the timeout of agent check in (#168438)\n\nThis
pull request extends the agent fleet check timeout from 2 minutes\r\nto
4 minutes. We've identified a number of unreliable tests that
fail\r\nduring the `beforeAll` stage while executing the
`createEndpointHost`\r\ntask. The following logs appear before the
timeout:\r\n\r\n```\r\ninfo Enrolling Elastic Agent with Fleet\r\n |
Installing service....... DONE\r\n | Starting service... DONE\r\n |
Enrolling Elastic Agent with Fleet..........Successfully enrolled the
Elastic Agent.\r\n | Elastic Agent has been successfully installed.\r\n
| info Waiting for Agent to check-in with Fleet\r\n```\r\n\r\nThe error
message we encounter is `> Timed out waiting for
host\r\n[test-host-4981] to appear in Fleet.`\r\n\r\nIt appears that all
the preceding steps are successful, and only the\r\nfinal one fails due
to either the agent not checking in with the fleet\r\nfor 2 minutes or
the agent being unhealthy for two minutes. Since I\r\nhaven't been able
to replicate this behavior locally, and there isn't a\r\nway to inspect
what's happening on the agent, I believe the best course\r\nof action at
this point is to extend the timeout and monitor
the\r\nresults.\r\n\r\nReports of this error:\r\ncloses
https://github.com/elastic/kibana/issues/168427\r\ncloses
https://github.com/elastic/kibana/issues/168394\r\ncloses
https://github.com/elastic/kibana/issues/168393\r\ncloses
https://github.com/elastic/kibana/issues/168390\r\ncloses
https://github.com/elastic/kibana/issues/168363\r\ncloses
https://github.com/elastic/kibana/issues/168362\r\ncloses
https://github.com/elastic/kibana/issues/168361\r\ncloses
https://github.com/elastic/kibana/issues/168360\r\ncloses
https://github.com/elastic/kibana/issues/168359\r\n\r\nAffected CI
runs:\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36483\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36497\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36501\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36526\r\n\r\nAnother
time out happens from time to time when previously set 10\r\nminutes
timeout on `createEndpointHost` task is not enough to set up
the\r\nenvironment. Its portrayed below, timeout happens during agent
setup\r\n```\r\n  | default: Running: inline script\r\n  | default:
Reading package lists...\r\n  | default: Building dependency
tree...\r\n  | default: Reading state information...\r\n  | default:
Suggested packages:\r\n  | default: zip\r\n  | default: The following
NEW packages will be installed:\r\n  | default: unzip\r\n  | default: 0
upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\r\n  |
default: Need to get 174 kB of archives.\r\n  | default: After this
operation, 385 kB of additional disk space will be used.\r\n  | default:
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip
amd64 6.0-26ubuntu3.1 [174 kB]\r\n  | default: dpkg-preconfigure: unable
to re-open stdin: No such file or directory\r\n  | default: Fetched 174
kB in 1s (210 kB/s)\r\n  | default: Selecting previously unselected
package unzip.\r\n  | (Reading database ... 63961 files and directories
currently installed.)\r\n  | default: Preparing to unpack
.../unzip_6.0-26ubuntu3.1_amd64.deb ...\r\n  | default: Unpacking unzip
(6.0-26ubuntu3.1) ...\r\n  | default: Setting up unzip (6.0-26ubuntu3.1)
...\r\n  | default: Processing triggers for man-db (2.10.2-1) ...\r\n  |
 \r\n  | CypressError: `cy.task('createEndpointHost')` timed out after
waiting
`600000ms`.\r\n```","sha":"91cdbe2d354100683b5d8670de88e0b2cf665ba9"}}]}]
BACKPORT-->

Co-authored-by: Konrad Szwarc <[email protected]>
@kibanamachine kibanamachine reopened this Oct 12, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

mistic added a commit that referenced this issue Oct 12, 2023
@mistic
skip flaky suite (#168427)
60f530f
@mistic
Copy link
Member

mistic commented Oct 12, 2023

Skipped.

main: 60f530f

dej611 pushed a commit to dej611/kibana that referenced this issue Oct 17, 2023
@szwarckonrad @dej611
[EDR Workflows][E2E] Increase the timeout of agent check in (elastic#…
745d267 
…168438)

This pull request extends the agent fleet check timeout from 2 minutes
to 4 minutes. We've identified a number of unreliable tests that fail
during the `beforeAll` stage while executing the `createEndpointHost`
task. The following logs appear before the timeout:

```
info Enrolling Elastic Agent with Fleet
  | Installing service....... DONE
  | Starting service... DONE
  | Enrolling Elastic Agent with Fleet..........Successfully enrolled the Elastic Agent.
  | Elastic Agent has been successfully installed.
  | info Waiting for Agent to check-in with Fleet
```

The error message we encounter is `> Timed out waiting for host
[test-host-4981] to appear in Fleet.`

It appears that all the preceding steps are successful, and only the
final one fails due to either the agent not checking in with the fleet
for 2 minutes or the agent being unhealthy for two minutes. Since I
haven't been able to replicate this behavior locally, and there isn't a
way to inspect what's happening on the agent, I believe the best course
of action at this point is to extend the timeout and monitor the
results.

Reports of this error:
closes elastic#168427
closes elastic#168394
closes elastic#168393
closes elastic#168390
closes elastic#168363
closes elastic#168362
closes elastic#168361
closes elastic#168360
closes elastic#168359

Affected CI runs:
https://buildkite.com/elastic/kibana-on-merge/builds/36483
https://buildkite.com/elastic/kibana-on-merge/builds/36497
https://buildkite.com/elastic/kibana-on-merge/builds/36501
https://buildkite.com/elastic/kibana-on-merge/builds/36526

Another time out happens from time to time when previously set 10
minutes timeout on `createEndpointHost` task is not enough to set up the
environment. Its portrayed below, timeout happens during agent setup
```
  | default: Running: inline script
  | default: Reading package lists...
  | default: Building dependency tree...
  | default: Reading state information...
  | default: Suggested packages:
  | default:   zip
  | default: The following NEW packages will be installed:
  | default:   unzip
  | default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
  | default: Need to get 174 kB of archives.
  | default: After this operation, 385 kB of additional disk space will be used.
  | default: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB]
  | default: dpkg-preconfigure: unable to re-open stdin: No such file or directory
  | default: Fetched 174 kB in 1s (210 kB/s)
  | default: Selecting previously unselected package unzip.
  | (Reading database ... 63961 files and directories currently installed.)
  | default: Preparing to unpack .../unzip_6.0-26ubuntu3.1_amd64.deb ...
  | default: Unpacking unzip (6.0-26ubuntu3.1) ...
  | default: Setting up unzip (6.0-26ubuntu3.1) ...
  | default: Processing triggers for man-db (2.10.2-1) ...
  |  
  | CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`.
```
dej611 pushed a commit to dej611/kibana that referenced this issue Oct 17, 2023
@mistic @dej611
skip flaky suite (elastic#168427)
daa9f28
@szwarckonrad szwarckonrad mentioned this issue Oct 25, 2023
Merged
szwarckonrad added a commit that referenced this issue Oct 25, 2023
@szwarckonrad @patrykkopycinski
[EDR Workflows][E2E] Recreate agent on createEndpointHost task fail (#…
1fdcb41 
…169092)

Restart vagrant vm on error during `beforeAll` task `createEndpointHost`

Defend Workflows Cypress suite ran 300 times through flaky test runner:
1. 100x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699
2. 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3707
3. 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3708
4. 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3709
5. 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3710


Flaky test runner runs with `createEndpointHost` task failure with
successful recovery:
1.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3710#018b62fd-9ae9-4988-b1e0-ab0f04d8efdc
2.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3710#018b62fd-9ae6-4340-992b-1474ee0f114b
3.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3708#018b62fd-578e-4817-ae1c-8c58e8774eec
4.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3708#018b62fd-5787-4245-85a6-cb446e42bc73
5.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3707#018b62fc-fc17-407e-88de-d0b43b6b1d44
(failed due to unrelated issue)
6.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2c3-430c-b3e3-72b9fbb22d24
7.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2c6-4315-b828-b3218a70f209
8.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2c7-4ff7-9a70-7354f90179e0
9.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2d7-418f-b043-049e5effb26f
10.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2da-47cc-b4ea-a4d4de3ba0a0

New errors not spotted before that got to do with env set up:

1. `vagrant up` failed:
1.1
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3708#018b62fd-5787-4245-85a6-cb446e42bc73
1.2
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2d0-4a52-87d9-34caa8927465

2. `CypressError: `cy.task('indexFleetEndpointPolicy')` timed out after
waiting `60000ms`.:
2.1
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3707#018b62fc-fc04-40d4-b155-46f094681edb
2.2
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2c9-4ebb-9174-eb9d79d04d02
2.3
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2dc-438f-94b0-9f94ae95701c
    

Closes:
#168284
#169343
#169468
#169469
#169467
#169465
#169466
#169157
#168719
#168427
#168359
#168340
#169689

---------

Co-authored-by: Patryk Kopyciński <[email protected]>
@kibanamachine kibanamachine reopened this Dec 15, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

Skipped

main: eefd9f0

kibanamachine added a commit that referenced this issue Jun 11, 2024
@kibanamachine
skip failing test suite (#168427)
eefd9f0
ashokaditya added a commit to ashokaditya/kibana that referenced this issue Jun 12, 2024
@ashokaditya
Update automated_response_actions.cy.ts
1bb54d9 
closes elastic/issues/168427
closes elastic/issues/168340
@ashokaditya ashokaditya mentioned this issue Jun 12, 2024
Merged
2 tasks
ashokaditya added a commit that referenced this issue Jun 12, 2024
@ashokaditya
[SecuritySolution][Endpoint] Re-enable several skipped tests (#186073)
6a2e5a4 
## Summary

Re-enables skipped tests that were failing and were skipped because of
an Endpoint bug.

closes /issues/168427
closes /issues/168340
closes /issues/168719
closes /issues/168284
closes /issues/171444
closes /issues/170373
closes /issues/173464
closes /issues/173465

### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@szwarckonrad szwarckonrad

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Defend Workflows “EDR Workflows” sub-team of Security Solution
Projects
None yet
Milestone
No milestone
5 participants
@mistic @elasticmachine @szwarckonrad @kibanamachine @kevinlog