unreliable test: generateZipTestGen #6160

justinmk3 · 2024-12-05T18:25:10Z

Test Details

Name of test: generateZipTestGen
OS of failure: macOS
VSCode version of failure (minimum/insider/stable): minimum
Link to failing run: https://github.com/aws/aws-toolkit-vscode/actions/runs/12170063371/job/33944295842
Link to failing test:

aws-toolkit-vscode/packages/amazonq/test/unit/codewhisperer/util/zipUtil.test.ts

Line 117 in 3a39b10

describe('generateZipTestGen', function () {

Log of Test Failure

  1) zipUtil
       generateZipTestGen
         Should generate zip for test generation successfully:
     Error: done() called multiple times in test <zipUtil generateZipTestGen Should generate zip for test generation successfully> of file /Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/amazonq/dist/test/unit/codewhisperer/util/zipUtil.test.js; in addition, done() received error: EntryNotFound (FileSystemError): Error: ENOENT: no such file or directory, scandir '/test/zip/utgRequiredArtifactsDir'
    at Function.e (/private/tmp/.vscode-test/vscode-darwin-arm64-1.83.0/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:26740)
    at Object.readDirectory (/private/tmp/.vscode-test/vscode-darwin-arm64-1.83.0/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:24744)
    at async processDirectory (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:190:29)
    at async ZipUtil.processMetadataDir (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:212:9)
    at async ZipUtil.zipProject (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:230:13)
    at async ZipUtil.generateZipTestGen (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:602:41)
    at async Context.<anonymous> (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/amazonq/test/unit/codewhisperer/util/zipUtil.test.ts:159:28) {
  code: 'FileNotFound'

The text was updated successfully, but these errors were encountered:

Hweinstock · 2024-12-16T18:10:24Z

Another failure: https://github.com/aws/aws-toolkit-vscode/actions/runs/12358099519/job/34487842964?pr=6249

## Problem Flaky test `generateZipTestGen` sometimes fails with ``` 1) zipUtil generateZipTestGen Should generate zip for test generation successfully: Error: done() called multiple times in test <zipUtil generateZipTestGen Should generate zip for test generation successfully> of file /Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/amazonq/dist/test/unit/codewhisperer/util/zipUtil.test.js; in addition, done() received error: EntryNotFound (FileSystemError): Error: ENOENT: no such file or directory, scandir '/test/zip/utgRequiredArtifactsDir' at Function.e (/private/tmp/.vscode-test/vscode-darwin-arm64-1.83.0/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:26740) at Object.readDirectory (/private/tmp/.vscode-test/vscode-darwin-arm64-1.83.0/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:24744) at async processDirectory (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:190:29) at async ZipUtil.processMetadataDir (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:212:9) at async ZipUtil.zipProject (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:230:13) at async ZipUtil.generateZipTestGen (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:602:41) at async Context.<anonymous> (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/amazonq/test/unit/codewhisperer/util/zipUtil.test.ts:159:28) { code: 'FileNotFound' ``` Issue: #6160 ## Solution Based on the error, the test is trying to readdir on a mock dirpath even though the entire `zipProject` method should be stubbed. This can be confirmed by commenting out the following line: https://github.com/aws/aws-toolkit-vscode/blob/aa332da854c9d17ba258b459c622500911b1b0d9/packages/amazonq/test/unit/codewhisperer/util/zipUtil.test.ts#L135 which will give the same error ``` Error: ENOENT: no such file or directory, scandir '/test/zip/utgRequiredArtifactsDir' ``` So instead of stubbing which seems to be unreliable, we can let the test actually create the zip file. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

Hweinstock · 2025-01-07T15:19:05Z

Potential fix in #6290, please reopen if seen again.

Hweinstock · 2025-01-09T20:03:40Z

Reopening due to related failure:

Test Details

Name of test: generateZipTestGen
OS of failure: Windows
VSCode version of failure (minimum/insider/stable): insider
Link to failing run: https://github.com/aws/aws-toolkit-vscode/actions/runs/12696996583/job/35392271325
Link to failing test:

aws-toolkit-vscode/packages/amazonq/test/unit/codewhisperer/util/zipUtil.test.ts

Line 117 in 3a39b10

describe('generateZipTestGen', function () {

Log of Test Failure

  1) zipUtil
       generateZipTestGen
         Should handle file copy to downloads folder error:

      AssertionError [ERR_ASSERTION]: mkdir should have been called 4 times

5 !== 4

      + expected - actual

      -5
      +4

## Problem Flaky test `generateZipTestGen` sometimes fails with ``` 1) zipUtil generateZipTestGen Should generate zip for test generation successfully: Error: done() called multiple times in test <zipUtil generateZipTestGen Should generate zip for test generation successfully> of file /Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/amazonq/dist/test/unit/codewhisperer/util/zipUtil.test.js; in addition, done() received error: EntryNotFound (FileSystemError): Error: ENOENT: no such file or directory, scandir '/test/zip/utgRequiredArtifactsDir' at Function.e (/private/tmp/.vscode-test/vscode-darwin-arm64-1.83.0/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:26740) at Object.readDirectory (/private/tmp/.vscode-test/vscode-darwin-arm64-1.83.0/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:24744) at async processDirectory (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:190:29) at async ZipUtil.processMetadataDir (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:212:9) at async ZipUtil.zipProject (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:230:13) at async ZipUtil.generateZipTestGen (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/util/zipUtil.ts:602:41) at async Context.<anonymous> (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/amazonq/test/unit/codewhisperer/util/zipUtil.test.ts:159:28) { code: 'FileNotFound' ``` Issue: aws#6160 ## Solution Based on the error, the test is trying to readdir on a mock dirpath even though the entire `zipProject` method should be stubbed. This can be confirmed by commenting out the following line: https://github.com/aws/aws-toolkit-vscode/blob/aa332da854c9d17ba258b459c622500911b1b0d9/packages/amazonq/test/unit/codewhisperer/util/zipUtil.test.ts#L135 which will give the same error ``` Error: ENOENT: no such file or directory, scandir '/test/zip/utgRequiredArtifactsDir' ``` So instead of stubbing which seems to be unreliable, we can let the test actually create the zip file. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

#6482) ## Problem - Github issue: #6160 (comment) ## Solution - Not able to reproduce this in local machine. Double checked with team and planned to disable this test for now to avoid hassle for partner teams. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

justinmk3 added the tests-ci-cd label Dec 5, 2024

Hweinstock mentioned this issue Dec 10, 2024

unreliable test: zipUtil-generateZip-Should generate zip for file scan and return expected metadata #6203

Closed

justinmk3 mentioned this issue Dec 11, 2024

feat(cwl): Release CloudWatch Logs LiveTail feature. #6193

Merged

ctlai95 self-assigned this Dec 20, 2024

ctlai95 mentioned this issue Jan 2, 2025

test(amazonq): generateZipTestGen is unreliable #6290

Merged

Hweinstock closed this as completed Jan 7, 2025

Hweinstock reopened this Jan 9, 2025

laileni-aws mentioned this issue Feb 1, 2025

perf(amazonq): Removing flaky test case in generateZipTestGen function #6482

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unreliable test: generateZipTestGen #6160

unreliable test: generateZipTestGen #6160

justinmk3 commented Dec 5, 2024

Hweinstock commented Dec 16, 2024

Hweinstock commented Jan 7, 2025

Hweinstock commented Jan 9, 2025

unreliable test: generateZipTestGen #6160

unreliable test: generateZipTestGen #6160

Comments

justinmk3 commented Dec 5, 2024

Test Details

Log of Test Failure

Hweinstock commented Dec 16, 2024

Hweinstock commented Jan 7, 2025

Hweinstock commented Jan 9, 2025

Test Details

Log of Test Failure