Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(cmd-api-server): fix flaky runtime-plugin-imports-test #1903

Merged
merged 1 commit into from
May 10, 2022

Conversation

Leeyoungone
Copy link
Contributor

@Leeyoungone Leeyoungone commented Mar 8, 2022

This is a flaky test case that can't be reliably reproduce. I've ran the test individually 40 times and all 40 times it has passed successfully.

This error, however, seems to pop up only when the yarn jest command is run (when I ran the test individually).

Based on the linked "hint" from the issue, it seems like using jest.useFakeTimers() a good solution. However, using the jest.useFakeTimers() withasync/await was not recommended.

However, I found a different source that showed examples of how to use the jest.useFakeTimers(). Our original problem, however, is probably because the after the test environment is torn down, the second test file is trying to import files from the first environment, triggering the error (explained here). So I've come to the solution of installing another node package: node-cleanup and running that within the afterAll at the very end of the test.

My laptop can't take running the yarn jest script too much so I haven't seen it reproduce the error yet. (WOT ruhrow)

_______________________edit below _________________________

So it turns out that this flaky test due to libraries trying to third libraries trying to run after the test runs is a simple fix that was documented here where we just add the useRealTimer in the afterEach. Lol so sorry!!

Fixes #1667

@petermetz petermetz added Flaky-Test-Automation Issues related to test stability (which is a long running issue that can never fully be solved) Tests Anything related to tests be that automatic or manual, integration or unit, etc. labels Mar 11, 2022
Copy link
Contributor

@petermetz petermetz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Leeyoungone Great investigation I'm pretty sure you are on the right track. I read the documentation of node-cleanup and the way you are using it now seems to be a no-op to me because the purpose of the library is to provide hooks where the user (who is the person writing the Cactus code in this case) can do the "cleanup" logic themselves if they know that some allocated resource (TCP socket, file handle, database lock, etc.) is still being held onto. The point is, the node-cleanup library doesn't magically know what to clean up, it just gives you the opportunity to do so yourself (and right now what you technically do when you invoke the library's function is just saying that you want do the cleanup, but then you aren't).
So in that sense, I think this warrants a little more investigation, but you are definitely on the right track!

@Leeyoungone Leeyoungone force-pushed the test-1667 branch 3 times, most recently from 3628838 to bb3b4d7 Compare March 25, 2022 14:37
@Leeyoungone
Copy link
Contributor Author

@petermetz Thank you so much for the advice! I've dug into it a little more and I think that adding the useRealTimers within the afterEach should fix this flaky issue (as mentioned here). My laptop with 8 GB of RAM isn't strong enough to withstand running all the jest tests to replicate this flaky issue. I might just have to Hail Mary this PR and hope this solves it.

@Leeyoungone Leeyoungone force-pushed the test-1667 branch 2 times, most recently from e9332c9 to 133d6a1 Compare March 25, 2022 14:57
@Leeyoungone Leeyoungone requested a review from petermetz March 28, 2022 14:36
package.json Outdated Show resolved Hide resolved
yarn.lock Outdated Show resolved Hide resolved
@Leeyoungone Leeyoungone force-pushed the test-1667 branch 2 times, most recently from 4c2c21a to 125625f Compare April 5, 2022 19:02
@Leeyoungone Leeyoungone requested a review from petermetz April 6, 2022 12:47
Copy link
Contributor

@petermetz petermetz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Leeyoungone Hmm, yarn.lock is still in diff if you look here: https://github.com/hyperledger/cactus/pull/1903/files
Did you forget to push your changes maybe?

@Leeyoungone Leeyoungone force-pushed the test-1667 branch 2 times, most recently from 5a91aa3 to 0f43596 Compare April 25, 2022 12:38
@Leeyoungone Leeyoungone force-pushed the test-1667 branch 2 times, most recently from 4006281 to b9de425 Compare May 9, 2022 16:18
@Leeyoungone Leeyoungone force-pushed the test-1667 branch 2 times, most recently from 36e1bc4 to 4d1c26c Compare May 9, 2022 16:27
@Leeyoungone Leeyoungone requested a review from petermetz May 9, 2022 16:27
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

=======================================================================

This is a flaky test case that can't be reliably reproduce. I've ran the
test individually 40 times and all 40 times it has passed successfully.

This error, however, seems to pop up only when the `yarn jest` command
is run (when I ran the test individually).

Based on the linked
["hint"](https://stackoverflow.com/a/50793993)
from the issue, it seems like using `jest.useFakeTimers()` a good
solution. However, using the `jest.useFakeTimers()` with`async/await`
was not recommended.

However, I found a [different source]
(https://gist.github.com/apieceofbart/e6dea8d884d29cf88cdb54ef14ddbcc4#file-test-js-L58)
that showed examples
of how to use the `jest.useFakeTimers()`.
Our original problem, however, is probably because the after the test
environment is torn down, the second test file is trying to import
files from the first environment, triggering the error
([explained here](jestjs/jest#11438 (comment))).

So I've come to the solution of installing another node package:
[node-cleanup](https://www.npmjs.com/package/node-cleanup) and running
that within the afterAll at the very end of the test.

My laptop can't take running the `yarn jest` script too much so I
haven't seen it reproduce the error yet. (WOT ruhrow)

_______________________edit below _________________________

So it turns out that this flaky test due to libraries trying to third
libraries trying to run after the test runs is a simple fix that was
documented [here](https://testing-library.com/docs/using-fake-timers/)
where we just add the `useRealTimer` in the `afterEach`. Lol so sorry!!

=======================================================================

This is a PARTIAL resolution to issue hyperledger-cacti#238

Fixes hyperledger-cacti#1667

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>
Signed-off-by: Peter Somogyvari <[email protected]>
Copy link
Contributor

@petermetz petermetz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Leeyoungone LGTM, thank you!

@petermetz petermetz merged commit 893ddd1 into hyperledger-cacti:main May 10, 2022
@petermetz petermetz deleted the test-1667 branch May 10, 2022 01:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Flaky-Test-Automation Issues related to test stability (which is a long running issue that can never fully be solved) Tests Anything related to tests be that automatic or manual, integration or unit, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test(cmd-api-server): fix flaky runtime-plugin-imports test
4 participants