Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: SurveyProgressControllerTest is approximately 1% flaky #5338

Open
BenHenning opened this issue Feb 9, 2024 · 0 comments
Open

[BUG]: SurveyProgressControllerTest is approximately 1% flaky #5338

BenHenning opened this issue Feb 9, 2024 · 0 comments
Assignees
Labels
bug End user-perceivable behaviors which are not desirable. Impact: Low Low perceived user impact (e.g. edge cases). Work: Medium The means to find the solution is clear, but it isn't at good-first-issue level yet.

Comments

@BenHenning
Copy link
Member

Describe the bug

I found in #5335 that SurveyProgressControllerTest has a rare chance of flaking. I encountered 11 failures out of 1000 runs, with the following results:

6/11 failures:

testSubmitAnswer_forMarketFitQuestion_succeeds(org.oppia.android.domain.survey.SurveyProgressControllerTest)
java.lang.IllegalStateException: Expected next result to be a success, not: Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor.retrieveSuccess(DataProviderTestMonitor.kt:130)
        at org.oppia.android.testing.data.DataProviderTestMonitor.waitForNextSuccessResult(DataProviderTestMonitor.kt:66)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.waitForNextSuccessfulResult(DataProviderTestMonitor.kt:201)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.startSuccessfulSurveySession(SurveyProgressControllerTest.kt:432)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testSubmitAnswer_forMarketFitQuestion_succeeds(SurveyProgressControllerTest.kt:166)

2/11 failures:

testMoveToPreviousQuestion_afterMovingToNextQuestion_isSuccess(org.oppia.android.domain.survey.SurveyProgressControllerTest)
java.lang.IllegalStateException: Expected next result to be a success, not: Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor.retrieveSuccess(DataProviderTestMonitor.kt:130)
        at org.oppia.android.testing.data.DataProviderTestMonitor.waitForNextSuccessResult(DataProviderTestMonitor.kt:66)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.waitForNextSuccessfulResult(DataProviderTestMonitor.kt:201)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.submitAnswer(SurveyProgressControllerTest.kt:452)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.submitUserTypeAnswer(SurveyProgressControllerTest.kt:459)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testMoveToPreviousQuestion_afterMovingToNextQuestion_isSuccess(SurveyProgressControllerTest.kt:295)

1/11 failures:

testStartSurveySession_sessionStartsWithInitialQuestion(org.oppia.android.domain.survey.SurveyProgressControllerTest)
java.lang.IllegalStateException: Expected next result to be a success, not: Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor.retrieveSuccess(DataProviderTestMonitor.kt:130)
        at org.oppia.android.testing.data.DataProviderTestMonitor.waitForNextSuccessResult(DataProviderTestMonitor.kt:66)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.waitForNextSuccessfulResult(DataProviderTestMonitor.kt:201)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.startSuccessfulSurveySession(SurveyProgressControllerTest.kt:432)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testStartSurveySession_sessionStartsWithInitialQuestion(SurveyProgressControllerTest.kt:97)

1/11 failures:

testSubmitAnswer_afterMovingToPreviousQuestion_isSuccess(org.oppia.android.domain.survey.SurveyProgressControllerTest)
expected not to be an instance of: org.oppia.android.util.data.AsyncResult$Pending
but was                          : Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.ensureDataProviderExecutes(DataProviderTestMonitor.kt:189)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.moveToPreviousQuestion(SurveyProgressControllerTest.kt:445)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testSubmitAnswer_afterMovingToPreviousQuestion_isSuccess(SurveyProgressControllerTest.kt:313)

1/11 failures:

testSubmitAnswer_submitNpsScore0f3_loadsDetractorFeedbackQuestion(org.oppia.android.domain.survey.SurveyProgressControllerTest)
java.lang.IllegalStateException: Expected next result to be a success, not: Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor.retrieveSuccess(DataProviderTestMonitor.kt:130)
        at org.oppia.android.testing.data.DataProviderTestMonitor.waitForNextSuccessResult(DataProviderTestMonitor.kt:66)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.waitForNextSuccessfulResult(DataProviderTestMonitor.kt:201)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.startSuccessfulSurveySession(SurveyProgressControllerTest.kt:432)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testSubmitAnswer_submitNpsScore0f3_loadsDetractorFeedbackQuestion(SurveyProgressControllerTest.kt:238)

Steps To Reproduce

Check out the latest develop (018e9c9) and run the following:

bazel test --runs_per_test=1000 //domain/src/test/java/org/oppia/android/domain/survey:SurveyProgressControllerTest

Expected Behavior

No tests should flake.

Screenshots/Videos

N/A

What device/emulator are you using?

N/A

Which Android version is your device/emulator running?

N/A

Which version of the Oppia Android app are you using?

N/A

Additional Context

From the stack traces, this appears to be a timing issue. Upon inspecting the controller and its tests, I don't see an obvious reason as to why there are some syncing problems; more investigation is needed to determine a possible root cause (since finding a deterministic repro is probably going to be even harder to do).

@BenHenning BenHenning added bug End user-perceivable behaviors which are not desirable. triage needed labels Feb 9, 2024
@adhiamboperes adhiamboperes added Impact: Low Low perceived user impact (e.g. edge cases). Work: Medium The means to find the solution is clear, but it isn't at good-first-issue level yet. Impact: Medium Moderate perceived user impact (non-blocking bugs and general improvements). and removed Impact: Low Low perceived user impact (e.g. edge cases). labels Feb 12, 2024
@adhiamboperes adhiamboperes added Impact: Low Low perceived user impact (e.g. edge cases). and removed Impact: Medium Moderate perceived user impact (non-blocking bugs and general improvements). labels Mar 12, 2024
@adhiamboperes adhiamboperes self-assigned this Mar 12, 2024
@BenHenning BenHenning added this to the 1.0 Global availability milestone Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug End user-perceivable behaviors which are not desirable. Impact: Low Low perceived user impact (e.g. edge cases). Work: Medium The means to find the solution is clear, but it isn't at good-first-issue level yet.
Development

No branches or pull requests

3 participants