Adjustments in Settings UI tests to avoid flakyness

[jesmrec]

Issues: #2732 #2709

Some actions performed:

• Camera Uploads section:

For these tests, the whole preferences activity was loaded, taking advantage that the section is on the top (visible at a glance). Some tests were flaky because the activity takes a bit longer in loading, before the action over the camera uploads checkbox is done (this is something not desirable that the testing framework should take care). Therefore, the dialog warning users to set up correctly the camera uploads is not displayed, failing the matcher over the "OK" button.

As fix, the activity is only loaded with the camera uploads section.

• LocalFolderPicker activity

Here, the fix is a bit hacky. An implicit wait till the UI componentes are visible to launch the tests. Although there is a loop, it will not be infinite since the activityRule always is created.

• Passcode

The test for cancelling the activity was removed. It was being flaky due to isFinishing assertion. Such test is not a critical one (too far from this.. it only checks that a cancelled view is dismissed). So, better to get rid of.

These fixes were tested with:

• Samsung Galaxy S9, Android 9
• Nexus 6P , Android 7
• Huawei P20 Lite, Android 9

[hannesa2]

Sorry, I see no improvement here

[jesmrec]

After merging #2725, the Log menu UI has changed, so the tests were failing in the BitRise push job. Latest commit adapts it, by simple checks on the new options.

[jesmrec]

@hannesa2 i guess you trigger the tests in a local emulator. By executing in a real device, and also BitRise are working. Taking a look as well

[davigonz]

I've been executing the tests in a Nexus 6P emulator and there's some intent errors triggered during OCSettingsMoreTest execution, something like E/TestRunner: androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: Wanted to match 1 intents. Actually matched 0 intents. and java.lang.IllegalStateException: #init was called twice in a row

I've performed some changes there and it increased a bit the success rate.

Could you check them @jesmrec ? It would be great to have 100% even with emulator.

Anyway, this issue is about the Settings tests, not all of instrumented ones so first we want the Settings tests to be fixed and the other related to shares and capabilities are being fixed via #2728

So as soon as the Setting tests are not flacky anymore (no more OCSettingsWhatever tests failing when executing ./gradlew :owncloudApp:connectedDebugAndroidTest), we will merge them, close this issue and follow up
#2709 for the whole instrumented test suite of presentation layer (owncloudApp module)

[jesmrec]

I have get rid of the ugly implicit wait in the LocalFolderPicker suite, since it does not fix the problem and it is not desirable to have.

I have updated and performed more testing, with the following results:

• Real devices: i have tested with different devices from Android 7 to Android 10, and all the Settings tests passed correctly.

• CI: BitRise is passing correctly the tests over their emulators

• Local emulators: With Pixel emulators, tests are running correctly as well. But, other emulators like Nexus are running into issues. After researching a bit about it, i've realised that most of them are caused by some kind of weird race condition in the moment of loading the ActivityRule. Activity is loaded, but the first action (in most cases, a click) is not performed but there is no NuSuchElementException or similar either.

Which emulators are you using @davigonz @hannesa2? do you trigger directly with ./gradlew test or use additional options?

[davigonz]

Which emulators are you using @davigonz @hannesa2? do you trigger directly with ./gradlew test or use additional options?

@jesmrec ./gradlew :owncloudApp:connectedDebugAndroidTest with a Nexus6P API 28

[davigonz]

@jesmrec thanks for this detailed report, the work done here is enough imho. We can not guarantee the tests passing in all emulators for now, Bitrise was the blocker here and the tests are passing there now and also in at least, a local emulator. There seems to be a problem with Nexus 6P emulators.

Ugly code of the review was removed from the PR, and conversation was resolved.

[davigonz]

Tests are properly passing on Bitrise

[hannesa2]

For the record: I can't confirm to work properly on an Emulator

[jesmrec]

which emulator are you using? Pixel? Nexus? as i have written above, in different emulators i got different results. Also, it'd be helpful to know which tests are failing, since there are other PRs with fixed tests.

[hannesa2]

Ohh, I my idea was a independency from hardware profile, just a dependency from system-image. That's why I only wrote in previous mesage just system info, here is the complete info.
But maybe I'm wrong

[jesmrec]

my idea is the same, but... when things do not work, you have to know where the problem is (in order to fix it... ). Thanks for the additional info, i will take a look.