Unable to run multiple tests in firebase

[ashwinichannaveerappa]

Steps to reproduce

Steps to Reproduce:

1. Build multiple _test.dart files
   example:command:
   patrol build android -v --target --target integration_test\tests\login_test.dart,integration_test\tests\settings_test.dart,integration_test\tests\forgot_password_test.dart,integration_test\tests\course_test.dart -dart-define=FLAVOR=dev

2. Run tests in firebase

gcloud firebase test android run --type instrumentation --app build/app/outputs/apk/dev/debug/app-dev-debug.apk --test build/app/outputs/apk/androidTest/dev/debug/app-dev-debug-androidTest.apk --device=model=redfin,version=30,orientation=portrait --timeout 45m

Actual results

login_test executed successfully. But other 3 testcases are failed.

io.grpc.StatusRuntimeException: UNKNOWN: 'package:patrol/src/native/patrol_app_service.dart': Failed assertion: line 129 pos 12: '_testExecutionCompleted.isCompleted == false': is not true.
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at pl.leancode.patrol.contracts.PatrolAppServiceGrpc$PatrolAppServiceBlockingStub.runDartTest(PatrolAppServiceGrpc.java:215)
at pl.leancode.patrol.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.java:40)
at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:128)
at pl.leancode.patrol.example.MainActivityTest.runDartTest(MainActivityTest.java:30)

Logs

Logs

<!-- Replace this line with your logs. Do not remove the backticks! -->

io.grpc.StatusRuntimeException: UNKNOWN: 'package:patrol/src/native/patrol_app_service.dart': Failed assertion: line 129 pos 12: '_testExecutionCompleted.isCompleted == false': is not true. at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) at pl.leancode.patrol.contracts.PatrolAppServiceGrpc$PatrolAppServiceBlockingStub.runDartTest(PatrolAppServiceGrpc.java:215) at pl.leancode.patrol.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.java:40) at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:128) at pl.leancode.patrol.example.MainActivityTest.runDartTest(MainActivityTest.java:30)

Patrol version

You're using Patrol CLI 2.0, which has breaking changes.
Read the migration guide at https://patrol.leancode.co/v2.
Disable this warning by setting the PATROL_MIGRATED environment variable.

patrol_cli v2.1.0

Patrol Doctor output

You're using Patrol CLI 2.0, which has breaking changes.
Read the migration guide at https://patrol.leancode.co/v2.
Disable this warning by setting the PATROL_MIGRATED environment variable.

Patrol CLI version: 2.1.0
Program adb found in C:\Users\achannav\scoop\shims\adb.exe
Env var $ANDROID_HOME set to C:\Users\achannav\AppData\Local\Android\sdk

Flutter Doctor output

[√] Flutter (Channel stable, 3.10.6, on Microsoft Windows [Version 10.0.19045.3208], locale en-US)
• Flutter version 3.10.6 on channel stable at C:\Users\achannav\flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision f468f3366c (4 weeks ago), 2023-07-12 15:19:05 -0700
• Engine revision cdbeda788a
• Dart version 3.0.6
• DevTools version 2.23.1

[√] Windows Version (Installed version of Windows is version 10 or higher)

[√] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
• Android SDK at C:\Users\achannav\AppData\Local\Android\sdk
• Platform android-33-ext4, build-tools 33.0.2
• ANDROID_HOME = C:\Users\achannav\AppData\Local\Android\sdk
• Java binary at: C:\Program Files\Android\Android Studio\jbr\bin\java
• Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-9505619)
• All Android licenses accepted.

[√] Chrome - develop for the web
• Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[X] Visual Studio - develop for Windows
X Visual Studio not installed; this is necessary for Windows development.
Download at https://visualstudio.microsoft.com/downloads/.
Please install the "Desktop development with C++" workload, including all of its default components

[√] Android Studio (version 2022.1)
• Android Studio at C:\Program Files\Android\Android Studio
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-9505619)

[√] IntelliJ IDEA Community Edition (version 2021.3)
• IntelliJ at C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2021.3
• Flutter plugin version 72.1.5
• Dart plugin version 231.8109.91

[√] VS Code (version 1.81.0)
• VS Code at C:\Users\achannav\AppData\Local\Programs\Microsoft VS Code
• Flutter extension version 3.70.0

[√] Connected device (3 available)
• Windows (desktop) • windows • windows-x64 • Microsoft Windows [Version 10.0.19045.3208]
• Chrome (web) • chrome • web-javascript • Google Chrome 115.0.5790.111
• Edge (web) • edge • web-javascript • Microsoft Edge 115.0.1901.188

[√] Network resources
• All expected network resources are available.

! Doctor found issues in 1 category.

[fylyppo]

Hi, you need to add --use-orchestrator to your gcloud firebase test android run command

[ashwinichannaveerappa]

Hi @fylyppo, thanks for the quick response, i tried with the --use-orchestrator in the command, as below.

gcloud firebase test android run --type instrumentation --app build/app/outputs/apk/dev/debug/app-dev-debug.apk --test build/app/outputs/apk/androidTest/dev/debug/app-dev-debug-androidTest.apk --device=model=redfin,version=30,orientation=portrait --use-orchestrator --ti meout 45m
but now all test cases are failed, not even one testcase executed completely.

[fylyppo]

Show me logs from FTL from each test

[ashwinichannaveerappa]

java.lang.AssertionError: Dart test failed: tests.course_test
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following PatrolActionException was thrown running a test:
Patrol action failed: GrpcError: enterTextByIndex() failed with
code NOT_FOUND (selector BySelector
[CLASS='\Qandroid.widget.EditText\E'] found nothing)

When the exception was thrown, this was the stack:
#0 NativeAutomator._wrapRequest (package:patrol/src/native/native_automator.dart:236:7)

#1 NativeAutomator.enterTextByIndex (package:patrol/src/native/native_automator.dart:621:5)

#2 BaseClass.loginWithValidCredentials (file:///C:/Users/achannav/redi_flutter/redi2-app/integration_test/screens/base_class.dart:25:5)

#3 main. (file:///C:/Users/achannav/redi_flutter/redi2-app/integration_test/tests/course_test.dart:18:3)

#4 patrolTest. (package:patrol/src/common.dart:129:7)

#5 testWidgets.. (package:flutter_test/src/widget_tester.dart:170:15)

#6 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:986:5)

The test description was:
Assigned Courses Test
═════════════════════════════════════════════════════════════════

at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:130)
at pl.leancode.patrol.example.MainActivityTest.runDartTest(MainActivityTest.java:30)

`java.lang.AssertionError: Dart test failed: tests.forgot_password_test

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following WaitUntilVisibleTimeoutException was thrown running
a test:
TimeoutException after 0:00:10.000000: Finder "zero widgets with
text "Welcome" (ignoring offstage widgets)" did not find
any visible (i.e. hit-testable) widgets

When the exception was thrown, this was the stack:
#0 PatrolTester.waitUntilVisible. (package:patrol/src/custom_finders/patrol_tester.dart:405:11)

#1 TestAsyncUtils.guard. (package:flutter_test/src/test_async_utils.dart:114:7)

The test description was:
Forgot Password Test
═════════════════════════════════════════════════════════════════

at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:130)
at pl.leancode.patrol.example.MainActivityTest.runDartTest(MainActivityTest.java:30)

`

[fylyppo]

It looks like they have been run well. Try to set up results bucket to have screenrecordings and take a look what happened on the device

[ashwinichannaveerappa]

@fylyppo i have set up the screenrecording, still the same issue. it will not even complete the atleast one
testcase completely.

[fylyppo]

Ok, but maybe now you have issues inside your tests? Does every test pump the app on the screen and tries to tap or checks something? Here's script we're using for FTL.

[ashwinichannaveerappa]

@fylyppo yes every tests has await $.pump`

patrol("Check Login functionality with multiple Scenarios", ($) async {
//load l10n
await loadLocalization();

//initialize app
await baseClass.initializeApp();

//login with valid credentials
await baseClass.loginWithValidCredentials($);
//logout
await settingsScreenObject.logOut($);

await $.pump();
await Future.delayed(_timeout);

//login with empty username and password
await signInScreenObject.loginEmptyUserNameAndPassword($);
await Future.delayed(_timeout);

//login with invalid credentials
await signInScreenObject.loginWithInValidCredentials($);
await $.native.pressHome();

});

settings_test.dart

` patrol(
"Check all the fields in Settings screen available for logged in user",
($) async {

    //l10n initialization
    await loadLocalization();
    await baseClass.initializeApp();
    await baseClass.loginWithValidCredentials($);
    await $.pump();
    await settingsScreenObject.tapOnSettings($);
    await $.pump();

    await testVolumeAndSpeechSettings($);

    await testNotifications($);

    await testAccountSettings($);

    await disableAlwaysShowText($);
  }

);`

[nzkn]

I experience the same issue

[bartekpacia]

The original issue is now solved ("unable to run multiple tests on firebase"). At this point, these issues are related to your tests failing.

If you believe your tests are failing because of a bug in Patrol, file a new issue for your specific problem and fill out the template.

[bartekpacia]

Also, please learn how to paste code correctly on GitHub.

[ashwinichannaveerappa]

@bartekpacia tests are not failing, if i run the individual tests in the firebase, thats getting passed.

[fylyppo]

Can you show me what's in "View issues"?

[ashwinichannaveerappa]

[fylyppo]

Are you sure you're using --use-orchestrator flag?

[ashwinichannaveerappa]

yes i am using the flag, please find the below command

gcloud firebase test android run --type instrumentation --app build/app/outputs/apk/dev/debug/app-dev-debug.apk --test build/app/outputs/apk/androidTest/dev/debug/app-dev-debug-androidTest.apk --device=model=redfin,version=30,orientation=portrait --use-orchestrator --ti meout 45m

[fylyppo]

Hm, here you can check our GHA workflow with Firebase Test Lab and here our gcloud command.
The difference is with this flag: --environment-variables clearPackageData=true, it clears app's local storage before each test case execution.
And try to use different device and different API level. Let me know if you noticed anything different

[ashwinichannaveerappa]

@fylyppo thanks a lot . --environment-variables clearPackageData=true this worked like a charm.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue.