TestRunner could be improved to report more results, better

[bartekpacia]

We could print a nice test result summary, similar to what spec does:

Without colors it looks like this:

PASS  test/command_runner_test.dart
PASS  test/features/drive/test_runner_test.dart
PASS  test/features/drive/drive_command_test.dart
PASS  test/features/drive/device_finder_test.dart
PASS  test/features/drive/test_finder_test.dart
PASS  test/common/artifacts_repository_test.dart

For Patrol, I'd like to print something like this:

PASS  integration_test/core_cutter_test.dart.dart (build passed, 10/10 runs passed)
FAIL  integration_test/dual_cbr_test.dart  (build failed)
FAIL  integration_test/hsv_test.dart (build passed, 9/10 runs passed)
CANC  integration_test/srd_test.dart  (build passed, 3/10 runs passed)

To learn more about failures, see the logs above. grep is your friend.

Code already written

Some code that would go in #560 but I decided it doesn't fit there.

// TODO: Use this
class SingleTargetRunResult {
  SingleTargetRunResult({
    required this.target,
    required this.device,
    required this.plannedRuns,
    this.buildPassed = false,
    required this.runs,
  });

  final String target;
  final Device device;
  bool buildPassed;

  final int plannedRuns;
  final List<RunStatus> runs;
}

And then TestRunner.run()'s signature would be:

Future<List<SingleTargetRunResult>> run() {}

• This would make code in DriveCommand cleaner (it'd be moved to TestRunner)
• All the complexity would be contained inside TestRunner and properly tested

To be precise, this issue is only about nicer logs in the CLI, not about detailed error reporting using flutter test with the JSON reporter to convert test logs into a JUnit format. See #421 and #495 for this.

cc @jBorkowska @shilangyu @jakubfijalkowski – as always, if you have any thought about this, share them here :)

[jBorkowska]

Is this a place to add the error messages for why a test failed? Or would it be kept separate?

[bartekpacia]

No. To find you why a test failed, you'd have to still look in the logs.

I mean, I don't know how we could make it fit in 1 line and make it look good.

[jBorkowska]

From my perspective: I need a place, where I can easily find info about which test failed and why. Logs are quite hard to read (at least now), that's why I'm asking.

[jakubfijalkowski]

Jest prints, apart from the passed/failed marker, a summary of failed tests at the end of the output. This is, IMO, quite good middle-ground between being fancy (so a pass/fail colored marker) and being useful. If everything passes you have clean output, if something fails, you see what failed right there, without looking for it.

[bartekpacia]

@jakubfijalkowski Jest is great, and its equivalent in Dart is spec (in this context I mean spec_cli, the interactive tester runner) . I use it often and it just works and improves test readability.

Maybe we should look for ways to integrate with them, or borrow their interactive test runner code :)

[bartekpacia]

Cross-referencing #528

[neiljaywarner]

@bartekpacia yes, something like this would be awesome, our QE says patrol is awesome but hard to find the issues in the log etc.

[bartekpacia]

Yeah, we're aware of it. We hope to improve this!

For now, you can use flutter logs while running patrol test to view Dart-side logs.

[bartekpacia]

This is a possible duplicate of #870.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue.