Take screenshot on failure option

[mateuszwojtczak]

Let's consider an option that defaults to taking screenshots after any failures.

[bartekpacia]

See also:

• Taking screenshots and recording videos #8

[bartekpacia]

Status

• taking screenshots on-demand implemented in Create a custom VM service extension and use it to implement the screenshot taking feature  #593
• regarding taking screenshots at the end of the test: it works only if the test fails, otherwise a black screen with a pink text is shown:

Screenshot on failure example (1)

Code

import 'package:example/main.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:patrol/patrol.dart';

import 'config.dart';

Future<void> main() async {
  late PatrolTester tester;

  patrolTest(
    'takes a screenshot after failed expect',
    config: patrolConfig,
    nativeAutomation: true,
    ($) async {
      tester = $;
      await $.pumpWidgetAndSettle(ExampleApp());

      expect('0', '1');
    },
  );

  tearDown(() async {
    await tester.host.takeScreenshot(name: 'after_failed_assertion');
  });
}

Screenshot

Screenshot on failure example (2)

I like this better.

Code

import 'package:example/main.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:patrol/patrol.dart';

import 'config.dart';

Future<void> main() async {
  patrolTest(
    'takes a screenshot after failed expect',
    config: patrolConfig,
    nativeAutomation: true,
    ($) async {
      addTearDown(() async {
        await $.host.takeScreenshot(name: 'after_failed_assertion');
      });

      await $.pumpWidgetAndSettle(ExampleApp());
      await $('Add').tap(); // increments counter by 10

      expect($('11'), findsOneWidget);
    },
  );
}

Screenshot

[bartekpacia]

Update: the below is fixed, see #593

Problem

Oops, we've got a serious problem – after an hour of searching, I can't find any way to do reverse port forwarding on physical iOS devices.

I've assumed it's not gonna be a problem because we have the iproxy program, which can forward port 2000 on my mac to port 3000 on my iPhone by doing:

$ iproxy 2000 3000

but apparently, iproxy can't forward port 3000 on my iPhone to port 2000 on my mac.

This makes it impossible to use any host features (such as taking screenshots, executing scripts, etc.) from withing a test running on a physical iOS device. Android and iOS simulator work very nicely.

[bartekpacia]

todo: check if we can take screenshot only test failed

[bartekpacia]

I couldn't find a way to execute code only if a particular test case failed.

[bartekpacia]

We tried to refine this today, but realized we don't know what the requirements are.

[suside]

It's worth noting that there is native support for this with flutter drive --screenshot=path/to/dir ...