Test Butler

Reliable Android testing, at your service.

Motivation

Test Butler was inspired by the Google presentation "Going Green: Cleaning up the Toxic Mobile Environment". For more background, read the Test Butler announcement blog post.

What does it do?

• Stabilizes the Android emulator, to help prevent test failures caused by emulator issues.
  • Disables animations so Espresso tests can run properly.
  • Disables crash & ANR dialogs so a system app misbehaving won't trigger a popup that causes your UI tests to fail.
  • Locks the keyguard, WiFi radio, and CPU to ensure they don't go to sleep unexpectedly while tests are running.
  • Disables system spell checker to avoid redline highlights and text auto correction.
• Handles changing global emulator settings and holds relevant permissions so your app doesn't have to.
  • Enable/disable WiFi: Test Butler allows tests to simulate a situation where WiFi is not connected or changes availability at some point.
  • Change device orientation: Tests can manually set the orientation of the device during test execution.
  • Set location services mode: Test Butler lets your code simulate different location services modes, like battery saver or high accuracy.
  • Set application locale: Tests can set a custom Locale object for their application to simulate running the app in another language.
  • Grant runtime permissions: Tests can grant Marshmallow's runtime permissions to their application directly from test code.
  • Use hardware IME: Tests can tell the system to prefer the hardware IME.
  • Control immersive mode confirmation: Tests can enable or disable immersive mode confirmation.

How does it work?

Test Butler is a two-part project. It includes an Android library that your test code can depend on, as well as a companion Android app apk that must be installed on your Android emulator before using Test Butler. You can build the Test Butler APK yourself from source, or download the binary from Bintray

The Test Butler library is a thin wrapper around an AIDL interface to give your tests a safe way to talk to the Test Butler app's service running in the background on the emulator.

The Test Butler app is signed using the system keystore for the Android emulator, so it is automatically granted signature-level permissions when installed. This means granting permissions via adb is not necessary. It also means that this app can only be installed on emulators that use the stock Android keystore!

Being a system app makes Test Butler much easier to use than existing Android solutions for changing emulator settings. To disable animations, you just need a single line of code in your app; no extra permissions in your debug manifest, no granting permissions via adb, no Gradle plugin to integrate.

Test Butler can even use permissions that can't be granted via adb, like the SET_ACTIVITY_WATCHER permission, which lets Test Butler disable crash & ANR dialogs during tests.

Any "gotchas" to look out for?

Only a few:

Test Butler helper app requires an emulator image with stock Android else it won't install, therefore it will not work with non-stock emulator images such as ones with Google APIs or Google Play!

Test Butler adds a custom IActivityController to the system to be able to suppress crash & ANR dialogs. This technique is also used internally by the Monkey tool. Unfortunately, the implementation of the isUserAMonkey() method takes advantage of the fact that the Monkey class is the only thing inside Android that sets an IActivityController and returns true whenever one is set.

This means that isUserAMonkey() will return true while Test Butler is running! If your app uses this method to invoke different behavior during actual monkey testing, you may encounter issues while running tests with Test Butler. An easy fix is to create a helper method in your app to call instead of isUserAMonkey(), which returns false while instrumentation tests are running and calls through to the real isUserAMonkey() when the app is not being instrumented.

Also, you can use methods in the TestButler class in the @BeforeClass & @AfterClass of a test class. However, be warned that the test suite will not report that the Test Butler app is not installed, instead it will skip the test. This is due to a bug in Android testing support.

Download

Download the latest .apk and .aar via Maven:

    <dependency>
      <groupId>com.linkedin.testbutler</groupId>
      <artifactId>test-butler-library</artifactId>
      <version>1.4.0</version>
      <type>pom</type>
    </dependency>
    <dependency>
      <groupId>com.linkedin.testbutler</groupId>
      <artifactId>test-butler-app</artifactId>
      <version>1.4.0</version>
      <type>pom</type>
    </dependency>

or Gradle:

    androidTestImplementation 'com.linkedin.testbutler:test-butler-library:1.4.0'

You can also download the apk file manually from Bintray if you prefer.

Getting Started

Install the Test Butler apk on your emulator prior to running tests, then add the following to your test runner class:

public class ExampleTestRunner extends AndroidJUnitRunner {
  @Override
  public void onStart() {
      TestButler.setup(InstrumentationRegistry.getTargetContext());
      super.onStart();
  }

  @Override
  public void finish(int resultCode, Bundle results) {
      TestButler.teardown(InstrumentationRegistry.getTargetContext());
      super.finish(resultCode, results);
  }
}

To change settings on the device from your tests, just use the methods in the TestButler class. For example:

@Before
public void setup() {
  TestButler.setLocationMode(Settings.Secure.LOCATION_MODE_OFF);
}

@Test
public void verifyBehaviorWithNoLocation() {
  // ...
}

@After
public void teardown() {
  TestButler.setLocationMode(Settings.Secure.LOCATION_MODE_HIGH_ACCURACY);
}

NB: See gotchyas above to see why @BeforeClass & @AfterClass aren't used here.