Skip to content

Commit

Permalink
[dPWA Testing] Moved docs and added test addition how-to.
Browse files Browse the repository at this point in the history 
This change updates a lot of documentation, adds a how-to guide for
adding new integration tests, and starts the documentation move to
docs/webapps.

Bug: 1264155
Change-Id: I44fbd3e79f4153c91b977170671d37d503d868fe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3256208
Commit-Queue: Daniel Murphy <[email protected]>
Reviewed-by: Evan Stade <[email protected]>
Reviewed-by: Phillis Tang <[email protected]>
Cr-Commit-Position: refs/heads/main@{#937979}
  • Loading branch information
@dmurph
dmurph authored and Chromium LUCI CQ committed Nov 3, 2021
1 parent 30f0535 commit 27a6bd0
Show file tree
Hide file tree
Showing 4 changed files with 402 additions and 319 deletions.
130 changes: 2 additions & 128 deletions chrome/browser/ui/views/web_apps/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,129 +1,3 @@
# dPWA Integration Tests

The dPWA integration tests use a special framework. Each test is defined by a
series of "testing actions", and test cases programmatically generated by a
python script. The script will only yield tests for which every "action" is
currently supported by the testing framework.

## Background

See this [design doc](https://docs.google.com/document/d/e/2PACX-1vTFI0sXhZMvvg1B3sctYVUe64WbLVNzuXFUa6f3XyYTzKs2JnuFR8qKNyXYZsxE-rPPvsq__4ZCyrcS/pub) for background information, and the testing script [README](../../../../test/webapps/README.md) for how integration tests are generated.

## Future work

* Fix how sync tests await web app quiescense. Currently, there is no waiting,
and tests will fail.

## Disabling a Test

Tests can be disabled in the same manner that other integration/browser tests
are disabled, using macros. See [On disabling
tests](https://chromium.googlesource.com/chromium/src/+/main/docs/testing/on_disabling_tests.md)
for more information.

## How to Contribute

### Adding a Testing Action

Adding support for a new testing action in the framework is as simple as adding
a new method in `WebAppIntegrationBrowserTestBase`. In some cases, you may want
to implement actions related to profile sync, and need to call into SyncTest
helper methods. In this case, the meat of the action can be implemented in
`TwoClientWebAppsIntegrationSyncTest`. From there, you implement a method of the
same name in the base class, and have it call your sync action on the
`TestDelegate` member (ie `delegate_->SwitchProfileClients()`).

The list of testing actions are maintained in
`chrome/test/webapps/data/framework_supported_actions.csv`. To add a new testing
action, add the action into that csv. Use the emojis to indicate which platforms
the action is supported on (at the time of your CL landing, rather than in the
future). Use the following emojis to specify coverage for an action on a given
platform:

* 🌕 - Full coverage - This means that the testing framework implements this
action in a way that completely matches (or almost matches) the code paths
that are used when the user triggers this action.
* 🌓 - Partial coverage - This means that the testing framework implements this
action in a way that accomplishes the state change or check, but does not
fully match the code path that is used when the user triggers this action.
* 🌑 - No coverage - This means the action is not supported.


### Managing State

After every state-change action, a state snapshot is constructed and stored as a
member of WebAppIntegrationBrowserTestBase, `after_state_change_action_state_`.
This is done in `ConstructStateSnapshot()`, called from
`AfterStateChangeAction()`. At the start of every state-change action, this
state snapshot gets moved into the `before_state_change_action_state_` member,
giving us the ability to compare the pre- and post-conditions of every
state-change action.

When adding actions, it may be useful to bolt onto this state snapshot in order
to verify the results of state-change actions within state-check actions. To do
this, simply add onto the relevant state objects, and update the objects `==`
operator. Then, you need to make sure that your new state fields are updated in
`ConstructStateSnapshot()`.

## How It Works

This testing framework uses a [script](../../../../test/webapps/README.md) to
generate a minimal set of test cases that produce the maximum amount of code
coverage. The script will output in stdout tests cases to add and remove, and
the developer then performs the instructed changes to make the testing
implementation up-to-date.


This suite of tests has two main parts:
1. A script that analyzes dPWA code ([See design
doc](https://docs.google.com/document/d/1YmeNZCpIwUbeV3K3HGUdXzJjZDKIDyKrGfyjnYaLR5k).
2. The test implementations ([See design
doc](https://docs.google.com/document/d/1Gd14fjwA4VKoRzL2TAvi9paXwyh36ehlS4gbpUmUeeI).

### Script
See the [README](../../../../test/webapps/README.md) and [design doc](https://docs.google.com/document/d/e/2PACX-1vTFI0sXhZMvvg1B3sctYVUe64WbLVNzuXFUa6f3XyYTzKs2JnuFR8qKNyXYZsxE-rPPvsq__4ZCyrcS/pub).

### Test Structure
The high level flow of execution is as follows:
* Each tests lives as a regular browsertest with a specific name so the script
can determine if it exists.
* The script outputs a test where actions are method calls, and before & after
each test there is Pre and Post action method to call allow the framework to
record state, clean up, or wait as necessary.

## Components
[Design
doc](https://docs.google.com/document/d/139ktCajbmbFKh4T-vEhipTxilyYrXf_rlCBHIvrdeSg).

### WebAppIntegrationBrowserTestBase
A helper class containing most of the test implementation, meant to be used
as a private member on the test-driving classes. Contains most of the test
framework implementation:
* Most action implementation methods.
* Capturing state snapshots.

### WebAppIntegrationBrowserTestBase::TestDelegate
An abstract class that’s an application of the delegate interface pattern.
`WebAppIntegrationBrowserTestBase` stores an instance of this class as a
private member, `delegate_`, allowing the base class to call into protected
members of `InProcessBrowserTest` or `SyncTest`, such as
`InProcessBrowserTest::browser()` or `SyncTest::GetAllProfiles()`. This also
has pure virtual methods for sync functionality that needs to be implemented
in `TwoClientWebAppsSyncTest`, but called from the base class.

### WebAppIntegrationBrowserTest
Class that drives the tests. Subclass of both `InProcessBrowserTest` and
`WebAppIntegrationBrowserTestBase::TestDelegate`. Responsible
for telling the base class where the test input files live, handling test setup,
and implementing `TestDelegate` methods to expose protected members of
`InProcessBrowserTest` to the base class. This class owns the base class, and
stores it as a private member, `helper_`, passing it an instance of itself (as
the TestDelegate) on construction.


### TwoClientWebAppsSyncTest
Similar to `WebAppIntegrationBrowserTest`, but inheriting from `SyncTest`
instead of from `InProcessBrowserTest`. In addition, some testing actions
related to profile sync are implemented in this class, and are exposed via
`TestDelegate` pure virtual method overrides.
# Desktop Web App Integration Tests

Documentation has moved [here](/docs/webapps/integration-testing-framework.md).
Loading

0 comments on commit 27a6bd0

Please sign in to comment.