Testing: Reimplement pressKeyWithModifier to emulate Cmd+A in macOS #14243

aduth · 2019-03-05T19:15:56Z

This pull request seeks to improve E2E test login mechanism. The loginUser function currently relies on keyboard interactions to "select all" and replace the contents of the username and password fields. This does not work correctly on macOS in the current version of Puppeteer:

puppeteer/puppeteer#1313

I was observing that when switching between users in the execution of E2E tests, it would often prepend the new username to the current login, rather than replace it.

The changes here update the utility function to assign the username and password fields explicitly using DOM APIs in a page.evaluate callback.

nerrad

Ya this looks like its an acceptable solution for now especially when testing the login form is not the primary focuses of these tests.

However, if the e2e test suite becomes a standard for use across all of WP (and not just GB), might want to consider just clearing the input using page.evaluate and then use the keyPress to add the value?

aduth · 2019-03-05T19:29:35Z

might want to consider just clearing the input using page.evaluate and then use the keyPress to add the value?

By value, do you mean to your previous point of it being something of a secondary focus of the test to ensure the login behavior? I'm a bit on the fence with this, where I think we should try to make utilities as lean and optimized as possible, even if it means bypassing real user interactions. Especially when any guarantee we have that it "works" is only incidental, and should probably be tested in standalone (i.e. a test suite covering login behaviors).

nerrad · 2019-03-05T19:38:09Z

By value, do you mean to your previous point of it being something of a secondary focus of the test to ensure the login behavior?

Yes that what I meant.

I'm a bit on the fence with this, where I think we should try to make utilities as lean and optimized as possible, even if it means bypassing real user interactions.

I can agree with this assuming that it's made clear in documentation that any of these utility functions should not be assumed as test coverage for what they do. This could probably just be something added to the documentation for the package itself.

I'm mostly wary about e2e tests getting written for WP proper using the same packages and the utils being considered as accurate for test coverage of the action being performed. I agree that if in the future WP core uses this library for future e2e test coverage of general WP ui/ux it should have an explicit test covering logging in (vs using this util).

aduth · 2019-03-06T13:49:04Z

Noting there's at least one other occurrence of the pressKeyWithModifier( 'primary', 'a' ), seen in #13899 (#13899 (comment)). We should probably fix this everywhere, perhaps with the hacky-but-consistent solution posed at puppeteer/puppeteer#1313 (comment) .

gziolo · 2019-03-08T16:04:36Z

Noting there's at least one other occurrence of the pressKeyWithModifier( 'primary', 'a' ), seen in #13899 (#13899 (comment)). We should probably fix this everywhere, perhaps with the hacky-but-consistent solution posed at puppeteer/puppeteer#1313 (comment) .

On a different note, should we throw an error when someone tries to use this keyboard shortcut to make developer aware that it's not supported?

aduth · 2019-03-08T19:36:57Z

On a different note, should we throw an error when someone tries to use this keyboard shortcut to make developer aware that it's not supported?

It'd probably be easy enough to create a custom restrict-syntax rule.

aduth · 2019-03-08T19:37:52Z

Current status of this one: I think I want to try to solve this in a more general fashion for both the login screen, and the other problem cases.

aduth · 2019-03-11T21:10:01Z

The recent force-pushed branch includes revisions for all instances of Cmd+A in the tests.

It's been a long time since I've seen this all-green in my local environment:

cc @getdave

aduth · 2019-03-11T21:10:48Z

packages/e2e-tests/specs/multi-block-selection.test.js

+		// alternative to Cmd+A. The second issuance of the key combination is
+		// handled internerally by the block editor's KeyboardShortcuts utility
+		// and is not subject to the same buggy emulation.
+		await __unstableSelectAll();


This one was a bit awkward to test, because it truly exists to ensure that Cmd+A twice has the correct behavior, so I opted to keep as close as possible to the original implementation to avoid using selectAllBlocks.

gziolo

There are repeated 2 failing tests on two independent nodes:

FAIL specs/reusable-blocks.test.js (12.006s)
  ● Reusable Blocks › multi-selection reusable block can be converted back to regular blocks

FAIL specs/multi-block-selection.test.js (10.526s)
  ● Multi-block selection › should speak() number of blocks selected with multi-block selection

It seems like it is related to the changes where selectAllBlocks() replaces double calls of using keyboard shortcuts.

gziolo · 2019-03-12T07:24:16Z

packages/e2e-test-utils/README.md

@@ -16,7 +16,7 @@ npm install @wordpress/e2e-test-utils --save-dev

 ### activatePlugin

-[src/index.js#L1-L1](src/index.js#L1-L1)
+[src/index.js#L2-L2](src/index.js#L2-L2)


@nosolosw - this only confirms that we need to find a way to output a line in the source code where all API methods are defined rather than where they are exported. In the majority of cases, we will use index.js file to collect public API methods from files which contain the actual implementation.

gziolo · 2019-03-12T07:31:20Z

packages/e2e-tests/specs/multi-block-selection.test.js

-		// Multiselect via keyboard.
-		await pressKeyWithModifier( 'primary', 'a' );
-		await pressKeyWithModifier( 'primary', 'a' );
+		await selectAllBlocks();


Should it be still called twice? It doesn't work properly on Travis.

gziolo · 2019-03-12T07:35:51Z

packages/e2e-tests/specs/reusable-blocks.test.js

@@ -210,8 +210,7 @@ describe( 'Reusable Blocks', () => {
 		await page.keyboard.type( 'Second paragraph' );

 		// Select all the blocks
-		await pressKeyWithModifier( 'primary', 'a' );
-		await pressKeyWithModifier( 'primary', 'a' );
+		await selectAllBlocks();


Should it still be called twice? It doesn't work properly on Travis.

gziolo · 2019-03-12T07:42:03Z

packages/e2e-test-utils/src/select-all-blocks.js

+	// handled internerally by the block editor's KeyboardShortcuts utility,
+	// and is not subject to the same buggy browser/OS emulation.
+	await __unstableSelectAll();
+	await pressKeyWithModifier( 'primary', 'a' );


It seems like on Travis, it only selects a single block.

aduth · 2019-03-18T20:56:31Z

I need to revisit this. I'd had some general thoughts on direction being that we remove __unstableSelectAll and handle it from within pressKeyWithModifier. There's also some more recent revelations that this may not ever be changed in Puppeteer (related).

The thought I'd had on an implementation idea was that we run the key-combination and consider whether the default behavior was prevented and, if it wasn't, run selectall afterward assuming that it would be idempotent with a Ctrl+A press. The problem with this is that it's not immediately apparent that we could detect whether an event emitted through page.keyboard would have its default prevented.

aduth · 2019-03-28T21:13:20Z

I've rebased and pushed a revised implementation which, while on one-hand is an awful abomination, on the other is at least quite well insulated in the implementation of pressKeyWithModifier.

I was observing some local failures which were not immediately explainable. I expect it may fail in Travis as well. I'll plan to look again in the morning.

gziolo

I like the direction where it is heading. I have no idea what happens inside the emulation function but if that solves the issue I'm all for it 😄

gziolo · 2019-03-29T09:31:08Z

packages/e2e-test-utils/src/press-key-with-modifier.js

 	const overWrittenModifiers = {
 		...modifiers,
 		shiftAlt: ( _isApple ) => _isApple() ? [ SHIFT, ALT ] : [ SHIFT, CTRL ],
 	};
-	const mappedModifiers = overWrittenModifiers[ modifier ]( isAppleOS );
+	const mappedModifiers = overWrittenModifiers[ modifier ]( IS_MAC_OS );


TypeError: _isApple is not a function

It looks like this should stay unmodified as it breaks keycodes implementation.

gziolo · 2019-03-29T09:32:31Z

packages/e2e-test-utils/src/press-key-with-modifier.js

@@ -16,12 +88,15 @@ import { modifiers, SHIFT, ALT, CTRL } from '@wordpress/keycodes';
 * @param {string} key Key to press while modifier held.
 */
 export async function pressKeyWithModifier( modifier, key ) {
-	const isAppleOS = () => process.platform === 'darwin';
+	if ( modifier.toLowerCase() === 'primary' && key.toLowerCase() === 'a' ) {
+		return await emulateSelectAll();


Should this emulation be limited to macOS?

Should this emulation be limited to macOS?

I'd thought about it, and while true that it doesn't need to occur outside macOS, I'd thought to avoid divergent implementations. The main downside is that, at least currently, emulateSelectAll has to do its own platform detection to determine Meta vs. Control as modifier, which wouldn't be necessary if only applying the emulation specific to macOS.

aduth · 2019-04-02T23:22:08Z

Aside from #14243 (comment) , there was another obvious outstanding issue fixed in 6b0bf94: I was using the wrong property names meta and ctrl (corrected to metaKey, ctrlKey). Passes locally now (and demonstrated standalone), fingers crossed for Travis 🤞

gziolo · 2019-04-03T06:53:20Z

@ashwin-pc - can you check if this PR solves any of the e2e test failures you observed locally?

getdave · 2019-04-03T08:46:47Z

@aduth I was experiencing issues with packages/e2e-tests/specs/block-hierarchy-navigation.test.js.

I've tested this against your branch using the following command:

npm run test-e2e packages/e2e-tests/specs/block-hierarchy-navigation.test.js

I'm sorry to say that the result is a test failure. This is due to the same issue (ie: modifiers not working on MacOS).

My code comments were addressed

gziolo

It works for me (macOS), there is one unrelated code failure but it should be fixed with rebase:

Without this PR I used to see some test failures related to selection from time to time.

aduth · 2019-04-03T12:35:28Z

@getdave Can you make sure to run npm run build before the npm run test-e2e command? If I recall correctly, the build no longer done automatically (Edit: #13420), and since pressKeyWithModifier is part of a built module, the changes wouldn't otherwise be reflected.

I'll probably plan to do a rebase here shortly as well, so worth testing again from the latest copy of the branch at that point.

getdave · 2019-04-04T09:54:21Z

✅ @aduth @gziolo You'll be pleased to see the npm build step has fixed things:

gziolo

Let's 🚢

gziolo · 2019-04-04T09:58:56Z

Great work @aduth adding all this magic behind to make it work regardless of limitation of Puppeteer 🎉
Thanks, @getdave for help with testing, and @nerrad for reviews.

gziolo · 2019-04-04T10:01:21Z

Can you make sure to run npm run build before the npm run test-e2e command? If I recall correctly, the build no longer done automatically (Edit: #13420), and since pressKeyWithModifier is part of a built module, the changes wouldn't otherwise be reflected.

We might consider doing a similar trick we do for unit tests where we always use sources for local packages:
https://github.com/WordPress/gutenberg/blob/master/test/unit/jest.config.js#L12-L14

It won't solve all issues but at least will ensure that test logic always uses the latest code.

ashwin-pc · 2019-04-04T22:39:32Z

@ashwin-pc - can you check if this PR solves any of the e2e test failures you observed locally?

I'm not sure if this comment is still necessary but yes, the e2e test failures i observed have been resolved with this commit. I believe #14732 can be closed now.

gziolo · 2019-04-05T05:04:40Z

Awesome @ashwin-pc, thanks for checking on your side 👍

aduth added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Mar 5, 2019

aduth requested a review from swissspidy March 5, 2019 19:15

aduth requested review from ajitbohra, gziolo, nerrad, ntwb and youknowriad as code owners March 5, 2019 19:15

nerrad approved these changes Mar 5, 2019

View reviewed changes

gziolo added this to the 5.3 (Gutenberg) milestone Mar 6, 2019

aduth force-pushed the update/e2e-login-assign branch from a7c76e0 to 85622de Compare March 11, 2019 21:07

aduth requested a review from talldan as a code owner March 11, 2019 21:07

aduth commented Mar 11, 2019

View reviewed changes

gziolo previously requested changes Mar 12, 2019

View reviewed changes

gziolo reviewed Mar 12, 2019

View reviewed changes

youknowriad removed this from the 5.3 (Gutenberg) milestone Mar 18, 2019

aduth mentioned this pull request Mar 18, 2019

Block Library: Avoid setting Embed className when empty, non-responsive #13706

Closed

oandregal mentioned this pull request Mar 27, 2019

docgen: add source URLs to the default formatter #14341

Closed

aduth force-pushed the update/e2e-login-assign branch from 7534582 to 8857e40 Compare March 28, 2019 21:11

gziolo reviewed Mar 29, 2019

View reviewed changes

gziolo added the [Type] Bug An existing feature does not function as intended label Mar 29, 2019

aduth changed the title ~~Testing: Assign E2E login credentials explicitly~~ Testing: Reimplement pressKeyWithModifier to emulate Cmd+A in macOS Apr 2, 2019

aduth mentioned this pull request Apr 2, 2019

Some E2E tests fail on the Docker environment #14732

Closed

6 tasks

gziolo assigned aduth Apr 3, 2019

gziolo reviewed Apr 3, 2019

View reviewed changes

aduth added 3 commits April 3, 2019 08:36

Testing: Assign E2E login credentials explicitly

3b91b6a

Testing: Create new selectAll, selectAllBlocks utilities

fbff4a4

Testing: Reimplement Ctrl+A as emulated in pressKeyWithModifier

a7e85b3

aduth force-pushed the update/e2e-login-assign branch from 6b0bf94 to a7e85b3 Compare April 3, 2019 12:40

gziolo approved these changes Apr 4, 2019

View reviewed changes

gziolo merged commit 454011e into master Apr 4, 2019

gziolo deleted the update/e2e-login-assign branch April 4, 2019 09:56

youknowriad removed the [Type] Bug An existing feature does not function as intended label Apr 12, 2019

youknowriad added this to the 5.5 (Gutenberg) milestone Apr 12, 2019

talldan mentioned this pull request Apr 15, 2019

Bump version number to 5.5.0-rc.1 #14981

Merged

aduth mentioned this pull request Apr 16, 2019

Plugin: Backport bug fixes from Gutenberg 5.5.0 to wp/trunk (WordPress 5.2 RC) #14987

Merged

This was referenced Apr 17, 2019

Bump/5.5 #15011

Closed

Release/5.5 #15012

Closed

Bump plugin version to 5.5 #15013

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing: Reimplement pressKeyWithModifier to emulate Cmd+A in macOS #14243

Testing: Reimplement pressKeyWithModifier to emulate Cmd+A in macOS #14243

aduth commented Mar 5, 2019

nerrad left a comment •

edited

Loading

aduth commented Mar 5, 2019

nerrad commented Mar 5, 2019

aduth commented Mar 6, 2019

gziolo commented Mar 8, 2019

aduth commented Mar 8, 2019

aduth commented Mar 8, 2019

aduth commented Mar 11, 2019

aduth Mar 11, 2019

gziolo left a comment

gziolo Mar 12, 2019

gziolo Mar 12, 2019

gziolo Mar 12, 2019

gziolo Mar 12, 2019

aduth commented Mar 18, 2019

aduth commented Mar 28, 2019 •

edited

Loading

gziolo left a comment

gziolo Mar 29, 2019

gziolo Mar 29, 2019

aduth Apr 2, 2019

aduth commented Apr 2, 2019 •

edited

Loading

gziolo commented Apr 3, 2019

getdave commented Apr 3, 2019

gziolo left a comment

aduth commented Apr 3, 2019 •

edited

Loading

getdave commented Apr 4, 2019

gziolo left a comment

gziolo commented Apr 4, 2019

gziolo commented Apr 4, 2019

ashwin-pc commented Apr 4, 2019

gziolo commented Apr 5, 2019

Testing: Reimplement pressKeyWithModifier to emulate Cmd+A in macOS #14243

Testing: Reimplement pressKeyWithModifier to emulate Cmd+A in macOS #14243

Conversation

aduth commented Mar 5, 2019

nerrad left a comment • edited Loading

Choose a reason for hiding this comment

aduth commented Mar 5, 2019

nerrad commented Mar 5, 2019

aduth commented Mar 6, 2019

gziolo commented Mar 8, 2019

aduth commented Mar 8, 2019

aduth commented Mar 8, 2019

aduth commented Mar 11, 2019

Choose a reason for hiding this comment

gziolo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aduth commented Mar 18, 2019

aduth commented Mar 28, 2019 • edited Loading

gziolo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aduth commented Apr 2, 2019 • edited Loading

gziolo commented Apr 3, 2019

getdave commented Apr 3, 2019

gziolo left a comment

Choose a reason for hiding this comment

aduth commented Apr 3, 2019 • edited Loading

getdave commented Apr 4, 2019

gziolo left a comment

Choose a reason for hiding this comment

gziolo commented Apr 4, 2019

gziolo commented Apr 4, 2019

ashwin-pc commented Apr 4, 2019

gziolo commented Apr 5, 2019

nerrad left a comment •

edited

Loading

aduth commented Mar 28, 2019 •

edited

Loading

aduth commented Apr 2, 2019 •

edited

Loading

aduth commented Apr 3, 2019 •

edited

Loading