Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent e2e test failures #6956

Closed
danielbachhuber opened this issue May 25, 2018 · 16 comments · Fixed by #7013, #7493, #7748 or #8330
Closed

Intermittent e2e test failures #6956

danielbachhuber opened this issue May 25, 2018 · 16 comments · Fixed by #7013, #7493, #7748 or #8330
Assignees
Labels
[Priority] High Used to indicate top priority items that need quick attention [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.

Comments

@danielbachhuber
Copy link
Member

The e2e test suite currently fails intermittently. Sometimes the test failure looks like this:

image

I have no idea on the cause of this, but it's getting in the way of other unrelated development.

@danielbachhuber danielbachhuber added [Priority] High Used to indicate top priority items that need quick attention [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. labels May 25, 2018
@danielbachhuber danielbachhuber added this to the 3.0 milestone May 25, 2018
@noisysocks
Copy link
Member

noisysocks commented May 28, 2018

I spotted a few mistakes to do with async functions in our E2E tests which could be contributing to this problem. #6971 attempts to fix some of them.

Update: #6995 should fix the specific error you encountered here.

@danielbachhuber
Copy link
Member Author

Comment from @youknowriad to address (#6995 (comment)):

I'm pretty sure that's not the only place in our tests where we open the inserter, we should probably be consistent.
Do you think we should factorize that in a insertBlock( 'core' ) utility?
And also, can we add a comment explaining why this is necessary? Is there an API request or something?

@aduth
Copy link
Member

aduth commented Jun 20, 2018

There's a new suspect from the change-detection.test.js suite:

✕ Should prompt if changes made while save is in-flight (1747ms)

Have seen separately on both #7419 and #7220 .

As author of the offending test, I shall plan to investigate.

@aduth aduth reopened this Jun 20, 2018
@aduth aduth self-assigned this Jun 20, 2018
@gziolo gziolo modified the milestones: 3.1, 3.2 Jun 21, 2018
@gziolo
Copy link
Member

gziolo commented Jun 21, 2018

Moving to 3.2. This is an issue with tests rather than with actual code.

@youknowriad
Copy link
Contributor

I'm also starting to see this failure from time to time.

screen shot 2018-06-27 at 13 33 47

@aduth
Copy link
Member

aduth commented Jun 27, 2018

FYI #7493 will resolve at least two of the intermittent failing tests I've seen.

@danielbachhuber
Copy link
Member Author

I just saw this:

image

On #7322, which was just rebased against master.

@aduth
Copy link
Member

aduth commented Jun 28, 2018

Based on the error message, I'm inclined to think it should be resolved with #7544 . Let's continue reopening if it persists, though.

@aduth
Copy link
Member

aduth commented Jun 28, 2018

A change was snuck into #6959 (689d79d) to run end-to-end tests in serial order (not parallel) which could help make the tests more deterministic and eliminate any chance of tests affecting one another (since, for example, some tests activate plugins which could have an impact site-wide if tests are run in parallel).

http://jestjs.io/docs/en/cli.html#runinband

@aduth
Copy link
Member

aduth commented Jul 6, 2018

I've been seeing this one a lot lately: #6956 (comment)

@aduth aduth reopened this Jul 6, 2018
@aduth aduth removed this from the 3.2 milestone Jul 6, 2018
@aduth
Copy link
Member

aduth commented Jul 9, 2018

Starting to see a new one.

FAIL test/e2e/specs/preview.test.js
  ● Preview › Should open a preview window for a new post
    Protocol error (Runtime.callFunctionOn): Cannot find context with specified id undefined
      
      at Promise (node_modules/puppeteer/lib/Connection.js:198:56)
      at CDPSession.send (node_modules/puppeteer/lib/Connection.js:197:12)
      at ExecutionContext.evaluateHandle (node_modules/puppeteer/lib/ExecutionContext.js:71:75)
      at ExecutionContext.evaluate (node_modules/puppeteer/lib/ExecutionContext.js:46:31)
      at Frame.evaluate (node_modules/puppeteer/lib/FrameManager.js:299:20)

Example: #7841 https://travis-ci.org/WordPress/gutenberg/jobs/401939944

@aduth aduth reopened this Jul 9, 2018
@talldan
Copy link
Contributor

talldan commented Jul 24, 2018

I'm also seeing block-icons.test.js fail locally consistently.

Seems to be due to the wrong block being focused during the test (the inserted blocks seem to have a nested paragraph block and the paragraph block end up being focused):
https://github.com/WordPress/gutenberg/blob/master/test/e2e/specs/block-icons.test.js#L119-L122

Had a quick look but couldn't figure out a workaround.

@gziolo
Copy link
Member

gziolo commented Jul 24, 2018

@talldan, can you check PR from @jorgefilipecosta #7896 and verify if that resolves this issue? It has been opened for some time but we didn't come up to the consensus if it resolves the issue or hides it deeper :)

@aduth
Copy link
Member

aduth commented Aug 13, 2018

#8707 (comment)

@gziolo
Copy link
Member

gziolo commented Aug 14, 2018

@aduth - there is an open issue for it logged for it. Actually, there are 2 open issues with e2e test failures which happen from time to time:

@aduth
Copy link
Member

aduth commented Aug 14, 2018

Ok, #8712 appears to be the same as in #8707 (comment) so let's close in favor of that one.

@aduth aduth closed this as completed Aug 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Priority] High Used to indicate top priority items that need quick attention [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.
Projects
None yet
6 participants