Project automation: don't run e2e tests on documentation-only PRs

[priethor]

What?

Don't run e2e tests on documentation-only PRs

Why?

The benefits of not running e2e tests when PRs only contain documentation changes are twofold:

• It improves the contributor experience by avoiding unnecessary waiting times and other issues that might arise with flaky tests.
• It alleviates the usage of GitHub runners.

How?

By adding a paths-ignore filter to the e2e tests action that checks for markdown-only changes.

Testing Instructions

@ockham, is there a way to test this without actually pushing the PR? In a repo fork? 😅

[github-actions]

Flaky tests detected in 7a8bc27.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/5518581627
📝 Reported issues:

• [Flaky Test] Block context propagates to inner blocks #48827 in specs/editor/plugins/block-context.test.js
• [Flaky Test] Pick the additional variation in the inserted Columns block #47057 in specs/editor/plugins/block-variations.test.js
• [Flaky Test] ensures base block toolbars use roving tabindex #51834 in /test/e2e/specs/editor/various/toolbar-roving-tabindex.spec.js

[ockham]

@ockham, is there a way to test this without actually pushing the PR? In a repo fork? 😅

Yeah, that could work. Unfortunately, we have some conditionals preventing some GHA jobs from running in a fork (to keep the load on GHA down), e.g. this, so we might need to comment that out (or maybe not, since the top-level criteria for the whole workflow should kick in before that anyway.)

Alternatively, we could try creating a new branch based on this one with only a doc change (but I'm not sure if GHA will see the yml change from this branch and determine that it's not just doc changes). So yeah, maybe a fork is the most reliable way to test.

(Aside, I could've sworn we were already skipping e2e tests for doc changes. Did we ever change that? Or am I mixing things up, and it was a different GHA workflow we were skipping? 🤔 )

[ockham]

Ah-hah! I think we tried to disable unit tests for doc-only changes before but couldn't, as they were marked as Required for a PR to pass:

gutenberg/.github/workflows/unit-test.yml

Lines 3 to 4 in 203789b

	# Since Unit Tests are required to pass for each PR,
	# we cannot disable them for documentation-only changes.

Doesn't the same argument apply to e2e tests? 🤔

Edit: PR for context: #28696

[priethor]

Thanks for looking into this, @ockham ❤️

Doesn't the same argument apply to e2e tests? 🤔

Yep, the same applies to e2e tests; they are both required checks in trunk protected branch settings. On the other hand, there are skipped checks like Build Gutenberg Plugin, so my best guess is we could skip e2e and unit tests, too, but would probably need much bigger changes to the current GH action architecture. I'm happy to close this and not open that whole can of worms 😅