Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make tests resilient against transforms added by plugins #14632

Conversation

jorgefilipecosta
Copy link
Member

Description

During #14625 I noticed that if a plugin adds a transform our transforms test case fails.
We have an end 2 end test that tests the editor with plugins installed, so the transforms added by the plugins should be ignored.

How has this been tested?

Verify the end to end tests with plugins installed pass.

@gziolo
Copy link
Member

gziolo commented Mar 27, 2019

We have an end 2 end test that tests the editor with plugins installed, so the transforms added by the plugins should be ignored.

What if the plugin would remove a transform from the list or unregister one of the blocks? What if plugins adds an additional transform between core blocks? This might get very tricky and have an impact on other tests as well. I don't know what would be the best solution here. Do you have other ideas? Maybe we should run this particular test suite only when external plugins are disabled?

@gziolo gziolo added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Mar 27, 2019
@jorgefilipecosta jorgefilipecosta force-pushed the fix/make-transforms-test-resilient-agains-plugin-transforms branch 3 times, most recently from 15791c2 to 5c1f718 Compare March 27, 2019 16:43
@gziolo
Copy link
Member

gziolo commented Mar 28, 2019

See my comment #14625 (comment). This might be another way of solving the same issue.

@jorgefilipecosta jorgefilipecosta force-pushed the fix/make-transforms-test-resilient-agains-plugin-transforms branch from 5c1f718 to c173dfe Compare March 28, 2019 14:17
@jorgefilipecosta
Copy link
Member Author

See my comment #14625 (comment). This might be another way of solving the same issue.

Hi @gziolo, after thinking about this problem I preferred to follow your suggestion of not executing the tests when plugins are enabled. Plugins can change the markup of the blocks, change the way a transform works and other actions that would make the tests fail. We don't know the way a plugin changes a transform so the best long term solution is to leave the transform testing as a responsibility of the plugins.
So this PR was updated to not execute transform tests when popular plugins are enabled.

] )
)
)
);

const testMessageFormat = 'block %s in fixture %s into the %s block';

if ( process.env.POPULAR_PLUGINS ) {
Copy link
Member

@gziolo gziolo Mar 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is hard to follow at the moment.

Is it possible to divide this test suite into 2 groups? One that runs always and the other one which runs only when POPULAR_PLUGINS is disabled?

describe( 'Block transforms', () => {
	const transformStructure = {};
	const testTable = flatMap( ... );
	const testMessageFormat = 'block %s in fixture %s into the %s block';

	const maybeDescribe = process.env.POPULAR_PLUGINS ? describe.skip : describe;

	maybeDescribe( 'group 1', () => {
		...
	} );
   
	describe( 'group 2', () => {
		...
	} );
} );

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the suggestion the code was updated and simplified to follow it :)

@jorgefilipecosta jorgefilipecosta force-pushed the fix/make-transforms-test-resilient-agains-plugin-transforms branch from 5faa7a5 to 4c7f258 Compare March 29, 2019 14:38
Copy link
Member

@gziolo gziolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It ended up easier than I could anticipate. Great job @jorgefilipecosta :)

:shipit:

@gziolo gziolo added this to the 5.4 (Gutenberg) milestone Mar 29, 2019
@jorgefilipecosta jorgefilipecosta merged commit 39171ca into master Mar 29, 2019
@jorgefilipecosta jorgefilipecosta deleted the fix/make-transforms-test-resilient-agains-plugin-transforms branch March 29, 2019 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants