Pass WordPress unit tests

[adamziel]

Description

It would be amazing to run PHPUnit tests using WordPress Playground:

• Testing on PHP 5.6 - 8.2 would be as easy as choosing a different PHP.wasm binary shipped in this repo
• It would reveal all the places where Playground behaves differently than real WordPress
• It would open doors to in-browser WordPress IDE where developers may test their changes

Current status

Tests: 14847, Assertions: 46641, Errors: 112, Failures: 68, Warnings: 34, Skipped: 177, Risky: 3.

See the comments for specific failures

cc @hellofromtonya

[adamziel]

Here's the output of Tests_Canonical_PostStatus test suite which relies on the database:

    Warning:       Your XML configuration validates against a deprecated schema.
    Suggestion:    Migrate your XML configuration using "--migrate-configuration"!
    
    ...............................................................  63 / 424 ( 14%)
    ............................................................... 126 / 424 ( 29%)
    ............................................................... 189 / 424 ( 44%)
    ............................................................... 252 / 424 ( 59%)
    ............................................................... 315 / 424 ( 74%)
    ............................................................... 378 / 424 ( 89%)
    ..............................................                  424 / 424 (100%)
    
    Time: 00:04.057, Memory: 144.00 MB
    
    OK (424 tests, 424 assertions)

[adamziel]

There are numerous test failures and many (or all) of them are related to running WordPress on SQLite:

	<p class='wpdberror'><strong>WordPress database error:</strong> [&lt;div style=&quot;clear:both&quot;&gt;&amp;nbsp;&lt;/div&gt;&lt;div class=&quot;queries&quot; style=&quot;clear:both; margin_bottom:2px; border: red dotted thin;&quot;&gt;Queries made or created this session were&lt;br/&gt;
	&lt;ol&gt;
		&lt;li&gt;Raw query:
SELECT TABLE_NAME AS &#039;table&#039;, TABLE_ROWS AS &#039;rows&#039;, SUM(data_length + index_length) as &#039;bytes&#039; FROM information_schema.TABLES WHERE TABLE_SCHEMA = &#039;wordpress_develop_tests&#039; AND TABLE_NAME IN (&#039;wptests_comments&#039;,&#039;wptests_options&#039;,&#039;wptests_posts&#039;,&#039;wptests_terms&#039;,&#039;wptests_users&#039;) GROUP BY TABLE_NAME;&lt;/li&gt;
		&lt;li&gt;Rewritten:
SELECT TABLE_NAME AS &#039;table&#039;, TABLE_ROWS AS &#039;rows&#039;, SUM(data_length + index_length) as &#039;bytes&#039; FROM information_schema.TABLES WHERE TABLE_SCHEMA = &#039;wordpress_develop_tests&#039; AND TABLE_NAME IN (&#039;wptests_comments&#039;,&#039;wptests_options&#039;,&#039;wptests_posts&#039;,&#039;wptests_terms&#039;,&#039;wptests_users&#039;) GROUP BY TABLE_NAME;&lt;/li&gt;
		&lt;li&gt;With Placeholders:
SELECT TABLE_NAME AS  :param_0 , TABLE_ROWS AS  :param_1 , SUM(data_length + index_length) as  :param_2  FROM information_schema.TABLES WHERE TABLE_SCHEMA =  :param_3  AND TABLE_NAME IN ( :param_4 , :param_5 , :param_6 , :param_7 , :param_8 ) GROUP BY TABLE_NAME;&lt;/li&gt;
		&lt;li&gt;Prepare:
SELECT TABLE_NAME AS  :param_0 , TABLE_ROWS AS  :param_1 , SUM(data_length + index_length) as  :param_2  FROM information_schema.TABLES WHERE TABLE_SCHEMA =  :param_3  AND TABLE_NAME IN ( :param_4 , :param_5 , :param_6 , :param_7 , :param_8 ) GROUP BY TABLE_NAME;&lt;/li&gt;
	&lt;/ol&gt;
&lt;/div&gt;&lt;div style=&quot;clear:both; margin_bottom:2px; border: red dotted thin;&quot; class=&quot;error_message&quot; style=&quot;border-bottom:dotted blue thin;&quot;&gt;Error occurred at line 1644 in Function prepare_query. &lt;br/&gt; Error message was: Problem preparing the PDO SQL Statement.  Error was: SQLSTATE[HY000]: General error: 1 near &quot;:param_0&quot;: syntax error &lt;/div&gt;&lt;pre&gt;#0 /Users/cloudnik/www/Automattic/core/wordpress-develop/build/wp-content/db.php(2745): WP_SQLite_DB\PDOEngine-&gt;get_error_message()

cc @aristath

[adamziel]

I just ran the full WordPress test suite on latest CLI integration (on PHP 8.0 and using an actual MySQL database).

I had to remove @runInSeparateProcess annotation from 5 tests as PHPUnit assumes the proc_open is available and calls it even if it isn't (like in this scenario).

Conclusions

Time: 23:36.675, Memory: 234.62 MB – much better than I expected

14,313 tests passed

That's much more than I expected. This is a huge win! Wow!

21 tests errored out

Particularly interesting: strtotime(): Epoch doesn't fit in a PHP integer. That must be because WASM integers are 32 bit. There must be a PHP compilation flag to emulate 64 bit integers. If not, maybe patching PHP wouldn't be too hard.

Most errors are caused by unavailable PHP extensions. These ones are unclear to me:

13) Tests_Media::test_quality_with_image_conversion_file_sizes
Error: Call to undefined method WP_Error::supports_mime_type()

19) Tests_REST_WpRestUrlDetailsController::test_will_return_from_cache_if_populated
Undefined array key "title"

17) WP_REST_Plugins_Controller_Test::test_create_item_and_activate_errors_if_no_permission_to_activate_plugin
An unexpected error occurred. Something may be wrong with WordPress.org or this server&#8217;s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/forums/">support forums</a>. (WordPress could not establish a secure connection to WordPress.org. Please contact your server administrator.)

101 tests failed

101 may seem like a lot, but fortunately the failures reasons were trivial:

• My config customizations. For example: http://example.org/ was expected as a domain, but http://127.0.0.1:2387 was found (WP_TESTS_DOMAIN)
• Outdated WordPress trunk (Bundled themes readme.txt's year needs to be updated to 2023)
• (probably) Customized error settings in php.ini (Failed asserting that exception of type "PHPUnit\Framework\Error\Notice" is thrown.)
• Not running WordPress on a local server
• Not preinstalling the Importer plugin (This test requires the WordPress Importer plugin to be installed in the test suite.)

175 tests were skipped.

The following problems were reported:

This test requires MariaDB 10.2 or later.
This test requires utf8mb4 to not be supported.
Extension intl is required.
Extension fileinfo is required.
Rendering PDFs is not supported on this system.
Extension exif is required.
The image editor engine WP_Image_Editor_Imagick is not supported on this system.
This test requires the XSL extension.
Function imagejpeg is required.
This test requires the index to be truncated.

Specific test suites that failed:

• Tests_Theme_wpThemeJson
• Tests_User_wpListAuthors
• Tests_REST_WpRestUrlDetailsController
• Tests_Sitemaps_Sitemaps
• Tests_Term_GetTerm
• Tests_Privacy_wpPrivacyGeneratePersonalDataExportFile
• Tests_Media_GetPreviousImageLink
• Tests_Media_GetAdjacentImageLink
• Tests_Mail
• Tests_Media
• Tests_Kses
• Tests_Import_Postmeta
• Tests_Import_Parser
• Tests_Import_Import
• Tests_HTTP_HTTP
• Tests_Image_Functions
• Tests_General_FeedLinksExtra
• Tests_Functions_CleanDirsizeCache
• Tests_Functions_Referer
• Tests_Functions_wpNonceAys
• Tests_Feed_Atom
• Tests_Dependencies_Styles
• Tests_Basic
• Tests_Date_GetPermalink
• Tests_Admin_wpSiteHealth
• Tests_Admin_wpPrivacyRequestsTable
• Tests_Admin_wpPostCommentsListTable
• Tests_Admin_IncludesFile

[swissspidy]

I was just thinking about this in the context of https://make.wordpress.org/hosting/handbook/tests/. It would be cool to see WP playground test results on https://make.wordpress.org/hosting/test-results/ eventually.

[adamziel]

One of the problems with SQLite support is that PDO SQLite doesn't seem to support parameters in the AS clause in queries like SELECT bas as :param_0 from foo:

php > $sqlite = new PDO('sqlite:memory');
php > $sqlite->exec("CREATE TABLE foo (bar TEXT)");
php > $stmt = $sqlite->prepare("SELECT bas as :param_0 from foo");
PHP Warning:  Uncaught PDOException: SQLSTATE[HY000]: General error: 1 near ":param_0": syntax error in php shell code:1
Stack trace:
#0 php shell code(1): PDO->prepare('SELECT bas as :...')
#1 {main}
  thrown in php shell code on line 1

Warning: Uncaught PDOException: SQLSTATE[HY000]: General error: 1 near ":param_0": syntax error in php shell code:1
Stack trace:
#0 php shell code(1): PDO->prepare('SELECT bas as :...')
#1 {main}
  thrown in php shell code on line 1

PDO MySQL, however, does support that:

php > $mysql = new PDO('mysql:host=127.0.0.1; port=55001; dbname=wptest','root','pw');
php > $mysql->exec("CREATE TABLE foo (bar TEXT)");
php > $stmt = $mysql->prepare("SELECT bas as :param_0 from foo");
(no errors in sight)

[adamziel]

Turns out there's many ways to write a MySQL query that the plugin won't correctly rewrite to SQLite. One solution would be to stop rewriting SQL queries using regular expressions and consume them accordingly to the correct grammar. The fix would need to be implemented in the upstream plugin.

[adamziel]

The new and improved SQLite support passes nearly all PHPUnit tests – let's bring it over as soon as it's merged into the WordPress/sqlite-database-integration plugin.

[adamziel]

Update: here's the status with the new SQLite plugin:

Tests: 14847, Assertions: 46641, Errors: 112, Failures: 68, Warnings: 34, Skipped: 177, Risky: 3.

Specific failures involve tests like

62) Tests_REST_WpRestUrlDetailsController::test_get_items_fails_for_user_with_insufficient_permissions
Response status is not 403
Failed asserting that 400 is identical to 403.

55) Tests_Menu_Walker_Nav_Menu::test_start_el_with_empty_attributes with data set #7 (array(), '')
Failed asserting that two strings are identical.

47) Tests_Image_Functions::test_wp_crop_image_should_return_correct_file_extension_if_output_format_was_modified
Cropping the image resulted in a WP_Error. No editor could be selected.
Failed asserting that WP_Error Object (...) is not an instance of class "WP_Error".

And multiple Tests_DB failures that are MySQL-specific and not relevant for SQLite.

I'd say this is pretty close!

[adamziel]

@JanJakes this might be interesting for you in exploring the MySQL parser