Script that creates web-platform-test wrappers for test262's tests #8980

dpino · 2018-01-10T16:11:33Z

The scripts reads out 'test262/test' and generates web-platform-test wrappers for each test. The generated tests are stored at 'js/test262'.

Each generated web-platform-test runs the target test262 test in 4 different agents: IFrame, Window, DedicatedWorker and SharedWorker. For each agent usually two tests are generated: one in strict-mode and another one in non-strict mode (unless the test indicates otherwise).

For IFrame and Window agents an HTML file is created on-the-fly containing the target test. In the case of DedicatedWorker and SharedWorker a JavaScript test is created on-the-fly containing the target test. The harnessing code runs the test and checks whether there was any error.

Currently when running the generated web-platform-test wrappers many tests fail. More details at issue #8308

I post this PR with no intention of merging it yet, but to gather feedback and discuss what needs to be done from here to get the PR landed.

This change is

The generated tests run the target test262 within different agent. Currently supported agents are: IFrame, Window, DedicatedWorker and SharedWorker. For each agent two tests are generated, one in strict mode and another in non-strict mode, unless the test indicates otherwise. The generated tests are placed at js/test262/.

dpino · 2018-01-10T16:24:08Z

To generate the tests target dir should point to root folder of test262 (https://github.com/tc39/test262).

$ ./test262-to-wpt test262
Generating web-platform-test wrappers for test262
Output: js/test262

Now run one of the generated WPT tests, for instance:

$ ./wpt run firefox js/test262/intl402/Date/prototype/toLocaleDateString/13.3.2_L15.html
web-platform-test
~~~~~~~~~~~~~~~~~
Ran 9 checks (1 tests, 8 subtests)
Expected results: 9
OK

Helper script to run all the tests and save output to file. Since several tests fail the tests are run with no-restart-on-unexpected flag on, to speed up overall running time.

run-all.sh

#!/usr/bin/env bash

FOLDERS=(
   annexB
   built-ins
   harness
   intl402
   language
)

for each in ${FOLDERS[@]}; do
   time ./wpt run --no-restart-on-unexpected firefox "js/test262/test/${each}" | tee "${each}.output"
done

dpino · 2018-01-10T16:41:05Z

Some considerations regarding the PR:

The generated web-platform-tests depend on test262's harness. Test262's harness code is expected to be found at folder 'resources/test262/harness'. This code is not part of the PR. Perhaps it should be copied when the tests are generated.
Should the generated tests be part of web-platform-test codebase? If so it would be necessary to discuss an updating process. Perhaps by maintaining a file with a timestamp for the original tests and generated test. Ideally generated test should not be manually modified.
Supporting ServiceWorkers is tricky as they cannot be generate on-the-fly with a Blob. Probably support can be added if the ServiceWorker test is generated beforehand along with the WPT generation process.
262 tests that use the JavaScript Shell Agent API should be treated in a special manner, covering all the possible scenarios for the agents supported (See Generalize tests that use the agent abstraction tc39/test262#1351).

ghost · 2018-01-10T17:13:53Z

Build PASSED

Started: 2018-01-28 21:43:40
Finished: 2018-01-28 21:59:45

View more information about this build on:

rwaldron · 2018-01-10T17:22:21Z

resources/test262-harness.js

+    return global.$262;
+}
+
+// TODO: $DONE is used by Promise tests and others, but I don't know from where to fetch it.


$DONE is added by test262-compiler: https://github.com/bterlson/test262-compiler/blob/master/lib/compilerdeps.js, but the plan is to eliminate the test262-compiler dependency by consolidating the "compiler" operations and boilerplate into test262-stream (bocoup/test262-stream#7). Once that's complete, test262-harness is going to be updated to use test262-stream.

rwaldron · 2018-01-10T17:27:11Z

I recommend blacklisting the test262/test/harness directory, this is just tests for test262's harness itself and is of no value outside of test262.

rwaldron · 2018-01-10T18:30:14Z

resources/test262-agent-harness.js

+    test = "'use strict';\n" + test;
+  }
+  if (attrs.type == 'SyntaxError' && attrs.phase == 'runtime') {
+      // If phase is 'runtime' the error will be caught here. If phase is 'early' the


negative: phases are more specific in the latest version (2.0.0) of https://github.com/tc39/test262/blob/master/INTERPRETING.md, see negative for more details, but the short version is that early is now parse.

You can safely check for both for the time being.

rwaldron · 2018-01-10T19:00:29Z

resources/test262-agent-harness.js

+      // error will be handled at the window.onerror event.
+      output.push('try { eval("');
+      output.push(escapeEvalExpr(test));
+      output.push('"); }\ncatch (e) { assert.sameValue(e instanceof SyntaxError, true); }');


The condition above will match tests containing the following:

negative: phase: runtime type: SyntaxError

The only tests that contain this particular set of attributes are those that test module code, direct eval and indirect eval:

eval-code/direct/var-env-global-lex-non-strict.js

eval-code/indirect/parse-failure-2.js

module-code/instn-iee-err-ambiguous-as.js

module-code/instn-iee-err-ambiguous.js

module-code/instn-iee-err-circular-as.js

module-code/instn-iee-err-circular.js

module-code/instn-iee-err-dflt-thru-star-as.js

module-code/instn-iee-err-dflt-thru-star.js

module-code/instn-iee-err-not-found-as.js

module-code/instn-iee-err-not-found.js

module-code/instn-named-err-ambiguous-as.js

module-code/instn-named-err-ambiguous.js

module-code/instn-named-err-dflt-thru-star-as.js

module-code/instn-named-err-dflt-thru-star-dflt.js

module-code/instn-named-err-not-found-as.js

module-code/instn-named-err-not-found-dflt.js

module-code/instn-named-err-not-found.js

module-code/instn-star-err-not-found.js

These can't be nested in a try { eval(...) }..., eg. nesting the module code is a SyntaxError that this test harness code has created, not the test itself.

Here's a better illustration from firefox nightly:

This is code exactly as prepareTest(...) returns it:

try { eval("import x from './instn-named-err-not-found-empty_FIXTURE.js';"); } catch (e) { assert.sameValue(e instanceof SyntaxError, true); }

If I modify it to:

try { eval("import x from './instn-named-err-not-found-empty_FIXTURE.js';"); } catch (e) { console.log(e); }

And put it in an html file, with the same <script type="text/javascript"> (this is important) :

<!DOCTYPE html> <script type="text/javascript"> try { eval("import x from './instn-named-err-not-found-empty_FIXTURE.js';"); } catch (e) { console.log(e); } </script>

This will log:

SyntaxError: import declarations may only appear at top level of a module

Which results in a false positive for this test.

If I change that to (<script type="module"> is the change):

<!DOCTYPE html> <script type="module"> try { eval("import x from './instn-named-err-not-found-empty_FIXTURE.js';"); } catch (e) { console.log(e); } </script>

...The same error occurs. Also, in neither case does a network request get made for instn-named-err-not-found-empty_FIXTURE.js.

Finally, when I run this:

<!DOCTYPE html> <script type="module"> import x from './instn-named-err-not-found-empty_FIXTURE.js'; </script>

The outcome is:

SyntaxError: import not found: default

Which is the SyntaxError this test was written to verify. There is also a network request made for instn-named-err-not-found-empty_FIXTURE.js. (I did this whole exercise in Firefox Nightly with dom.moduleScripts.enabled set to true and Safari Technology Preview.

Note: I had to manually make a file called instn-named-err-not-found-empty_FIXTURE.js, because the real instn-named-err-not-found-empty_FIXTURE.js has been turned into a file called instn-named-err-not-found-empty_FIXTURE.html, with all of the same treatments. Easily fixed by blacklisting _FIXTURE.js files.

Nice catch! I will rework the code as suggested.

rwaldron · 2018-01-10T19:10:25Z

resources/test262-agent-harness.js

+
+  <!-- Test262 required libraries -->
+  <script src="http://localhost:8000/resources/test262/harness/assert.js"><\/script>
+  <script src="http://localhost:8000/resources/test262/harness/sta.js"><\/script>


This domain doesn't work if the tests are run from http://web-platform.test:8000 (in fact, my localhost:8000 is not the same as http://web-platform.test:8000)

Add .sub. to the filename and then you can use substitutions like {{host}} c.f. the rather outdated documentation at https://wptserve.readthedocs.io/en/latest/pipes.html#sub

(or use just paths, of course, in this case, but in general the feature is there if you need it).

@jgraham to clarify, do you mean, eg:

<script src="http://{{host}}:{{ports[http][0]}}/resources/test262/harness/sta.js"><\/script>

Thanks for the hint. I wasn't happy with hardcoding the host and the port but I couldn't find other way to make it work. I'll fix.

rwaldron · 2018-01-10T19:12:34Z

test262-to-wpt

+  <head>
+    <meta charset="utf-8">
+    <title>###TITLE###</title>
+    <meta name="help" href="https://storage.spec.whatwg.org/#dom-storagemanager-persisted">


I don't think you meant to include this?

Right, likely a copy-paste error.

rwaldron · 2018-01-10T20:10:06Z

resources/test262-harness.js

+			if (typeof postMessage !== 'function') {
+				throw new Error('No method available to detach an ArrayBuffer');
+			} else {
+                postMessage(null, '*', [buffer]);


Strange indentation 0_o

It seems there's a mix up of tabs and spaces in that file. I will review all the files and make sure there are only spaces.

rwaldron · 2018-01-11T16:02:35Z

I know that @jugglinmike and @jgraham have discussed this extensively in the past and I want to make sure that all of their interests/concerns are heard and/or addressed.

If a test expects a SyntaxError, regardless of the value of phase, the error is always managed in window.onerror.

Replace CR + LF for LF only. Replace CR for LF.

Tests that use ES functions such as createRealm, evalScript or dettachArrayBuffer are not supported in a Worker context. These operations are implemented in a Window context by appending new elements to the DOM (an iframe in the case of createRealm and a script element in the case of evalScript). Workers don't have access to the Window object. On the other hand, creating a new realm by instantiating a new Worker cannot be a solution. The new realm (a Worker) won't provide the same functions as a Window, which what the tests expect to find after calling createRealm (for instance $262.createRealm().eval(...))

Tests marked with flag 'async' should include doneprintHandle.js. Implement ES Shell function 'print' that receives a message 'Test262:AsyncTestComplete' in case the test completed successfully. Otherwise, throw error. If print is never called, rely on WPT timeout mechanism.

Tests with a negative property are supposed to throw an Exception. The thrown exception should have the same type as negative.type.

Syntax errors, the type that are emitted during parse or early phase, cab be caught if the test is wrapped within a try/catch.

For many tests Firefox throws a SyntaxError instead of a ReferenceError.

Many module-code tests fail cause they try to import or export a JavaScript module which path is no longer valid. It's necesary to replace the path with an URL. This change implies to activate CORS in the WPT WebServer.

dpino · 2018-01-28T21:59:24Z

I have pushed some more commits that help fixing several tests.

The most important change it's fixing tests that are related with modules. The tests complain they could not locate the module, since the path was broken. The following error was returned:

SyntaxError: import declarations may only appear at top level of a module

I tried to fix the path by using relative and absolute paths but it only worked if I replaced the original path with an URL. This led me to activate CORS in the WPT WebServer plus hardcoding "host" and "port" in the URL (which I know it's wrong). I tried to encode the URL using placeholders and passing ?pipe=sub to the URL that points to the target file containing the test, but it didn't work in this case (does this mechanism work with modules import too?).

foolip · 2019-10-20T14:17:20Z

@dpino what is the status of this? I see this has been stale for almost two years, but if you do want to keep working on it, I think it'd be appropriate to put it through the RFC process at https://github.com/web-platform-tests/rfcs.

There are also other external test suites that might be interesting to run as part of wpt, or at least get results for into wpt.fyi, see #19438 for a suggested experiment for Gecko's mochitests.

annevk · 2019-10-21T12:59:08Z

This was an experiment funded by Mozilla to see if there's value in running JavaScript tests with different types of agents. We didn't proceed eventually as it seems there doesn't appear to be a lot of value to be gained and properly abstracting everything would be quite a bit of effort.

foolip · 2019-10-21T13:13:46Z

I see, in that case I guess this PR should be closed?

annevk · 2019-10-21T14:16:21Z

I think that's reasonable, but I'll let @dpino make the call.

foolip · 2019-10-28T22:11:02Z

No reply from @dpino, so I'm closing.

wpt-pr-bot added infra test262-to-wpt labels Jan 10, 2018

wpt-pr-bot requested review from ayg, gsnedders and jgraham January 10, 2018 16:11

gsnedders added the do not merge yet label Jan 10, 2018

dpino mentioned this pull request Jan 10, 2018

Testing test262 in different (web platform) agents #8308

Open

rwaldron requested review from rwaldron and jugglinmike January 10, 2018 16:51

rwaldron reviewed Jan 10, 2018

View reviewed changes

dpino added 5 commits January 11, 2018 10:37

Remove tabs

323d4ca

Remove unneeded meta tags

2b0fee3

Include test262-harness at global level

459584b

Parametrized host and port

d1c2076

Blacklist harness and copy _FIXTURE.js files

a9400b4

dpino added 4 commits January 12, 2018 13:20

Use Python's YAML and JSON libraries

6a749ca

Simplify SyntaxError handling

8ba46ad

If a test expects a SyntaxError, regardless of the value of phase, the error is always managed in window.onerror.

Refactor test262_as_html

a41cc55

Include test262-harness.js

788e7ac

dpino added 17 commits January 15, 2018 11:45

Return global.62

bd744ce

Remove trailing whitespaces

47b9621

Remove custom tojson function

5f968b8

Replace carriage-return

add7ec1

Replace CR + LF for LF only. Replace CR for LF.

Return MIME type 'module' if flags includes 'module'

5a18c61

Implement IsHTMLDDA

93b7423

Tests marked with 'raw' flag are only executed in non-strict mode

e452070

Properly handling of negative tests

552f3c6

Tests with a negative property are supposed to throw an Exception. The thrown exception should have the same type as negative.type.

Reintroduce handling of early errors

feed957

Syntax errors, the type that are emitted during parse or early phase, cab be caught if the test is wrapped within a try/catch.

Caught early tests on SyntaxError too

daedd45

For many tests Firefox throws a SyntaxError instead of a ReferenceError.

IsHTMLDDA should return null if argument is null or empty string

d91d047

global should be the global object

586a081

Generate tests for any path

7ccaa23

Improve module tests results

06506f4

Many module-code tests fail cause they try to import or export a JavaScript module which path is no longer valid. It's necesary to replace the path with an URL. This change implies to activate CORS in the WPT WebServer.

Fix run_in_eval

cd1f125

foolip mentioned this pull request Oct 20, 2019

Experiment: run Gecko's mochitests with some adapter code #19438

Closed

wpt-pr-bot added the wptserve label Oct 20, 2019

wpt-pr-bot assigned dpino Oct 20, 2019

foolip closed this Oct 28, 2019

This was referenced Aug 25, 2022

Requirements for scoring results from Test262 web-platform-tests/interop#117

Open

Revive Test262 integration #35621

Closed

foolip mentioned this pull request Jul 8, 2024

[ENHANCEMENT] show support percentage for TC39 features via test262 result data GoogleChrome/webstatus.dev#490

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Script that creates web-platform-test wrappers for test262's tests #8980

Script that creates web-platform-test wrappers for test262's tests #8980

dpino commented Jan 10, 2018 •

edited by wpt-reviewable-bot

Loading

dpino commented Jan 10, 2018 •

edited

Loading

dpino commented Jan 10, 2018

ghost commented Jan 10, 2018 •

edited by ghost

Loading

rwaldron Jan 10, 2018

rwaldron commented Jan 10, 2018

rwaldron Jan 10, 2018 •

edited

Loading

rwaldron Jan 10, 2018 •

edited

Loading

rwaldron Jan 10, 2018 •

edited

Loading

dpino Jan 11, 2018

rwaldron Jan 10, 2018

jgraham Jan 10, 2018

jgraham Jan 10, 2018

rwaldron Jan 10, 2018

dpino Jan 11, 2018

rwaldron Jan 10, 2018

dpino Jan 11, 2018

rwaldron Jan 10, 2018

dpino Jan 11, 2018

rwaldron commented Jan 11, 2018

dpino commented Jan 28, 2018

foolip commented Oct 20, 2019

annevk commented Oct 21, 2019

foolip commented Oct 21, 2019

annevk commented Oct 21, 2019

foolip commented Oct 28, 2019

Script that creates web-platform-test wrappers for test262's tests #8980

Script that creates web-platform-test wrappers for test262's tests #8980

Conversation

dpino commented Jan 10, 2018 • edited by wpt-reviewable-bot Loading

dpino commented Jan 10, 2018 • edited Loading

dpino commented Jan 10, 2018

ghost commented Jan 10, 2018 • edited by ghost Loading

Build PASSED

Choose a reason for hiding this comment

rwaldron commented Jan 10, 2018

rwaldron Jan 10, 2018 • edited Loading

Choose a reason for hiding this comment

rwaldron Jan 10, 2018 • edited Loading

Choose a reason for hiding this comment

rwaldron Jan 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwaldron commented Jan 11, 2018

dpino commented Jan 28, 2018

foolip commented Oct 20, 2019

annevk commented Oct 21, 2019

foolip commented Oct 21, 2019

annevk commented Oct 21, 2019

foolip commented Oct 28, 2019

dpino commented Jan 10, 2018 •

edited by wpt-reviewable-bot

Loading

dpino commented Jan 10, 2018 •

edited

Loading

ghost commented Jan 10, 2018 •

edited by ghost

Loading

rwaldron Jan 10, 2018 •

edited

Loading

rwaldron Jan 10, 2018 •

edited

Loading

rwaldron Jan 10, 2018 •

edited

Loading