test262 tests

[domenic]

One of the requirements for stage 4 is test262 tests. Besides being a process requirement, it's also great for interoperability.

Are there people that are willing to help work on this? I know @caitp has been working on an implementation in V8, which comes with its own tests presumably in mjsunit format; apparently it's now possible to write test262 tests directly in V8 for later upstreaming, which would be one route.

Alternately, the community and/or myself could help port @caitp's tests and write our own. @benjamingr seems to have volunteered :)

One valuable thing we can do besides even writing tests is come up with a test plan. That is, go through the spec and write down all the testable things and edge case behaviors that we want to be sure to hit when writing tests. Just getting that done alone will help the test writers a lot.

[littledan]

cc @leobalter @rwaldron

[benjamingr]

I'd love to help with this but I'm not really sure how to get started.

[gskachkov]

+1 I would like to participate also.

[domenic]

I'd love to help with this but I'm not really sure how to get started.

I think a test plan would be the most valuable thing to get started. If you really want to dive in and start writing tests, you can use e.g. https://github.com/tc39/test262/pull/479/files as a guide.

[caitp]

Here's a starter test plan informed by my experience prototyping. There's plenty of stuff missing, though.

• Symbol.asyncIterator
  1. test exposed on Symbol with correct property descriptor
• AsyncGeneratorFunction builtin
  1. not exposed on global
  2. exposed via .constructor with correct property descriptor, etc
  3. has the right length value
  4. tests relating to CreateDynamicFunction
  5. tests relating to Function.prototype.toString() (similar to other tests upstream in test262 --- I really don't want to be the one to write these ones :))
• For-Await-OF / GetIterator
  1. Test Symbol.asyncIterator is invoked if present on an object during for-await-of (only caller of GetIterator(iterable, async)
  2. Test restriction that Symbol.asyncIterator must return an Object else throw a TypeError
  3. Test that, in the absence of a a Symbol.asyncIterator property (property is undefined or null), Symbol.iterator is invoked instead (and again, if absent, a TypeError is thrown).
  4. Test that in the absence of an asyncIterator symbol, the iterator returned is always an instance of Async-from-Sync Iterator, with [[Prototype]] == %AsyncFromSyncIteratorPrototype%, regardless of what iterator type is returned (see Async-from-Sync iterator section)
• Async-from-Sync Iterator
  1. Unfortunately, short of for-await-of, there isn't a way to observe access to the Async-from-Sync iterator. If no method for programmatically creating an Async-from-Sync iterator is introduced, we need to expose VM hooks for doing it (similar to $.detachArrayBuffer (Use $.detachArrayBuffer API instead of throwing test262#795)).
  2. Test that .next(), .throw() and .return() return Promise instances, which are eventually resolved with expected values.
  3. Test corner cases. In normal Iterators, .return() and .throw() are optional, and for-of loops invoke them conditionally if present. Async-from-Sync iterators always expose them, so we want to check behaviours of when the sync iterator provides them, doesn't provide them, or provides misbehaving versions of them. I don't have a complete list of these corner cases in mind, but I am happy to help get someone started (or finish it myself), just ping me on mozilla IRC in #jslang, or in #v8 on freenode.
  4. Test that, If the value of a sync iterator is a Promise, the resolved value of the Promise becomes the value of the unwrapped iterator result
  5. Similarly, test that if the promise is rejected, the returned Promise from the async-from-sync iterator method is also rejected (Ping @domenic: I think currently Async-from-Sync Iterators just eat these rejections and aren't specified to reject the returned Promise (https://tc39.github.io/proposal-async-iteration/#sec-async-iterator-value-unwrap-functions), but that seems like an oversight. Edit: this seems to work as expected, currently, although I have no idea why it works for rejected Promises, but not resolved Promises. Edit2: I think either A) the spec was not actually eating rejections here and the default promise rejection handler does the right thing, or B) there's a bug in v8 re the default Promise rejection handler, that happens to make things work slightly better/faster :) But we'll look at that elsewhere)
• Await Expressions in Async Generators
  1. Test that the Generator is not resumed when an Await is in progress (Await must block resumption, otherwise resuming from the Await can have the Generator at an unexpected continuation, or even crash if the VM has "invalid jump table index" assertions like V8 does!)
  2. Test that invalid optimizations (such as https://bugs.chromium.org/p/v8/issues/detail?id=5691, https://bugs.chromium.org/p/v8/issues/detail?id=5694#c3) are not present when evaluating AwaitExpressions.
• Async Generators
  1. Test that it's possible to call .next(), .resume() and .throw() during execution of an async generator, and those requests will be honoured in order (order of returned Promise resolution for those requests should match the order the requests were made)
  2. Test that once the async generator is closed, it is not possible to resume it any longer

There's a lot of stuff I've left out of this list:

• yield* in async generators
• Prototype inheritance of builtin AsyncIterators
• For-await-of being allowed in "ordinary" Async Functions
• More in-depth testing of Async Generator resume mechanics (similar to the Sync Iterator tests)

[benjamingr]

Awesome, thanks! @domenic @caitp I'll take a look this Friday and I'll ensure I know what I'm doing that I'll let you know if I get stuck.

[domenic]

Unfortunately, short of for-await-of, there isn't a way to observe access to the Async-from-Sync iterator. If no method for programmatically creating an Async-from-Sync iterator is introduced, we need to expose VM hooks for doing it (similar to $.detachArrayBuffer

I don't think we want to add this kind of direct test to test262. Instead, we only want to test the aspects that are observable from the language, i.e. for-await-of and yield*. If some implementation were able to implement those without every creating a concrete Async-from-Sync iterator class, then that's great, and should be allowed! We shouldn't write test262 tests that they would start failing.

Maybe we should even mention in the spec the aspects of Async-from-Sync iterator that are unobservable. It sounds like there are a lot.

[benjamingr]

Ok, I have a few volunteers and we'll meet and discuss this this Friday.

[caitp]

I think it's worth having tests upstream wrt interactions between the async iterator and sync iterator (and whoever is calling the iterator methods), because those are observable. It's awkward to only test it in for-await-of, especially if eventually there is another user of GetIterator(async) user.

[domenic]

It may be awkward, but I think it's worth writing awkward tests to allow implementations to optimize fully.

[caitp]

well, by "VM Hooks" I'm not necessarily saying "expose Async-from-Sync Iterator to normal JS", I'm saying "allow VM test harnesses to provide a way to perform this thing, even if normal JS code can't"

I'm absolutely in favour of minimizing the cost of the runtime Async-from-Sync iterator proxy, and if subclassing it or manually creating it, or modifying its prototype chain is out of the question, that works for me

[domenic]

Yeah, but my point is that some implementations might not even be able to expose it, if they've optimized it away super-extensively, and so we shouldn't write tests that those implementations are unable to pass (since they are unable to provide $.createAsyncFromSyncIterator at all).

[thefourtheye]

Could someone please change @caitp's test plan, a check-list? We can tick them whenever tests land.

[benjamingr]

Here's a link to @thefourtheye 's work which we'll build on on Friday (if he won't finish it by then :P)

#71

[caitp]

I'm going to start writing these tests today, as I believe my prototype has become feature complete today.

[benjamingr]

Great, it would be very helpful if you kept a list (like thefourtheye suggested) of tests you've already written and tests you'd like help with - I have 5 hours blocked in my schedule to work on this on Friday and thefourtheye already did some work.

[caitp]

Runtime

• %AsyncGeneratorFunction%.[[Extensible]] === true
• %AsyncGeneratorFunction%.[[Call]] exists (is callable)
• %AsyncGeneratorFunction%.[[Construct]] exists (is a constructor)
• %AsyncGeneratorFunction%.[[Prototype]] === %FunctionPrototype%
• %AsyncGeneratorFunction%.name === "AsyncGeneratorFunction" ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
• %AsyncGeneratorFunction%.prototype === %AsyncGenerator% ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: false })
• %AsyncGeneratorFunction%.length === 1 ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
• %AsyncGenerator%.constructor === %AsyncGeneratorFunction% ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
• %AsyncGenerator%.prototype === %AsyncGeneratorPrototype% ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
• %AsyncGenerator%[@@toStringTag] === "AsyncGeneratorFunction" ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
• %AsyncGeneratorPrototype%.[[Prototype]] === %AsyncIteratorPrototype%.
• %AsyncGeneratorPrototype%.[[Extensible]] === true.
• %AsyncGeneratorPrototype%.constructor === %AsyncGenerator% ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
• %AsyncGeneratorPrototype%.next.[[Call]] exists (is callable).
• %AsyncGeneratorPrototype%.next.[[Construct]] does not exist (is not a constructor).
• %AsyncGeneratorPrototype%.next() throws a TypeError when receiver is not an AsyncGenerator instance.
• %AsyncGeneratorPrototype%.next() returns an instance of %Promise%.
• %AsyncGeneratorPrototype%.return.[[Call]] exists (is callable).
• %AsyncGeneratorPrototype%.return.[[Construct]] does not exist (is not a constructor).
• %AsyncGeneratorPrototype%.return() throws a TypeError when receiver is not an AsyncGenerator instance.
• %AsyncGeneratorPrototype%.return() returns an instance of %Promise%.
• %AsyncGeneratorPrototype%.throw.[[Call]] exists (is callable).
• %AsyncGeneratorPrototype%.throw.[[Construct]] does not exist (is not a constructor).
• %AsyncGeneratorPrototype%.throw throws a TypeError when receiver is not an AsyncGenerator instance.
• %AsyncGeneratorPrototype%.throw() returns an instance of %Promise%.
• %AsyncGeneratorPrototype%[@@toStringTag] === "AsyncGenerator" ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
• AsyncGeneratorDeclaration.prototype.[[Prototype]] === %AsyncGeneratorPrototype.
• AsyncGeneratorExpression.prototype.[[Prototype]] === %AsyncGeneratorPrototype.
• AsyncGeneratorMethod.prototype.[[Prototype]] === %AsyncGeneratorPrototype.
• AsyncGeneratorDeclaration().[[Prototype]] === %AsyncGeneratorPrototype.
• AsyncGeneratorExpression().[[Prototype]] === %AsyncGeneratorPrototype.
• AsyncGeneratorMethod().[[Prototype]] === %AsyncGeneratorPrototype.
• Promises returned from async iterator protocol methods of %AsyncGeneratorPrototype% are resolved in FIFO order.
• AsyncGeneratorExpression YieldExpression with an AwaitExpression as its operand, where the AwaitExpression's operand waits for a Promise or Promise subclass (and invokes .then()).
• AsyncGeneratorExpression YieldExpression with an AwaitExpression as its operand, where the AwaitExpression's operand waits for a custom non-Promise Object with a then method (and then method is invoked)
• AsyncGeneratorExpression YieldExpression with an AwaitExpression as its operand, where the AwaitExpression's operand waits for a non-thenable.
• AsyncGeneratorExpression YieldExpression with a YieldExpression as its operand

Grammar

• AsyncGeneratorDeclaration throw a SyntaxError if BindingIdentifier is "await" and function is nested inside of an AsyncFunctionDeclaration, AsyncFunctionExpression or AsyncMethod, AsyncGeneratorDeclaration, AsyncGeneratorExpression or AsyncGeneratorMethod,
• AsyncGeneratorDeclaration throw a SyntaxError if BindingIdentifier is "yield" and function is nested inside of a GeneratorDeclaration, GeneratorExpression, GeneratorMethod, AsyncFunctionExpression or AsyncMethod.
• AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains yield (invalid BindingIdentifier)
• AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains await (invalid BindingIdentifier)
• AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains AwaitExpression
• AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains SuperCall
• AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains SuperProperty
• AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains YieldExpression
• AsyncGeneratorDeclaration throw a SyntaxError if AsyncGeneratorBody contains SuperCall
• AsyncGeneratorDeclaration throw a SyntaxError if AsyncGeneratorBody contains SuperProperty
• AsyncGeneratorDeclaration throw a SyntaxError if ContainsUseStrict of AsyncGeneratorBody is true and IsSimpleParameterList of FormalParameters is false.
• AsyncGeneratorDeclaration throw a SyntaxError if any element of the BoundNames of FormalParameters also occurs in the LexicallyDeclaredNames of AsyncGeneratorBody.
• AsyncGeneratorExpression throw a SyntaxError if BindingIdentifier is "await".
• AsyncGeneratorExpression throw a SyntaxError if BindingIdentifier is "yield".
• AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains yield (invalid BindingIdentifier)
• AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains await (invalid BindingIdentifier)
• AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains BindingIdentifier
  arguments in strict mode
• AsyncGeneratorExpression throw a SyntaxError if function BindingIdentifier is
  arguments in strict mode
• AsyncGeneratorExpression throw a SyntaxError if function BindingIdentifier is
  eval in strict mode
• AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains BindingIdentifier eval in strict mode
• AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains AwaitExpression
• AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains SuperCall
• AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains SuperProperty
• AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains YieldExpression
• AsyncGeneratorExpression throw a SyntaxError if AsyncGeneratorBody contains SuperCall
• AsyncGeneratorExpression throw a SyntaxError if AsyncGeneratorBody contains SuperProperty
• AsyncGeneratorExpression throw a SyntaxError if ContainsUseStrict of AsyncGeneratorBody is true and IsSimpleParameterList of FormalParameters is false.
• AsyncGeneratorExpression throw a SyntaxError if any element of the BoundNames of FormalParameters also occurs in the LexicallyDeclaredNames of AsyncGeneratorBody. (let variables)
• AsyncGeneratorExpression throw a SyntaxError if any element of the BoundNames of FormalParameters also occurs in the LexicallyDeclaredNames of AsyncGeneratorBody. (const variables)
• AsyncGeneratorExpression throw a SyntaxError if LabelIdentifier is yield (invalid LabelIdentifier in [+Yield] production)
• AsyncGeneratorExpression throw a SyntaxError if LabelIdentifier is await (invalid LabelIdentifier in [+Await] production)
• AsyncGeneratorExpression throw an early ReferenceError if AsyncGeneratorExpression is the target of an assignment operation or count operation
• AsyncGeneratorExpression throw a SyntaxError if there is a line terminator between yield and *.
• AsyncGeneratorExpression YieldExpression parsed as an ExpressionStatement
• AsyncGeneratorExpression YieldExpression with automatic semicolon insertion between yield and operand yields undefined
• AsyncGeneratorExpression yield* YieldExpression does not insert semicolon if line terminator between * and operand.
• AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains yield (invalid BindingIdentifier)
• AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains await (invalid BindingIdentifier)
• AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains duplicate entries.
• AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains AwaitExpression
• AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains YieldExpression.
• AsyncGeneratorMethod throw a SyntaxError if HasDirectSuper of AsyncGeneratorMethod is true.
• AsyncGeneratorMethod throw a SyntaxError if ContainsUseStrict of AsyncGeneratorBody is true and IsSimpleParameterList of UniqueFormalParameters is false.
• AsyncGeneratorMethod throw a SyntaxError if any element of the BoundNames of UniqueFormalParameters also occurs in the LexicallyDeclaredNames of AsyncGeneratorBody.

[caitp]

There's a bunch of things to check off (I know some of these have been covered already).

There's a ton more stuff to add to the list, though.

[domenic]

I wonder if there's a better way to organize this than GitHub checklists, given that those require write access to check. Trello board? Separate shared repo? I dunno, suggestions welcome.

[benjamingr]

Going to start working on this now. Separate repo and trello board sound good.

[benjamingr]

WIP - doing (updating as I go):

• %AsyncGeneratorPrototype%.next() throws a TypeError when receiver is not an AsyncGenerator instance.
• Added tests for Symbol.asyncIterator
• Added tests under built-ins

[domenic]

%AsyncGeneratorPrototype%.next() throws a TypeError when receiver is not an AsyncGenerator instance.

This seems to be incorrect per spec actually. It should reject with a TypeError.

[sashakru]

WIP - doing (updating as I go):

AsyncGeneratorDeclaration throw a SyntaxError if AsyncGeneratorBody contains SuperProperty
AsyncGeneratorDeclaration throw a SyntaxError if AsyncGeneratorBody contains SuperCall
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains YieldExpression
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains SuperProperty
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains SuperCall
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains AwaitExpression
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains await (invalid BindingIdentifier)
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains yield (invalid BindingIdentifier)

[benjamingr]

@domenic ptal at https://github.com/caitp/test262/pull/2/files and let me know if what we're doing makes sense.

[benjamingr]

This seems to be incorrect per spec actually. It should reject with a TypeError.

Would you prefer it if we wrote tests for the current spec and update when it updates or would you prefer it if we test it rejects?

Also, I'm on IRC if you want more direct comm

[domenic]

I don't understand the question? The current spec says it rejects.

[benjamingr]

@domenic roger, will fix.

Question about: AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains duplicate entries.:

async functions don't do this outside of strict mode - do async generators throw in sloppy mode?

[caitp]

Yeah, you're right, should be written as rejects promise

Would you prefer it if we wrote tests for the current spec and update when it updates or would you prefer it if we test it rejects?

The current spec says to reject, not throw, and it's likely to stay that way. In async functions/generators, pretty much any instance of "throw" really means reject, other than for early errors

[caitp]

async functions don't do this outside of strict mode - do async generators throw in sloppy mode?

These are specific to the Method versions, which always have strict formal parameters whether in strict mode or not

[benjamingr]

@caitp can you please take a look at the start of the work (as a PR to your fork of test262) and let us know if we're on the right track or not?

[benjamingr]

@domenic are async generators subclasses of AsyncFunction or Function? From the spec it looks like they're a direct subclass of Function and I'm wondering if that's intentional (just making sure).

[domenic]

Function. That's intentional.

[sashakru]

Hey,

how can I test this one: AsyncGeneratorDeclaration throw a SyntaxError if BindingIdentifier is "yield" and function is nested inside of a GeneratorDeclaration, GeneratorExpression, GeneratorMethod, AsyncFunctionExpression or AsyncMethod? Is there any specific folder to put this tests?

[Linkgoron]

Hi,

Added tests for:
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains yield (invalid BindingIdentifier)
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains await (invalid BindingIdentifier)
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains duplicate entries.
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains AwaitExpression
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains YieldExpression.
AsyncGeneratorMethod throw a SyntaxError if HasDirectSuper of AsyncGeneratorMethod is true.

[benjamingr]

We're done for now - @thefourtheye might add more tests later today. All collaborators (me, @sashakru , @Linkgoron and @thefourtheye signed the CLA) and I'll rebase after today so it's all neatly done in a single PR (to @caitp's own PR for test262 for easy validation)

[littledan]

Great to see more contributors to test262 here!

[benjamingr]

@caitp asked that we submit separate PRs to test262 (we'll do so this Friday). Let us know if there is anything else you'd like us to take a look at

…

On Wed, Jan 18, 2017 at 2:43 AM, littledan ***@***.***> wrote: Great to see more contributors to test262 here! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQSzYB7TLxlBlUHjlEzmYM3QfretJTNks5rTWA8gaJpZM4Le2va> .

[littledan]

Separate PRs seem like a good idea, for licensing at the very least. Let me know if you need any help getting that set up. Looks like there's still a bunch more checkmarks unchecked, so if you're available for it, then just plugging away sounds great.

[caitp]

I think the checklist is pretty incomplete, anyways :( there's not really much mention of for-await-of or it's interactions with rejected promises or AsyncIteratorClose.

I'm sure there's a ton of stuff left to add.
Has anybody hosted a separate checklist that more people have write access to? That would make this easier to track.

[benjamingr]

I'll do more work on Friday to ensure that our PRs are split into separate items - then I'll look at the tests for iterators and for async functions and see what's tested. Hopefully I'll be able to figure it out.

[littledan]

I don't think it's necessary to split the PRs up to be small. The more important thing is that the PRs are filed against the upstream test262, rather than Caitlin's repository. It's fine for a commit to have tons of tests in it or just one test.

[arai-a]

started writing tests for yield* in async generator
tc39/test262#844

[domenic]

@benjamingr @thefourtheye would you mind submitting your tests to the test262 repository directly?

@arai-a sorry for missing that ping. I will give feedback on the tests on Monday!

[arai-a]

another small chunk for abrupt cases in yield* in async generator.
tc39/test262#873

[caitp]

and tc39/test262#871

[arai-a]

added another chunk in yield* tc39/test262#883

[jburton-scottlogic]

Was a list ever created to track the remaining tests?