Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create complete tests for syntax #843

Closed
catamorphism opened this issue Jul 29, 2024 · 3 comments · Fixed by #844
Closed

Create complete tests for syntax #843

catamorphism opened this issue Jul 29, 2024 · 3 comments · Fixed by #844
Assignees
Labels
blocker-candidate The submitter thinks this might be a block for the Technology Preview LDML46.1 MF2.0 Draft Candidate test-suite

Comments

@catamorphism
Copy link
Collaborator

This is from https://github.com/unicode-org/message-format-wg/wiki/Things-That-Need-Doing . I didn't see an existing issue for it.

Full test coverage is probably impossible given the size of the grammar. But there are a couple possibilities:

  1. Manually generate tests from the grammar, with a limited search depth (i.e. expand every production up to a certain depth and write a test that fits it)
  2. Use an exhaustive test generator tool like abnfgen, Eusthasius or Gramtest to generate tests.
  3. Use a fuzzer like abnffuzzer and extract interesting tests to include in the test suite.

As far as option 2, with a few modifications, I was able to run the grammar through abnfgen and Eusthasius. The problem is that both of them generate tests that resemble line noise, suggesting that fuzzing might be preferable.

Option 1 has the advantage that it's easy to write readable, minimal test cases. Of course the disadvantage is that it's tedious.

Any suggestions are welcome.

@catamorphism catamorphism self-assigned this Jul 29, 2024
catamorphism added a commit to catamorphism/message-format-wg that referenced this issue Jul 30, 2024
catamorphism added a commit to catamorphism/message-format-wg that referenced this issue Jul 30, 2024
@catamorphism
Copy link
Collaborator Author

Doing much better than #844 would probably require some automation.

@catamorphism
Copy link
Collaborator Author

Note that completeness also requires exhaustively generating tests that should produce syntax errors, which I have less of an idea how to do systematically (other than through fuzzing.)

@catamorphism
Copy link
Collaborator Author

I wrote a simple random test generator, hard-wired for the MF2 grammar, and was able to find some bugs in the ICU4C parser with the generated test cases.

catamorphism added a commit to catamorphism/message-format-wg that referenced this issue Jul 31, 2024
catamorphism added a commit to catamorphism/message-format-wg that referenced this issue Jul 31, 2024
catamorphism added a commit to catamorphism/message-format-wg that referenced this issue Jul 31, 2024
catamorphism added a commit that referenced this issue Aug 5, 2024
)

* Add syntax tests for unsupported expressions and statements (#843)

* Update test/tests/unsupported-statements.json

Co-authored-by: Eemeli Aro <[email protected]>

---------

Co-authored-by: Eemeli Aro <[email protected]>
catamorphism added a commit that referenced this issue Aug 5, 2024
* Add more exhaustive syntax tests (#843)

* Use 'description' instead of 'comment'

* Remove 'expCleanSrc' property

* Add two more tests

* rm trailing comma
@echeran echeran linked a pull request Aug 5, 2024 that will close this issue
@aphillips aphillips added blocker-candidate The submitter thinks this might be a block for the Technology Preview LDML46.1 MF2.0 Draft Candidate labels Nov 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker-candidate The submitter thinks this might be a block for the Technology Preview LDML46.1 MF2.0 Draft Candidate test-suite
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants