Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds openapi v3.0.3 unit test spec, includes test cases, autogenerates model tests with them #12619

Merged
merged 43 commits into from
Jul 1, 2022

Conversation

spacether
Copy link
Contributor

@spacether spacether commented Jun 16, 2022

What Is This

This PR adds an auto generated openpai v3.0.3 spec document which includes schemas and tests from Json Schema Draft6 tests
That spec is used to auto generate a python client using the python-experimental generator
That client includes the auto generated tests that came from the json schema test suite
Those tests are run in our CI (in node 4) to verify that the json schema features are working correctly in component schemas for the python-experimental generator.

Why Do This

Because our petstore spec is not the best way to unit test our features. We need a more basic unit test spec which includes json schema and openapi feature tests. This is that spec.

Why Put The tests In components.x-schema-test-examples

There are a number of places one could put the tests case data. They are:

  • component.schemas.SomeComponent.examples: not present in v3.0.3
  • component.schemas.SomeComponent.example: I want to store multiple tests, not just one
  • component.schemas.SomeComponent.x-schema-test-examples: this would work most of the time, but it's possible to have a schema whose value is true or false in v3.1.0 so one could not have that and use SomeComponent.x-schema-test-examples
  • component.examples: there is not any standard field to store whether an example should pass or fail, it's unclear what component schema the example should be testing. Language in the openapi spec and json schema spec leans towards these examples be passing cases NOT failing cases. I want tests cases to be passing and failing
  • component.x-schema-test-examples.SomeComponent = {testName: testCase}: this allows positive and negative test cases, this allows for testing (future) boolean schemas, this allows for storing multiple tests

Details

  • Adds an openapi v3.0.3 test spec built off of json schema draft 6 tests
  • A python file processes draft6 json schema tests and puts them into an openapi spec document
    • Some tests are omitted because openapi excludes them because those features are not supported by openapi
    • Some tests are added because they are openapi specific
  • That openapi spec document is used to generate a sample and auto-gegnerate tests

When writing this PR, I found bugs in swagger-parser that block generating correct code or successfully running some unit tests:

Python-experimental Improvements

  • escapes newline, carriage return, backslash, double quote, tab, form feed, null character in property names + enum values
  • eliminated multiline strings by using the above escaping
  • fixes bug where maxProperties set to 0 was not written as a validation
  • fixed bug where uniqueItems saw 1 == True and 0 == False when they should be different values

Proposed Future Work (out of scope)

  • turn on the oneOf anyOf, allOf, infinite-loop-detection tests
  • test tx and rx of request and response json bodies
  • test serialization and deserialization of parameters(query/path/body/cookie/header)
  • add openapi specific feature tests
  • With this feature, one could use a tool like https://github.com/json-schema-faker/json-schema-faker to generate passing examples, write them into a test spec and then auto-generate tests against schemas + an api.
  • Add a generate command line additional property skippedTests or input file which allows generators to skip specific tests because they would fail

PR checklist

  • Read the contribution guidelines.
  • Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
  • Run the following to build the project and update samples:
    ./mvnw clean package 
    ./bin/generate-samples.sh
    ./bin/utils/export_docs_generators.sh
    
    Commit all changed files.
    This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
    These must match the expectations made by your contribution.
    You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
    For Windows users, please run the script in Git BASH.
  • File the PR against the correct branch: master (6.0.1) (patch release), 6.1.x (breaking changes with fallbacks), 7.0.x (breaking changes without fallbacks)
  • If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

@spacether spacether changed the title Adds 3.0.1 unit test spec Adds 3.0.3 unit test spec Jun 16, 2022
@spacether spacether linked an issue Jun 17, 2022 that may be closed by this pull request
@spacether spacether force-pushed the feat_adds_unit_test_spec branch from 537cf51 to 2624882 Compare June 21, 2022 05:06
@spacether spacether force-pushed the feat_adds_unit_test_spec branch from 2624882 to 9feb1e4 Compare June 21, 2022 05:07
@spacether spacether changed the title Adds 3.0.3 unit test spec Adds openapi v3.0.3 unit test spec Jun 28, 2022
@spacether spacether changed the title Adds openapi v3.0.3 unit test spec Adds openapi v3.0.3 unit test spec which includes test cases Jun 28, 2022
@spacether spacether added Client: Python Enhancement: CI/Test Feature: OAS 3.0 spec support Bleeding Edge Experimental features (e.g. non-standard spec, "futures", or IETF draft) labels Jun 30, 2022
@spacether spacether added this to the 6.0.1 milestone Jun 30, 2022
@spacether spacether changed the title Adds openapi v3.0.3 unit test spec which includes test cases Adds openapi v3.0.3 unit test spec, includes test cases, autogenerates model tests with them Jul 1, 2022
@spacether spacether merged commit 6158274 into OpenAPITools:master Jul 1, 2022
@spacether spacether deleted the feat_adds_unit_test_spec branch July 1, 2022 20:46
@spacether
Copy link
Contributor Author

spacether commented Jul 1, 2022

Golang team, are any of you interested in generating a golang client using this new unit tests spec? It could automatically run the included test cases in model test and verify your supported feature set.
@antihax @grokify @kemokemo @jirikuncar @ph4r5h4d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bleeding Edge Experimental features (e.g. non-standard spec, "futures", or IETF draft) Client: Python Enhancement: CI/Test Feature: OAS 3.0 spec support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Start unit testing... seriously...
2 participants