[BUG] Issue with --hypothesis-phases=explicit and GET requests without parameters

[Chamarez]

Description:

When using the --hypothesis-phases=explicit parameter in SCHEMATHESIS, it has been observed that GET requests without parameters are being skipped during testing. This creates a problem for users as they have to resort to adding fake parameters and then try to exclude them from the documentation, which is not an optimal solution.

The example that reproduces the error:

paths:
  /books:
    get:
      security: []
      summary: books
      description: Get list of books
      tags:
        - "books"
      operationId: get_books
      responses:
        '200':
          description: Successful response
        '400':
          description: Bad request

Example with fake param:

paths:
  /books:
    get:
      security: []
      summary: books
      description: Get list of books
      tags:
        - "books"
      operationId: get_books
      parameters:
        - name: fake
          in: query
          description: Fake param for schemathesis
          schema:
            type: string
          required: false
          examples:
            fake:
              value: fake
      responses:
        '200':
          description: Successful response
        '400':
          description: Bad request

To Reproduce
Steps to reproduce the behavior:
Add a get petition without params and use the --hypothesis-phases=explicit configuration.
Schemathesis will skip this test.

Expected behavior
Execute gets requests without params.

Environment :
Docker container, image: schemathesis/schemathesis:stable

[IvanRibakov]

Hi @Stranger6667, this issue is somewhat painful and stops us from transitioning from dredd to schemathesis with piece of mind. I'd be willing to try to put together a PR if you could give some pointers where to start as I'm not familiar with schemathesis/hypothesis internals.

[Stranger6667]

Hi @IvanRibakov

My current thinking is that the interface of Schemathesis CLI is misleading in this regard, or the purpose of the "explicit" phase does not match the expectations.

Essentially, providing --hypothesis-phases=explicit means "verify all examples you can find in the schema" and from this perspective, I'd find it counterintuitive if Schemathesis would generate any examples beyond the ones in the schema (as an empty example still is an example).

My reading of this issue (and, in fact, a few other similar issues) is that often users want to run all explicit examples and generate a test case when an API operation does not declare any examples. Such kind of behavior does not clearly fit the concept of phases, as it involves 2 phases (explicit + generate).

The following is a separate block of information, but I want to mention it as I think it gives a bit more context. The whole area of how Schemathesis handles explicit examples is a bit hacky and involves data generation for required parameters that are missing in examples. It also limits the number of parameter combinations because of combinatorial blowup - but there are no heuristics or anything to choose what combinations should get to the final examples list. However, Hypothesis has such heuristics and it might be better to offload this completely to data generation, but then it won't be a quick run over explicit examples anymore.

In any event, I think that the aforementioned expectation is completely legit, but it might be better to eventually add a new concept of profiles that essentially will cover such use cases without hacking existing abstractions. So, a profile could be seen as a combination of different configuration knobs that together represent some usage scenario.

However, it seems like the before_add_examples hook could work here:

import schemathesis
from schemathesis import Case
import hypothesis


@schemathesis.hook
def before_add_examples(context, examples) -> None:
    if not examples:
        strategy = context.operation.as_strategy()

        # Adopted from Hypothesis source code
        @hypothesis.given(strategy)
        @hypothesis.settings(
            database=None,
            max_examples=1,
            deadline=None,
            verbosity=hypothesis.Verbosity.quiet,
            phases=(hypothesis.Phase.generate,),
            suppress_health_check=list(hypothesis.HealthCheck),
        )
        def example_generating_inner_function(ex):
            examples.append(ex)

        example_generating_inner_function()

Maybe the path forward could be to use the snippet above as some contrib extension (as we already have multiple things like that).

P.S. I noticed that some code samples in the linked documentation page are missing the @schemathesis.hook decorator, I am going to fix that.

[IvanRibakov]

@Stranger6667 Thank you for the expanded answer. Now I can see why you've applied "Needs Design" label. I'm afraid working on a proper solution according to the vision that you laid out above is out of my depth, but I have to say that the workaround that you mentioned works well for us - thanks for sharing it!

[Stranger6667]

@IvanRibakov

You're very welcome!

No worries, I am going to make a PR for this contrib feature soon and hopefully ship it before the end of the week.

[Stranger6667]

I added --contrib-openapi-fill-missing-examples that enables the hook above, so it should provide the capability :) Re: shipping a new release - I think to implement some more fixes related to explicit examples before making a new release. Not sure about the estimation though.