Allow to return null for mocks / fuzz testing

[amannn]

I've noticed in my apps that null pointer exceptions seem to be the major cause of runtime errors. Ideally these would be prevented by a type system, however I currently don't use one and it would be too much effort to add types to the code bases at this point.

However, the amount of components that connect to API data aren't that many. I'm using the mock capabilities of graphql-tools to generate mock API data that I can feed into those components to test if they generally work. Currently graphql-tools always returns an actual value if a field is nullable.

I was thinking if there was an option to allow nullable properties to return null, I could generate a bunch of sample data and see if my components are able to render with all of them. If the set is large enough, I think I could catch most issues where null checks are missing.

Would you be open to introducing such a flag to graphql-tools to allow nullable properties to actually be null? Something I'm not sure yet, is how this would work if you have explicitly defined mock resolvers for a given field. For my use case I'd like to also possibly skip them, without having to put a condition into all of them.

So something like this would be great:

addMockFunctionsToSchema({
  schema: mockSchema,
  mocks: enhancedMocks,
  randomizeNullableFields: true
});

Or:

addMockFunctionsToSchema({
  schema: mockSchema,
  mocks: enhancedMocks,
  resolveFieldToNull: (some, args, to, identify, the, field) =>
    Math.random() < 0.5
});

The latter example might be helpful for other use cases, as the user can decide per field if it should be null.

What do you think?

[JakeDawkins]

Hey! You're right. There's no way to do this easily right now. One thing you could do would be to use the customizable mocks ability to null out the appropriate fields randomly: https://www.apollographql.com/docs/graphql-tools/mocking.html#Customizing-mocks.

I think we could extend the mocking abilities to include something like this though in the future. Maybe allowNullFieldMocks would be a more descriptive name?

[amannn]

Hey @JakeDawkins, thanks for your reply.

Yes, you could implement it by customizing mocks, however you'd have to implement this for every single mock resolver. That's why I'd prefer a flag, which can also potentially return null for a field where a custom mock resolver is defined.

I like your suggestion for calling it allowNullFieldMocks. Would you consider adding this? Or would you like to see a PR? Thanks!

[amannn]

An even better implementation would be to not randomize null values, but generate a full response with all nullable types filled (as it is currently the case). And then creating the minimum, but complete set of variations of responses where nullable types are set to null. This way the test would also be deterministic and not depending on randomness 🙂

[mschipperheyn]

There is a related more serious issue: #930 without an obvious workaround. If you have a preservedResolver (so, not mocked) mixed with mocked resolvers (preserveResolvers: true), null fields on the non mocked resolvers will also receive mock data.

[mschipperheyn]

If you use partial mocking this will create havoc on your login functionality. I just managed to create a workaround:
Let's say you have a Viewer type that contains your logged in user

type Viewer {
    user: User
}

Query: {
   viewer: (source, params, context) => ({
      user: context.user, // can be undefined
   }),
}

If you use mocking with preserveResolvers: true and no mocks are loaded for this type, user will still get a mocked value. Not good.

The workaround I implemented was explicitly returning null

Query: {
   viewer: (source, params, context) => ({
      user: context.user || null, 
   }),
}

If you don't return null and leave the user undefined, you will still get a mock value. So, perhaps this issue is more related to a value being undefined than it being null

[amannn]

Yep, I think that's a bit a different story. As you mentioned || null returns explicitly null there, that's probably also the solution for the linked issue. When a field is undefined, that's when graphql-tools knows that it should mock the field. I'm not sure if there's a better solution than this.

[willhowlett]

This would be a great feature so am adding a very late +1

[agileurbanite]

👍 +1

[ardatan]

Mocking has been almost rewritten from scratch.
See https://www.graphql-tools.com/docs/mocking/
Closing this issue for now, if this is still relevant please create a new issue. Thanks!

[jmtimko5]

I am still seeing this issue: #4792 would really appreciate any guidance