How to mock responses for UI tests?

[rnystrom]

General question about how I could go about writing UI tests that don't actually hit the network? I'd love to create some basic UI tests for GitHawk, but I'm not sure how I can inject some local JSON responses into Apollo so the tests are consistent and don't hit the network.

Any pointers to what I could do?

[MrAlek]

Once #186 is in, you could just add a terminating link returning fake responses.

Right now, I think the caching system could be used. @martijnwalraven was gonna add a function for injecting results straight into the cache, don’t think that’s in yet though.

[rnystrom]

Thanks @MrAlek! Are you talking about URL cache or the built-in Apollo cache? Eg I would need to record a run and store the cache contents in the test bundle? Or something like that.

_{Sent with GitHawk}

[MrAlek]

Using the built-in Apollo cache. I don't know if you could pre-warm the cache by recording a run and just copying the sqlite cache db into the test bundle. Otherwise, you could inject results runtime once Martijn has exposed the function needed.

[martijnwalraven]

You could also initialize the store with records before each test, that is how the Apollo iOS tests work. Something like:

let store = ApolloStore(cache: InMemoryNormalizedCache(records: [
      "QUERY_ROOT": ["hero": Reference(key: "hero")],
      "hero": ["__typename": "Droid", "name": "R2-D2"]
    ])
)
let client = ApolloClient(networkTransport: networkTransport, store: store)

[pittNearsoft]

Hello guys, I have issues mocking my models too. Any update in Apollo about this topic?

Btw, I didn't understand the answer posted above. If you @martijnwalraven could extend your explaining would be great!, thanks!

[designatednerd]

Yes I definitely want to get us handling this case better.

[cameroncooke]

When we investigated GraphQL as a solution and alternative to a Restful API to be consumed by native and web clients we mistakenly assumed mocking was out of the box due to this article https://www.apollographql.com/docs/graphql-tools/mocking/ and others in Google.

Only when we went all in did we realise that only the web folks are able to take advantage of...

The strongly-typed nature of a GraphQL API lends itself extremely well to mocking.

Well, only if you're using JavaScript it seems.

We've been in a world of pain trying to mock and stub GraphQL for our UI feature tests where the sut is a black box.

Unit Tests not too bad, we just stub at the network layer but UI tests have been hell. We're using an HTTP transport but as GraphQL is not Restful the URL of the requests is not unique so we can' t just set up a small local web-server and just return stubbed responses.

We thought about parsing the incoming request body to a local server but then we would need to be able to parse GraphQL syntax to identify the type of request in order to return the correct mocked response and that wouldn't even allow for variations i.e. testing error scenarios etc.

The other issue even if we did something like this is keep the mock responses up-to-date with the schema. If we for example kept a bunch of JSON file every time the schema changed we would have to go through a world of pain again while all our tests fail and we would need to generate new JSON stubs and variations.

For me this is the number one issue that is blocking us from really adopting GraphQL in a commercial scope where automation tests are a mandatory requirement to ensure product quality.

It's disappointing that questions like this have been asked since 2017 and very little movement has been made.

If anyone has any ideas or solutions in order to mock GraphQL for UI tests in a robust a future proof way I would really appreciate it.

[cameroncooke]

You could also initialize the store with records before each test, that is how the Apollo iOS tests work. Something like:

let store = ApolloStore(cache: InMemoryNormalizedCache(records: [
      "QUERY_ROOT": ["hero": Reference(key: "hero")],
      "hero": ["__typename": "Droid", "name": "R2-D2"]
    ])
)
let client = ApolloClient(networkTransport: networkTransport, store: store)

Two issues with this:

1. With anything other than a trivial graph as shown in the example is going to be unwieldy to maintain using the literal dictionary Swift syntax.
2. A UI test can't change the system under test, that's kind of the point so injected mocked data into the cache is not an option from the test runner.

[designatednerd]

One thing I'm going to be looking at after we get 0.13 out the door is potentially making it possible to pass in a URLSession. That way you can mock out responses with tools like OHHTTPStubs or VOKMockURLProtocol, or some other URL protocol of your own design.

Easier mocking is a lower-down reason on the list to do this, but it does seem like it'd be a great side benefit.

[cameroncooke]

We already do this, we've got our own networking framework which has stubing built-in and we've got our own custom NetworkTransport that uses our networking framework.

But it still has the following disadvantages:

1. It's not much use for UI tests unless we somehow change the app under test which kind of defeats the purpose of UI black box testing.
2. Works okay for Unit Testing but is very fragile and breaks when the schema changes as you have to maintain hardcoded mocked responses.
3. Even if we inject an environment variable from the UI test runner and stub the network layer we still have the issue from point 2.

It would be nice if there was a similar solution to graphql-tools for iOS that takes advantage of ...

the strongly-typed nature of a GraphQL API lends itself extremely well to mocking.

[designatednerd]

It's not much use for UI tests unless we somehow change the app under test which kind of defeats the purpose of UI black box testing.

Personally I'd call anything where you're mocking out the API layer gray-box testing rather than black-box testing, but that's probably more of a philosophical argument than a relevant point here.

In terms of purely black-box testing, especially if you're using XCUI, you're right, this is a total pain in the ass because XCUI tests run out of process and changes have to be made within the shipping app to accommodate it (which drives me up a wall and I have been whining about it on Radar since XCUI was introduced).

This is a huge reason why a graphql-tools style mocker is going to be considerably more difficult to replicate on iOS than it is in a web app: Bringing in something from out of process involves a ton of work, and graphql-tools doesn't have to do that because of the architecture of JS testing. To support XCUI black box testing, we'd have to do this.

Works okay for Unit Testing but is very fragile and breaks when the schema changes as you have to maintain hardcoded mocked responses.

In my personal experience this has always been an issue if I'm returning responses from a file, even in REST APIs. When something changes on the backend, you have to account for it in your mocks, or they won't work.

Overall, I agree that this situation is not ideal. The solution I've used in the past is to use KIF for UI testing, since that runs off an in-process unit test target so you can actually pass in mocks that only exist in your test framework. Obviously if you've already invested in XCUI this is a bit of a non-starter, though.

Four years later, I'm still cranky with Apple for not allowing gray-box with XCUI, exactly because of issues like this one. Wish I had a better answer for you.

[cameroncooke]

Yeah Apple have never seemed that invested in testing in general. I kind of understand the reasons for the out-of-process approach XCUITest uses, it's basically the closest to how a user will interact with an app. They have no access to the process of inner workings but it's a pain for feature testing. Will take a look at KIF might be an option.

[designatednerd]

It basically comes from XCUI being adapted from a black-box JS testing framework they used to use for QA, which to me is a wholly different use case than developer testing. I've probably complained about it enough in this thread already though 😛

[designatednerd]

Two new options for this have shipped with 0.15.0:

• You can now pass in a URLSession rather than just a URLSessionConfiguration, so you can set up any existing mocking libraries that take advantage of intercepting things in a session or using NSURLProtocol for mocking.
• There is now an ApolloClientProtocol you can use to fully mock anything the client is doing.

If you have further requests around this or have problems with either of the new methods, please open a new issue. Thank you!

[pigeondotdev]

@martijnwalraven Hey there! How are you defining networkTransport in this snippet? I'm having a hard time finding docs for how to mock this. Any advice or guidance would be greatly appreciated.

let store = ApolloStore(cache: InMemoryNormalizedCache(records: [
      "QUERY_ROOT": ["hero": CacheReference(key: "hero")],
      "hero": ["__typename": "Droid", "name": "R2-D2"]
    ])
)
let client = ApolloClient(networkTransport: networkTransport, store: store)

Thank you! Love Apollo. :)

[calvincestari]

@noisypigeon - I believe this test demonstrates how we're mocking network fetches.

[pigeondotdev]

@calvincestari Thank you! 🙏🏻

[petarbelokonski]

@calvincestari

@noisypigeon - I believe this test demonstrates how we're mocking network fetches.

Since we have ApolloStore with preconfigured RecordSet why do we have to mock the network fetches as well ? What am I missing here - if I add my mock data to the ApolloStore as shown above do I have to also mock the network fetches ? It feels like having the ApolloStore mock should be sufficient - if that is the case how could we add an implementation of the NetworkTransport that doesn't do anything ? Thanks