Some UI tests fail when I run all the test suite at once

[MiguelAFH]

In what area do you have a technical challenge?

Xcode, Simulator & Previews

Description

I am testing pulling my project from Assignment2 from GitHub, opening it and running all the tests. When I do so, a couple of UI tests fail with the following error:

The test runner exited with code -1 before finishing running tests. This may be due to your code calling 'exit', consider adding a symbolic breakpoint on 'exit' to debug.

However, if I run the same suite of tests a second time it works with no issues. I'm mostly concerned that this behavior might happen during grading.

Reproduction

1. Git clone my project to simulate grading.
2. Run the entire test suite.

Expected behavior

It should be able to download and run execute all tests with no errors. Each test is isolated and the entire suite runs with no errors after the first run.

Additional context

No response

Code of Conduct

• I agree to follow this course's Code of Conduct and Contributing Guidelines

[vishnuravi]

Hi @MiguelAFH, can you provide a link to your repository here so we can try to reproduce this issue?

[MiguelAFH]

Here it is https://github.com/MiguelAFH/CS342-Assignment2

[PSchmiedmayer]

@MiguelAFH I tried to clone your repo and reproduce the errors but could not see a test error like the one noted above. It could most likely be from a simulator that takes a longer time to boot (Xcode by default tries to spin up multiple ones for the different tests, especially ones that can be parallelized including the launch tests that are in an empty project setup).

I would suggest the following:

1. Remove all the placeholder tests and UI test files that are not needed and should not effect your test coverage or test any of your application code such as the ones that are part of an empty Xcode project by default to reduce the number of parallel test that could potentially run.
2. Try to limit the number of parallel executions by
   a. Creating a test plan based on the autogenerated one (see instructions found in the Apple Developer documentation: https://developer.apple.com/documentation/xcode/organizing-tests-to-improve-feedback).
   b. Navigate to the test plan and disable parallelization in the test plan options:

[MiguelAFH]

Thank you, that solved the problem!

[PSchmiedmayer]

You're welcome 👍