Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create & Run Integration Tests for Flank #1072

Closed
Sloox opened this issue Sep 1, 2020 · 3 comments · Fixed by #1109
Closed

Create & Run Integration Tests for Flank #1072

Sloox opened this issue Sep 1, 2020 · 3 comments · Fixed by #1109
Assignees

Comments

@Sloox
Copy link
Contributor

Sloox commented Sep 1, 2020

Author the user story for this feature

As a developer, I want to run integration test on flank so I can test the library with a set of default test cases as though i was using flank as a user.

Is your feature request related to a problem? Please describe.
It is linked to the need to test a full production based build of flank, and to test if the minimization of flank will not introduce errors. It can also help with bug testing issues where we can replicate the issue by running it on a production version of flank.

Describe the solution you'd like
Either have it built into test_runner project or a seperate project like flank_scripts, whereby we can run a set of integration tests simply and easily.

@Sloox
Copy link
Contributor Author

Sloox commented Sep 4, 2020

@bootstraponline @jan-gogo @pawelpasterz @piotradamczyk5 @adamfilipow92
do we have any preferences/ideas/comments as to how integrations testing may look like for this task?

shall it appear as a subproject i.e. test_integration with its own gradle build etc? Will there actually be any code? or all gradle commands? Should we head in the direction of the flank_scripts and have a fully fledged sub project that we can run?

At the very least it should in my opinion start by compiling flank and either copying that lib/binary out of the test_runner folder and running a set of tests on that binary so we have a reference point.

Any ideas or suggestions would be greatly appreciated.
WDYT?

@bootstraponline
Copy link
Contributor

bootstraponline commented Sep 4, 2020

Should we head in the direction of the flank_scripts and have a fully fledged sub project that we can run?

Yes. I think we'd create a new Kotlin project responsible for running Flank and verifying the tests execute successfully. The integration tests can be JUnit tests. Inside the Junit test, we'd use a freshly built version of Flank to run an existing apk/ipa and verify that the final result from Firebase Test Lab is what we expect.

I think these should be smoke tests.

Smoke tests are a subset of test cases that cover the most important functionality of a component or system, used to aid assessment of whether main functions of the software appear to work correctly.

https://en.wikipedia.org/wiki/Smoke_testing_(software)

The 3 critical integration tests I see are:

  • Given a test apk on Android, does the test run successfully
  • Given a test bundle for iOS, does the test run successfully on macOS
  • Given a test bundle for iOS, does the test run successfully on Linux

Each test should be simple, designed to execute quickly, and be stable. If we have 3 solid integration smoke tests, then we can think about adding more advanced tests in the future.

@jan-goral
Copy link
Contributor

Each test should be simple, designed to execute quickly, and be stable. If we have 3 solid integration smoke tests, then we can think about adding more advanced tests in the future.

Right, I love the idea to get something simple first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants