Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: [IT] Add test cases #1316

Merged
merged 32 commits into from
Dec 3, 2020
Merged

test: [IT] Add test cases #1316

merged 32 commits into from
Dec 3, 2020

Conversation

pawelpasterz
Copy link
Contributor

@pawelpasterz pawelpasterz commented Nov 12, 2020 •
edited
Loading

Fixes #1313

This PR introduces a couple of new features

  1. New integrationTests task in integration_test module
  2. Few test cases for IT (sanity robo, multiple apks, multiple devices, filtering tests, ignoring failed, cancelling)
  3. IT test code -- it basically compares output printed to stdout with expected (resources/compare files)

Test Plan

How do we know the code works?

  1. Run ./gradlew build -- to ensure you have flank.jar
  2. Run ./gradlew integrationTests
  3. All tests should finish without error (note: it may take up to 15 min to finish)

@pawelpasterz pawelpasterz self-assigned this Nov 12, 2020
@mergify
Copy link

mergify bot commented Nov 12, 2020

Title does not follow the guidelines of Conventional Commits.
Please adjust title before merge and use one of following prefix:

  • build - Changes that affect the build system or external dependencies (dependencies update)
  • ci - Changes to our CI configuration files and scripts (basically directory .github/workflows)
  • docs - Documentation only changes
  • feat - A new feature
  • fix - A bug fix
  • chore - Changes which does not touch the code (ex. manual update of release notes). It will not generate release notes changes
  • refactor - A code change that contains refactor
  • style - Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
  • test - Adding missing tests or correcting existing tests and also changes for our test app
  • perf - A code change that improves performance (I do not think we will use it)

@pawelpasterz pawelpasterz changed the title [IT] Add test cases test: [IT] Add test cases Nov 12, 2020
@pawelpasterz pawelpasterz marked this pull request as draft November 12, 2020 07:09
@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2020 •
edited
Loading

Timestamp: 2020-12-03 15:39:19
Buildscan url for ubuntu-workflow run 398692444

@codecov-io
Copy link

codecov-io commented Nov 12, 2020 •
edited
Loading

Codecov Report

Merging #1316 (c8e6b0e) into master (106cb4f) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #1316      +/-   ##
============================================
- Coverage     77.81%   77.79%   -0.03%     
  Complexity      705      705              
============================================
  Files           244      244              
  Lines          4693     4693              
  Branches        901      901              
============================================
- Hits           3652     3651       -1     
  Misses          550      550              
- Partials        491      492       +1

@pawelpasterz pawelpasterz force-pushed the 1313-add-it-test-cases branch 2 times, most recently from 11f9ccc to 1e961fa Compare November 12, 2020 08:01
@pawelpasterz pawelpasterz marked this pull request as ready for review November 12, 2020 10:55
@pawelpasterz
Copy link
Contributor Author

There might be some 'strange' compare files formatting (on GH) -- those look normal in IDE/text editor

Sloox
Sloox previously requested changes Nov 12, 2020
View reviewed changes
@pawelpasterz pawelpasterz marked this pull request as draft November 12, 2020 13:18
@adamfilipow92 adamfilipow92 marked this pull request as ready for review November 16, 2020 09:22
@adamfilipow92 adamfilipow92 requested a review from Sloox November 16, 2020 10:50
@pawelpasterz pawelpasterz dismissed Sloox’s stale review November 17, 2020 05:02

Changes implemented

@adamfilipow92 adamfilipow92 self-assigned this Nov 17, 2020
pawelpasterz and others added 23 commits December 3, 2020 16:24
@pawelpasterz
Remove redundant changes
ac1d736
@pawelpasterz
Refactor multiple apks case
bb72e74
@pawelpasterz
Fix filter
a8ecceb
@adamfilipow92 @pawelpasterz
Cr fixes and turn off ios test on windows
e3370a1
@adamfilipow92 @pawelpasterz
Fix android filtered integration test on windows
b842121
@adamfilipow92 @pawelpasterz
Fix RunTimeout integration test on windows
31ef140
@adamfilipow92 @pawelpasterz
Fix test filtering on windows
526d586
@adamfilipow92 @pawelpasterz
Fix Sanity test on windows
0555f87
@adamfilipow92 @pawelpasterz
Fix IgnoreFailed tests on windows
4e93e1c
@adamfilipow92 @pawelpasterz
Update IntergrationTestsUtils.kt
5447245
@adamfilipow92 @pawelpasterz
Fix multiple apks it on windows
6e408f5
@adamfilipow92 @pawelpasterz
Fix multiple apks tests on windows
d6ebc8a
@adamfilipow92 @pawelpasterz
Update TestFilteringIT-compare
687b80b
@adamfilipow92 @pawelpasterz
Update SanityRoboIT-compare
869873b
@pawelpasterz
Review changes
f7f2d48
@pawelpasterz
Update compare files
768d27e
@adamfilipow92 @pawelpasterz
Fix integration tests
fb2c090 
In case when we have - in user name
@adamfilipow92 @pawelpasterz
Fix multiple devices and multiple apk tests
ee8190f
@adamfilipow92 @pawelpasterz
Fix integration tests on windows
7d22941
@adamfilipow92 @pawelpasterz
Update MultipleDevicesIT.kt
1b5df51
@adamfilipow92 @pawelpasterz
Cr fixes
93fbabc
@adamfilipow92 @pawelpasterz
Cr changes
075ff25
@pawelpasterz
Update compare and yml files
99df937
@pawelpasterz pawelpasterz force-pushed the 1313-add-it-test-cases branch from f06b4a4 to 99df937 Compare December 3, 2020 15:27
@piotradamczyk5
Copy link
Contributor

@Mergifyio refresh

@mergify
Copy link

mergify bot commented Dec 3, 2020

Command refresh: success

Pull request refreshed

@mergify mergify bot merged commit 8cc0786 into master Dec 3, 2020
@mergify mergify bot deleted the 1313-add-it-test-cases branch December 3, 2020 15:51
mergify bot pushed a commit that referenced this pull request Dec 11, 2020
@adamfilipow92
feat: Add logic to verify xml results (#1362)
354213d 
Fixes #1315 

## Test Plan
> How do we know the code works?

1. run ```./gradlew integrationTests```
1. integration tests should pass

## Description

Flank should also verify XML results for android and ios integration tests.

This ticket should be merged after #1316 when we update all tests. 

```filter all tests - ios``` is disabled and should be updated in #1388

## Checklist
- [X] Add verification for basic android test
- [X] Add verification for tests from #1316
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Sloox Sloox Sloox approved these changes

@piotradamczyk5 piotradamczyk5 piotradamczyk5 approved these changes

@adamfilipow92 adamfilipow92 Awaiting requested review from adamfilipow92

@zuziaka zuziaka Awaiting requested review from zuziaka

@bootstraponline bootstraponline Awaiting requested review from bootstraponline

@jan-goral jan-goral Awaiting requested review from jan-goral

Assignees

@pawelpasterz pawelpasterz

@adamfilipow92 adamfilipow92

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[IT] Add test cases
6 participants
@pawelpasterz @codecov-io @adamfilipow92 @Sloox @piotradamczyk5 @jan-goral