[CI] Run tests on more browsers #615
Comments
Reinmar
added
status:confirmed
type:task
|
I'm testing the BrowserStack from the Friday and I have to say – I'm soooooo excited! This tool really allows testing our project on different browsers. All important things I described below. There is nothing to say more except the one thing – bootstrap time (everything that is done before starting tests execution) is not counted to "BrowserStack automate testing" time. Integration with existing projectBecause we use Karma, there is no additional stuff to do. The one thing to do is install karma-browserstack-launcher. Available browsersIn order to see all available browser, please follow this instruction – https://www.browserstack.com/automate/rest-api#rest-api-browsers. There are 1193 available browsers which we can use to tests. node -e "console.log( require( './browser.json' ).length );"
1193I chose 4 browsers on different operation systems:
Execute tests on CIIt will work because CI uses the same code which we use. I created an example repository with an integration with Travis and BrowserStack. See the latest build: https://travis-ci.org/pomek/browserstack-test/builds/293608963. How to run it on my local machine?
$ tree -L 1
── ckeditor5
└── ckeditor5-devIf you don't have git clone -b t/ckeditor5/615 [email protected]:ckeditor/ckeditor5-dev.git
cd ckeditor5-dev
lerna bootstrapIf you have the repository already cloned, just check out the branch to # git fetch
git checkout t/ckeditor5/615
lerna bootstrap # or `npm i` from `ckeditor5-dev-tests`
./scripts/switch-to-dev-dev.shIt will link dependencies from the
gulp test --username=BS_USERNAME --access-key=BS_ACCESSKEY --files=autoformat
# or using aliases
gulp test -u BS_USERNAME -a BS_ACCESSKEY -f autoformatAn access token is specified
Plan for the future@Reinmar, I'm waiting for your feedback, so feel free to add it :D I didn't execute all tests using BS so it will be the thing to do. I will prepare a report which shows how many tests fails on other browsers. Then we will see what do to next. ScreenshotsNot too many but few are always better than nothing. gulp test -u ... -a ... -f autoformat -c
Every browser contains a video record:
Coverage reports are generated for each browser:
|
|
I've just prepared the report which shows how many tests fail. Test command: gulp test -u ... -a ... -f [Package / Scope]Legend: Executed Tests / All tests (X FAILED). If I had any problem with the package, I reported an issue in proper repository. A link to the issue is added in the last cell in a row. The table below does not contain
ckeditor5-engineAs I mentioned above, we aren't able to execute all tests on Edge. I decided to execute tests separately:
Dedicated issue: https://github.com/ckeditor/ckeditor5-engine/issues/1177. |
|
I executed the passed tests once again. Unfortunately, Edge does not want to work :( List of the tests- Edge 16.16299.0 (Windows 10 0.0.0) ClassicEditor constructor() ui creates a single div editable root in the view FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ClassicEditor create() "before each" hook for "creates an instance which inherits from the ClassicEditor" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ClassicEditor create() "after each" hook for "creates an instance which inherits from the ClassicEditor" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ClassicEditor create - events fires all events in the right order FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ClassicEditor create - events "after each" hook for "fires all events in the right order" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ClassicEditor destroy "before each" hook for "sets the data back to the editor element" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ClassicEditorUI "before each" hook for "sets #editor" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) Spellchecking integration "before all" hook FAILED - Edge 16.16299.0 (Windows 10 0.0.0) Spellchecking integration "after all" hook FAILED - Edge 16.16299.0 (Windows 10 0.0.0) Bug ckeditor5-typing#59 "before each" hook for "editor does not blow up when deleting last styled character" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) Bug ckeditor5-typing#59 "after each" hook for "editor does not blow up when deleting last styled character" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) UndoEngine integration "before each" hook for "add and undo" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ImageUpload "before each" hook for "should include ImageUploadProgress" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ImageUpload "after each" hook for "should include ImageUploadProgress" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ImageUploadButton "before each" hook for "should include ImageUploadEngine" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ImageUpload "after each" hook for "should include ImageUploadProgress" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ImageUploadButton "before each" hook for "should include ImageUploadEngine" FAILED - Edge 16.16299.0 (Windows 10 0.0.0) ImageUploadButton "after each" hook for "should include ImageUploadEngine" FAILED - Edge 16.16299.0 (Windows 10 0.0.0): Executed 1291 of 1533 (18 FAILED) (skipped 6) DISCONNECTED (1 min 46.771 secs / 11.632 secs)Most of these tests fail because: Safari and Chrome work fine. |
|
Almost all tests listed above after https://github.com/ckeditor/ckeditor5-ui/issues/337 work. This one still fails: |
|
Other than running automated tests, we can use BrowserStack's live testing.
(It will ask you to install additional extension to your browser - it will allow to open your local addresses inside started browser).
You can even put a breakpoint and run debugger in devtools there:
NOTE: testing on Firefox and Edge is free, that means it will not drain your free live testing minutes limit. |
|
Status of things:
|
|
The latest build with changed branches required for BrowserStack – https://travis-ci.org/ckeditor/ckeditor5/builds/311797957. |
|
Yesterday we switched on the BrowserStack for builds on Travis. Current status:
I'm closing because everything works as we want. |
Right now, using Travis, we run tests on Chrome only. This is great for day-to-day development but we don't control much the status of things on different browsers (Firefox, Safari, Edge and some mobile ones).
This is a problem before releases because we're left with many failing tests and it's really hard to figure out at that point whether to care about those. And once we have some red tests already, it's also hard to see if we're not breaking something more when making changes or implementing new features.
BrowserStack
Therefore, let's try to introduce BrowserStack to our CI. I actually don't know what options does it bring so we need a proper research here. But, let's write down some requirements just to start:
master-revisionsandmasterare one thing for sure,Failing tests
Besides dealing with BrowserStack we need to fix our tests on those browsers. Here, we can identify a couple of issues:
The text was updated successfully, but these errors were encountered: