Support headless browsers

[joeyparrish]

Developers working with Shaka Player have always been able to launch headless versions of browsers, for example with --browsers ChromeHeadless. Since Shaka Player started using this launcher, users can no longer do that. (Though --use-xvfb is still available for Linux.)

We should support headless browser variants where possible.

[ppaneksamsung]

I made some progress on this. @joeyparrish please take a look at ppaneksamsung@b7987a7 if you'll have some time and share your opinion if this approach is correct. I'm still working on this because a lot shaka-player's tests fail, for example:

Chrome Headless 80.0.3987.132 (Linux x86_64) TextDisplayer layout using UI moves cues to avoid controls FAILED
        Error: ui-cue-with-controls: Expected 0.9394849000884838 not to be less than 0.95.
            at <Jasmine>
            at Function._callee4$ (test/test/util/util.js:460:46 <- test/test/util/util.js:759:58)
            at tryCatch (node_modules/babel-polyfill/dist/polyfill.js:6900:40)
            at Generator.invoke [as _invoke] (node_modules/babel-polyfill/dist/polyfill.js:7138:22)

[joeyparrish]

Thanks for working on this! I'll take a look at your fork soon.

For the purposes of this, you can ignore "TextDisplayer layout" failures. Those Shaka Player tests are newly-enabled by this module, and it's possible that our thresholds for passing screenshots are too high. (Screenshots depend on rendering, which depends on your GPU, so small differences are to be expected on the edges of things.)

You can disable them for now in Shaka Player by editing test/ui/text_displayer_layout_unit.js and changing filterDescribe to xfilterDescribe:

https://github.com/google/shaka-player/blob/6d76a135e5128dfd47653acea025d0a264d121d5/test/ui/text_displayer_layout_unit.js#L20

Or you can modify the threshold (and send a PR to Shaka Player) by editing the same file and changing minSimilarity:

https://github.com/google/shaka-player/blob/6d76a135e5128dfd47653acea025d0a264d121d5/test/ui/text_displayer_layout_unit.js#L38

[joeyparrish]

I think your changes so far look perfect to me. Please send a PR when you are ready!

[ppaneksamsung]

Okay, I sent a PR.

To summarize:

$ python3 build/test.py --use-xvfb --browsers Edge Chrome Firefox
...
Chrome 80.0.3987.132 (Linux x86_64): Executed 2192 of 2208 (skipped 16) SUCCESS (7 mins 33.103 secs / 7 mins 29.178 secs)
Edge 98.0.1108.43 (Linux x86_64): Executed 2145 of 2208 (29 FAILED) (skipped 63) (7 mins 31.821 secs / 7 mins 28.343 secs)
Firefox 73.0 (Ubuntu 0.0.0): Executed 2144 of 2208 (3 FAILED) (skipped 64) (8 mins 10.906 secs / 8 mins 5.297 secs)
TOTAL: 32 FAILED, 6449 SUCCESS

$ python3 build/test.py --browsers EdgeHeadless ChromeHeadless FirefoxHeadless
...
Chrome Headless 80.0.3987.132 (Linux x86_64): Executed 2192 of 2208 (6 FAILED) (skipped 16) (7 mins 51.683 secs / 7 mins 47.684 secs)
Edge 98.0.1108.43 (Linux x86_64): Executed 2145 of 2208 (29 FAILED) (skipped 63) (7 mins 9.529 secs / 7 mins 6.256 secs)
Firefox 73.0 (Ubuntu 0.0.0): Executed 2144 of 2208 (skipped 64) SUCCESS (8 mins 25.067 secs / 8 mins 19.563 secs)
TOTAL: 35 FAILED, 6446 SUCCESS

In case of Chrome Headless, 5 of fails can be caused by --disable-gpu flag as you pointed out (Expected 0.9394849000884838 not to be less than 0.95.). There are some problems with screenshots on Edge browsers (headless and not), because screenshots are not similar at all (for example: Expected 0.0010140377756393578 not to be less than 0.95.) and there is one flaky test:

Chrome Headless 80.0.3987.132 (Linux x86_64) StreamingEngine gap jumping won't jump large gaps with preventDefault() FAILED
        Error: Expected 2 to be 1.
            at <Jasmine>
            at _callee17$ (test/media/streaming_engine_integration.js:530:25 <- test/media/streaming_engine_integration.js:896:33)
            at tryCatch (node_modules/babel-polyfill/dist/polyfill.js:6900:40)
            at Generator.invoke [as _invoke] (node_modules/babel-polyfill/dist/polyfill.js:7138:22)

Basing on above, I think that there are no issues with headless browsers itself.

[joeyparrish]

The screenshots for Edge on Linux are completely missing. We don't have that combination in our test infrastructure yet. So those 29 are expected until somebody adds them. Sorry for that! The screenshot tests are only very recently available outside the Google test lab, and we're still dealing with a few issues like that.

If you want to send a PR with adjustments to the threshold for screenshot similarity, that would be appreciated!

[joeyparrish]

Your feature was just released in v1.4.0. If you update the dependency in shaka-player and send a PR there, that should make the headless versions available in Shaka tests again. Thanks!