Run tests also with Selenium 3.x #354

mvorisek · 2022-04-18T07:44:52Z

There were 3 errored and 2 failing tests, see post below for analysis and how they were fixed.

codecov · 2022-04-18T07:46:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.73%. Comparing base (5d15043) to head (e84a258).
Report is 40 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #354      +/-   ##
============================================
- Coverage     90.02%   87.73%   -2.29%     
+ Complexity      150      149       -1     
============================================
  Files             1        1              
  Lines           421      367      -54     
============================================
- Hits            379      322      -57     
- Misses           42       45       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

.github/workflows/tests.yml

mvorisek · 2022-10-23T14:06:26Z

1) Behat\Mink\Tests\Driver\Basic\IFrameTest::testIFrame
WebDriver\Exception\InvalidArgument: data did not match any variant of untagged enum FrameId at line 3 column 1

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:286
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:426
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/behat/mink/src/Session.php:321
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Basic/IFrameTest.php:17

test must be fixed, only integer indexes are accepted, when string id is passed, the following exception is shown:

name key tested, but it is not supported, not handle, nor element itself (might be supported when serialized correctly, but the latest driver does not support it)

WebDriver\Exception\InvalidArgument: invalid argument: 'id' can not be string
(Session info: chrome=84.0.4147.105)
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'DX6B48F375QD', ip: '10.8.136.130', os.name: 'Windows 10',
os.arch: 'amd64', os.version: '10.0', java.version: '12.0.2'

2) Behat\Mink\Tests\Driver\Js\ChangeEventTest::testSetValueChangeEvent with data set "file" ('the-file', '/home/runner/work/MinkSeleniu...e1.txt', '/home/runner/work/MinkSeleniu...e2.txt')
WebDriver\Exception\InvalidArgument: File not found: /home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/../../web-fixtures/file1.txt

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:234
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Container.php:241
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:679
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/behat/mink/src/Element/NodeElement.php:105
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/ChangeEventTest.php:50

the uploaded paths are newly validated if exist and canonical, of course from browser, thus we must use some paths available within the browser container

3) Behat\Mink\Tests\Driver\Js\WindowTest::testWindow
WebDriver\Exception\InvalidArgument: missing field `handle` at line 3 column 1

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:269
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:418
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/behat/mink/src/Session.php:311
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/WindowTest.php:17

Selenium 3.x + Chrome is working, Firefox is broken - and only handle must be passed, main window handle must be retrived from webdriver first, null/empty is not supported, nor name is supported (/w or /wo handle)

impossible to fix https://stackoverflow.com/questions/25760357/get-title-of-window-without-switching-in-selenium-webdriver, developer must pass window handle instead of window name (which can also be duplicate)

(only) handle support fixed in instaclick/php-webdriver#119

null handle fixed by retriving all handles and using the 1st one (the results is ordered, so the 1st one should always be main windows handle)

4) Behat\Mink\Tests\Driver\Custom\TimeoutTest::testInvalidTimeoutSettingThrowsException
Failed asserting that exception of type "\Behat\Mink\Exception\DriverException" is thrown.

/w Selenium 2x the following exception is thrown:

Behat\Mink\Exception\DriverException: Error setting timeout: Unknown wait type: invalid

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:336
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:322
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/tests/Custom/TimeoutTest.php:33

Caused by
WebDriver\Exception\UnknownError: Unknown wait type: invalid

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:319
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:333
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:322
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/tests/Custom/TimeoutTest.php:33

/w Selenium 3.x no exception is thrown with Firefox, in Chrome, exception is thrown, but a different one - WebDriver\Exception\InvalidArgument, test depending on the webdriver impl. removed

5) Behat\Mink\Tests\Driver\Js\EventsTest::testRightClick
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'right clicked'
+'single clicked'

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/EventsTest.php:45

natively via webdriver it seems impossible/broken, contextClick method is also not supported, can be probably fixed by using syn.js #358 for right click - https://github.com/bitovi/syn/blob/v0.15.0/test/mouse_button_test.js#L439 (non upgraded syn.js does not trigger any click $this->trigger($xpath, 'rightClick'); (but 'click' is delivered correctly via syn.js))

$this->wdSession->buttondown(array('button' => 2));
$this->wdSession->buttonup(array('button' => 2));

is not emitting right click either

the only fix seems to migrate to new webdriver actions - https://stackoverflow.com/questions/2171589/how-can-you-do-right-click-using-selenium/19123275#19123275

aik099 · 2022-11-18T10:48:48Z

1) Behat\Mink\Tests\Driver\Basic\IFrameTest::testIFrame
WebDriver\Exception\InvalidArgument: data did not match any variant of untagged enum FrameId at line 3 column 1

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:286
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:426
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/behat/mink/src/Session.php:321
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Basic/IFrameTest.php:17

test must be fixed, only integer indexes are accepted, when string id is passed, the following exception is shown:

name key tested, but it is not supported, not handle, nor element itself (might be supported when serialized correctly, but the latest driver does not support it)

WebDriver\Exception\InvalidArgument: invalid argument: 'id' can not be string
(Session info: chrome=84.0.4147.105)
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'DX6B48F375QD', ip: '10.8.136.130', os.name: 'Windows 10',
os.arch: 'amd64', os.version: '10.0', java.version: '12.0.2'

Does Selenium 3 still follow the JsonWire protocol, that allows id parameter to be a string (see https://www.selenium.dev/documentation/legacy/json_wire_protocol/#sessionsessionidframe)?

Otherwise, an upgrade Selenium is only removing stuff instead of adding it :( .

Offtopic:
Some time ago, in the #298 , we're discussing another driver creation (like WebDriverDriver), that will use https://github.com/php-webdriver/php-webdriver (previously developed at Facebook) internally instead of https://github.com/instaclick/php-webdriver, which supports Selenium 2 only (as per the repo's README.md file).

There are also unmerged/closed PRs with attempts to use https://github.com/php-webdriver/php-webdriver in this driver instead.

Maybe we should move in that direction, @stof ?

aik099 · 2024-11-02T12:27:35Z

1) Behat\Mink\Tests\Driver\Basic\IFrameTest::testIFrame
WebDriver\Exception\InvalidArgument: data did not match any variant of untagged enum FrameId at line 3 column 1

Fixed in #382 .

2) Behat\Mink\Tests\Driver\Js\ChangeEventTest::testSetValueChangeEvent with data set "file" ('the-file', '/home/runner/work/MinkSeleniu...e1.txt', '/home/runner/work/MinkSeleniu...e2.txt')
WebDriver\Exception\InvalidArgument: File not found: /home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/../../web-fixtures/file1.txt

Fixed in #385 .

3) Behat\Mink\Tests\Driver\Js\WindowTest::testWindow
WebDriver\Exception\InvalidArgument: missing field `handle` at line 3 column 1

Fixed in #384 .

4) Behat\Mink\Tests\Driver\Custom\TimeoutTest::testInvalidTimeoutSettingThrowsException
Failed asserting that exception of type "\Behat\Mink\Exception\DriverException" is thrown.

Fixed in #380 , but now test fails again, because no exception is thrown on Selenium 3: #397 .

5) Behat\Mink\Tests\Driver\Js\EventsTest::testRightClick
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'right clicked'
+'single clicked'

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/EventsTest.php:45

This will work only through actions and actions not supported in current https://github.com/instaclick/php-webdriver version. So dead end. Issue created: #381 . Recommended switch to https://github.com/minkphp/webdriver-classic-driver driver if such functionality is needed.

Testing on Selenium 3 was added in #383.

Closing.

mvorisek · 2024-11-02T12:42:40Z

Thank you. I am very happy this funcitonality has landed, but I am quite disappointed that I put a lot of efford into this 2 years ago fow nothing. It would be great to warm developers to not invest into this project any energy. I am sorry to say this. Have a great day!

aik099 · 2024-11-02T13:25:33Z

@mvorisek , it wasn't for nothing. Every contribution is welcome. In this case, this was research work, that allowed PRs creation at a later time.

Due to limited maintainer time it might have looked abandoned, but it's not.

mvorisek · 2024-11-02T13:39:04Z

Thank you for taking care of this project.

aik099 approved these changes Apr 27, 2022

View reviewed changes

stof reviewed Apr 27, 2022

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

mvorisek force-pushed the selenium3_ci branch from e101450 to 6580799 Compare April 27, 2022 18:07

mvorisek force-pushed the selenium3_ci branch from 6580799 to d358fb1 Compare October 16, 2022 16:54

mvorisek marked this pull request as draft October 17, 2022 08:58

Run tests also with Selenium 3.x

87e6203

mvorisek force-pushed the selenium3_ci branch from d358fb1 to 87e6203 Compare October 23, 2022 12:39

This was referenced Oct 23, 2022

Test drop onto itself and hide when dragging minkphp/driver-testsuite#59

Merged

No WebDriver\Session::moveto() call when dragging onto itself #359

Merged

mvorisek force-pushed the selenium3_ci branch from caf6a13 to 99f3677 Compare October 23, 2022 14:00

Fix switchToWindow for Selenium 3.x and Firefox (Chrome is always ok)

3d1c8d1

mvorisek force-pushed the selenium3_ci branch from 99f3677 to 3d1c8d1 Compare October 23, 2022 14:59

Drop invalid timeout test depending on specific WD impl

8c09245

mvorisek mentioned this pull request Oct 23, 2022

Fix tests for Selenium 3.x minkphp/driver-testsuite#60

Closed

Update GH Actions images to fix node 12 warnings

e84a258

This was referenced Feb 24, 2024

Also test on Selenium 3 #383

Merged

The "EventsTest::testRightClick" test fails on Selenium 3.x #381

Open

aik099 closed this Nov 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run tests also with Selenium 3.x #354

Run tests also with Selenium 3.x #354

mvorisek commented Apr 18, 2022 •

edited

Loading

codecov bot commented Apr 18, 2022 •

edited

Loading

mvorisek commented Oct 23, 2022 •

edited

Loading

aik099 commented Nov 18, 2022 •

edited

Loading

aik099 commented Nov 2, 2024

mvorisek commented Nov 2, 2024

aik099 commented Nov 2, 2024

mvorisek commented Nov 2, 2024

Run tests also with Selenium 3.x #354

Run tests also with Selenium 3.x #354

Conversation

mvorisek commented Apr 18, 2022 • edited Loading

codecov bot commented Apr 18, 2022 • edited Loading

Codecov Report

mvorisek commented Oct 23, 2022 • edited Loading

aik099 commented Nov 18, 2022 • edited Loading

aik099 commented Nov 2, 2024

mvorisek commented Nov 2, 2024

aik099 commented Nov 2, 2024

mvorisek commented Nov 2, 2024

mvorisek commented Apr 18, 2022 •

edited

Loading

codecov bot commented Apr 18, 2022 •

edited

Loading

mvorisek commented Oct 23, 2022 •

edited

Loading

aik099 commented Nov 18, 2022 •

edited

Loading