Firefox Behat testing is very fragile

[mvorisek]

Firefox Behat testing is failing too often. Problem should be identified and fixed.

[mvorisek]

  Scenario: search no ajax                              # tests-behat/grid.feature:13
    Given I am on "collection/grid.php?no-ajax=1"       # Behat\MinkExtension\Context\MinkContext::visit()
    Then I search grid for "kingdom"                    # Atk4\Ui\Behat\Context::iSearchGridFor()
    ┌─ @BeforeStep # Atk4\Ui\Behat\Context::closeAllToasts()
    │
    ╳  WebDriver\Exception\JavaScriptError: TypeError: jQuery(...).toast is not a function
    ╳  Stack trace:
    ╳  #0 vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php(166): WebDriver\Exception::factory()
    ╳  #1 vendor/instaclick/php-webdriver/lib/WebDriver/Session.php(436): WebDriver\AbstractWebDriver->curl()
    ╳  #2 vendor/atk4/behat-mink-selenium2-driver/src/Selenium2Driver.php(1068): WebDriver\Session->execute()
    ╳  #3 src/Behat/MinkSession.php(28): Behat\Mink\Driver\Selenium2Driver->executeScript()
    ╳  #4 src/Behat/Context.php(66): Atk4\Ui\Behat\MinkSession->executeScript()

  Scenario: Checkbox click event must not bubble to row click                    # tests-behat/grid.feature:19
    ┌─ @BeforeStep # Atk4\Ui\Behat\Context::closeAllToasts()
    │
    ╳  WebDriver\Exception\JavaScriptError: TypeError: jQuery(...).toast is not a function
    ╳  Stack trace:
    ╳  #0 vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php(166): WebDriver\Exception::factory()
    ╳  #1 vendor/instaclick/php-webdriver/lib/WebDriver/Session.php(436): WebDriver\AbstractWebDriver->curl()
    ╳  #2 vendor/atk4/behat-mink-selenium2-driver/src/Selenium2Driver.php(1068): WebDriver\Session->execute()
    ╳  #3 src/Behat/MinkSession.php(28): Behat\Mink\Driver\Selenium2Driver->executeScript()
    ╳  #4 src/Behat/Context.php(66): Atk4\Ui\Behat\MinkSession->executeScript()

  Scenario: popup column header                                                  # tests-behat/grid.feature:40
    ┌─ @BeforeStep # Atk4\Ui\Behat\Context::closeAllToasts()
    │
     ...

--- Failed scenarios:

    tests-behat/grid.feature:13
    tests-behat/grid.feature:19
    tests-behat/grid.feature:40
    tests-behat/grid.feature:52
    tests-behat/grid.feature:65
    tests-behat/grid.feature:80
    tests-behat/grid.feature:89
    tests-behat/js.feature:3
    tests-behat/js.feature:28
    tests-behat/lookup.feature:3
    tests-behat/lookup.feature:12

92 scenarios (81 passed, 11 failed)

for some reason failing mostly on Oracle, but in 1/10 cases also on MSSQL and in 1/20 in SQLite too

PHP_CLI_SERVER_WORKERS=4 for webserver does not help. The most decisive repro config is Oracle DB!

UPDATE: The reason seems to be Firefox updates document.readyState only after at least part of the new page has been loaded (and Oracle testing needed a little more time per request as connect to an Oracle DB takes a little more time)