EZEE-1710 change selenium version

[mnocon]

JIRA: EZEE-1710

Description

The Firefox version used together with Selenium Server 2.53.1 is no longer able to login to Studio admin panel, so there is a need for upgrade to keep Behat tests running.

My solution is to change the Selenium version to 3.4.0 and the driver to Chrome.
My reasoning is as follows:

• there have been huge changes to Firefox driver between 2.* and 3.* (switching to Marionette etc.). Studio tests rely heavily on drag&drops, which started working only recently (in Selenium 3.5.*, it failed in 3.4.* in my experiments). Related Selenium issue: JWP actions endpoints do not get translated to W3C by standalone SeleniumHQ/selenium#3693
• unfortunately Mink does not support Selenium 3.5.* yet - TraversableElement::findField method does not find elements (and there are many usages of this method in our code)
• With that in mind I've decided that for now it's best for us to switch the driver to Chrome - to keep environments identical, I suggest Selenium Server 3.4.0 (it's the latest official image available: https://hub.docker.com/r/selenium/standalone-chrome/tags/). It uses Chrome 59.0.3071.115, which has been released 26.06.2017.

I think it's best when the same driver is used cross the whole project (that's why this PR is to ezplatform and not ezplatform-ee). So far I've tested it locally and the tests look pretty stable to me.

Any suggestions are welcome.

I'm marking it as WIP until the Travis build passes.

[andrerom]

Sounds like a plan for now!

But please keep an eye out on Mink Selenium 3.5 support so that we can update to that later, end result should be that we run some tests with chrome (default) and some with firefox (by setting custom SELENIUM_IMAGE on job matrix). For wider browser testing we have Saucelabs as intended to be used used by UI repos.

[andrerom]

One failure on last job:

  @javascript @common
  Scenario: Verify the existence of Users page and it's content # vendor/ezsystems/platform-ui-bundle/Features/Users/users.feature:7
    When I click on the navigation zone "Admin Panel"           # EzSystems\PlatformUIBundle\Features\Context\Users::clickNavigationZone()
      WebDriver\Exception\UnknownError: unknown error: Element <p class="ez-zone-name">...</p> is not clickable at point (674, 34). Other element would receive the click: <p class="ez-user-profile-name" id="yui_3_18_1_1_1503911189342_1150">...</p>

Restarting just in case, but seems like a legit failure right?

[mnocon]

Yes, it's something I'm looking at right now. There is also something else: the job kept running even after the tests have finished (I had to cancel it manually, because it went for 15 minutes and looked like it would keep running). We'll see what happens this time.

Note: the failing tests passes on my local env, but I haven't run it in docker (simple local Selenium Server + Chrome), I'll keep digging.

[andrerom]

@mnocon I guess it is because of the docker-compose logs output which is triggered on failure, in this case it way to verbose for this and hence it keeps on generating output for along time.

[mnocon]

Ok, that's plausible (and also unwanted?). I imagine in case of test failure the build would take a lot of time. I'm not sure why the Chrome driver would output that much more than the previous one.

[andrerom]

I'm not sure why the Chrome driver would output that much more than the previous one.

never said that, I think it's this particular job that has long log. I added this log output on failures just some months back to help debug failures. Disregard this, it can be tuned or we can disabled debug or reduce verbosity level, it's not a blocker for this change, so rather focus on the failure instead :)

EDIT: Hopefully fixed the logs freezing travis here: b3c722e

[mnocon]

Solution suggested here: ezsystems/PlatformUIBundle#895

[mnocon]

Yesterday I've tried maximise the browser windows in tests with ChromeOptions (this would make ezsystems/PlatformUIBundle#895 unnecessary), but I've stumbled on another issue that I'll describe here.

All my tries were failing with error: WebDriver\Exception\UnknownError: unknown error: failed to change window state to maximized, current state is normal (Example: travis build), which seemed odd - just 2 days ago it worked fine.

What happened:
2 days ago Selenium pushed an update to the 3.4.0 standalone-chrome image, which changed Chrome version from 59.0.3071.115 to 60.0.3112.113. Headless Chrome 60 has a specific bug (only when running in a docker container) that throws this error.

I've changed the Docker image to standalone-chrome-debug (it includes an additional VNC server).
This allows us to bypass this bug and might have another advantage: it should be easier to develop/debug the tests locally using this image, and I think images used should be the same locally and on CI.

Right now there is another issue with stability: tests start and fail randomly with WebDriver\Exception\UnknownError: unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed
Example

It seems to be a Docker specific issue: elgalu/docker-selenium#20 and one comment suggest possible solutions. I'm a begginer in the Docker world, so a question for you @andrerom: do any of these seem reasonable/possible for us to use? I'll try them locally to see if it helps.

[andrerom]

Option 2 would be the cleanest, volumes is not an option always and then depends on host shm.
So shm_size: 1G on selenium service, in doc/docker/selenum.yml and comment before the line linking to the issue so we remember why we have that there.

[mnocon]

Thanks! Added shm_size and the build is green (twice in a row). I've squashed my commits.
Looks like part one (changing the driver for Platform) is done, any suggestions?

[vidarl]

Besides my comment on shm size, I am all good

But please note that these changes will have no effect on ezrobot on ci, as ezrobot atm uses https://github.com/ezsystems/ezpublish-docker

[vidarl]

Is 1g really needed?
Fortunately, 1g is then not reserved immediately, but there will be an issue on ci if 5 of such containers run concurrently and they all actually use 1g of shm each....

[vidarl]

64m is default value in docker... Maybe 128m would do?

[mnocon]

Good idea, I guess tomorrow I'll iterate possible values (let's say 128mb, 256mb, 512mb and 1gb) and we'll se at which point the tests become stable.