Build failing on master

[PragTob]

Recent master build: https://travis-ci.org/keathley/wallaby/jobs/230747171

Also seems to be failing on the PRs (of course) so we should make it a prio to get this fixed.

It might be a flaky thing, but from what I can see we can see the following:

Elixir 1.4.2 with Erlang 19 on Selenium 3 (at least on master also Elixir 1.3 with Selenium 3 fails) seems to fail on both builds with quite weird errors... like...

test all/2 returns all of the elements matching the query (Wallaby.Integration.Browser.AllTest)
     integration_test/cases/browser/all_test.exs:11
     ** (FunctionClauseError) no function clause matching in Wallaby.Experimental.Selenium.WebdriverClient.cast_as_element/2
     stacktrace:
       (wallaby) lib/wallaby/experimental/selenium/webdriver_client.ex:383: Wallaby.Experimental.Selenium.WebdriverClient.cast_as_element(%Wallaby.Session{driver: Wallaby.Experimental.Selenium, id: nil, screenshots: [], server: :none, session_url: "http://localhost:4444/wd/hub/session/", url: "http://localhost:4444/wd/hub/session/"}, {"additionalInformation", "\nDriver info: driver.version: unknown"})
       (elixir) lib/enum.ex:1233: anonymous fn/3 in Enum.map/2
       (stdlib) lists.erl:1263: :lists.foldl/3
       (elixir) lib/enum.ex:1772: Enum.map/2
       (wallaby) lib/wallaby/experimental/selenium/webdriver_client.ex:34: Wallaby.Experimental.Selenium.WebdriverClient.find_elements/2
       (wallaby) lib/wallaby/browser.ex:995: anonymous fn/3 in Wallaby.Browser.execute_query/2
       (wallaby) lib/wallaby/browser.ex:142: Wallaby.Browser.retry/2
       (wallaby) lib/wallaby/browser.ex:549: Wallaby.Browser.find/2
       integration_test/cases/browser/all_test.exs:13: (test)

seems to be failing on master though since the merge of #217 but maybe that's just the first time the flakyness showed up

[aaronrenner]

@keathley It looks like there was an issue downloading Firefox from Mozilla. Here are the relevant logs:

Resolving download.cdn.mozilla.net (download.cdn.mozilla.net)... failed: Temporary failure in name resolution.
wget: unable to resolve host address `download.cdn.mozilla.net'
tar: This does not look like a tar archive
bzip2: Compressed file ends unexpectedly;
	perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
	Input file = (stdin), output file = (stdout)
It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.
You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.
tar: Child returned status 2
tar: Error is not recoverable: exiting now

I just tried it on my own travis ci environment and the test suite is green: https://travis-ci.org/aaronrenner/wallaby/builds/230883580.

As a side note, we should probably build a much friendlier error api. 😄

[PragTob]

haha I see that :) Maybe we can build in a retry there or something?

[PragTob]

Also, I just restarted a bunch of hoping that it was flaky behaviour (we had some weird internet problems at work today)

[aaronrenner]

Sorry I mentioned Chris not you, @PragTob. For some reason I thought he was the one that opened the issue. Should we see if this becomes a regular issue before we build retry in to the start_server.sh script? This is the first I've seen of this particular issue.

[PragTob]

sure thing, I'm a friend of being pragmatic but also of resilient tests. We'll see how it goes and keep it open

[keathley]

I've noticed this a bit in the past. So it seems like the root is that travis can't download firefox. Is that an issue with the firefox cdn potentially? Or travis? Is there a way to provide a custom build image to travis?

[keathley]

Also can we just use the firefox that exists on travis? https://docs.travis-ci.com/user/ci-environment/#Firefox

Or do we need a specific version?

[aaronrenner]

The reason I didn't use the travis provided Firefox is because we need a different version for selenium 2 and selenium 3. I think the max version supported by Selenium 2 is 46.0 and Selenium 3 needs 47.0 or later. I'm not sure how to install different version of firefox in per environment using .travis.yml

[keathley]

Is it always selenium 3 that's failing? Is there anyway to create a vm or docker container that already has everything we need and use that as the base image for Travis? I know circle CI has support for that sorta thing (not sure if it's available for open source or not)

[PragTob]

for the cases I observed it was always Selenium 3 that was failing...

Using docker images in travis is totally possible it's not my strong suite though :)

[keathley]

If we keep having issues with flaky tests we can use that as a backup strategy. We'd probably get a pretty big speed boost as well since it wouldn't have to install anything.

[PragTob]

Recent builds were all successful, closing for now.