Chromedriver frequently hangs when attempting to start a new session. #87
Comments
|
I can reproduce this issue. My first run was actually clean, but it hung on the 41st attempt during my second run. |
|
Here's a bit more detail about what I see happening on the system when the launch hangs. The last line from the selenium-server stdout just says that it's launching a new Chrome session: Inside of the container, I see that some Chrome processes are in fact running: I am a bit suspicious of the defunct proc there, but it doesn't seem to consistently appear along with the problem, so I'm not sure that it's related to the hang at all. For what it's worth, I don't see anything out of the ordinary in the Chrome debug log: (Those errors appear even when the session launch succeeds.) As a sort of control I've executed the same test (repeatedly launch/quit Chrome sessions) against an Ubuntu 14.04 VM that is set up just like the Docker images -- the same Chrome and chromedriver versions, using xvfb with the same screen geometry, etc... The session launches are successful there 100% of the time, as far as I can tell, so the problem does appear to be specific to running chromedriver within a container. |
|
Happens to me on an Ubuntu 14.04 VM (not in a container): Chrome process list: |
ghost
mentioned this issue
|
I have also experienced this and trying to discover the cause has not been very successful. If I switch our tests to run using firefox, I do not experience this issue. When using the selenium-node-chrome-debug, I can see the browser open and running tests. During a hang, however, the browser is only open on the task bar and not able to be interacted with. |
|
I am also having this issue. This will prevent me to use docker-selenium. I need it to be stable so I can have continuous integration and monitoring over it. And I can't use firefox since I am using some chromeOptions. |
|
Seeing this issue as well. |
|
Checking in to report I am also seeing this issue. Google search led me here... |
|
I am having these issues as well, I found that decreasing the memory available to docker reduced how frequently this happens. |
|
Seeing this as well |
chromedriver has a bug that causes it to hang and then everything fails: SeleniumHQ/docker-selenium#87 detect if phantomjs is installed and use it, since that's less painful for local dev workflows. also remove the js Dockerfile; just use 'node' since the JS tests are pretty straightforward
|
Any progress here? I had this issues on rare occasions in previous releases, but after updating it occures so often that I can't run my tests anymore, no matter how often I try. |
|
I am having similar issues that I haven't investigated closely yet. Is there any reason to believe that the container should be running an I think I'll try to build a Chrome WebDriver container that uses Yelp's |
|
This has definitely gotten worse in the last month or two. I can't run our tests at all now. |
|
Have you guys seen the instructions in https://github.com/SeleniumHQ/docker-selenium#running-the-images? |
|
Yes, I have /dev/shm bind mounted. |
|
My team has been seeing these issues as well and we're not using docker, just a regular grid setup based on Ubuntu 14.04, so perhaps this is not specifically related to docker? |
Is it with or without selenium grid? Probably simple node would work? |
|
@Vanuan |
|
So it's anywhere in this chain According to the log: the hub reaches the node, but the node is unable to reach chromedriver because this message isn't printed: |
|
@Vanuan Yes, and considering that when using a local ChromeDriver instance on that same machine it never hung during our tests, one could assume that the |
|
I've just reproduced it by directly connecting to the node (without a hub). |
|
@Vanuan Is that using Docker? What's the OS? Our test script is using the Python selenium bindings, yours as well? |
|
No, mine is using ruby bindings. Yes. It's a docker node. Here's a run command: And I reproduce the timeout by connecting directly to It happens less frequently though. Client, hub and node are all separate machines. |
|
This does look strange: chromedriver is running there though: And it's listening: |
|
Reproduced again. Port used by chromedriver: 1081 How can I check where the connection times out? What's the code is responsible for writing |
|
For me, this NEVER happens if I start chrome from inside an X11 session. No matter if it runs against Xvfb or not, or whether Xvfb has been started on the console. The problem appears to be X11 resource usage related to a live/real X11 user desktop session. I traced it down to DBUS. Setting DBUS_SESSION_BUS_ADDRESS=/some/nonsense in Jenkins fixed my testing there and chrome/chromium are starting up there again without any problems at all......... |
|
@jjYBdx4IL how exactly did you configure the |
|
You you guys try installing dbus-x11 and see if that helps? |
…//codereview.chromium.org/2861163002/ ) Reason for revert: Speculative revert -- the TSan bots have been reporting a data race when setting Envvars (in this case, appending to the python path to start a websocket server). The race appeared immediately after this patch landed, so it may be legitimate. Reverting this to see if it clears the failures up; if so, we'll probably just have to serialize the calls to setenv. Filed crbug.com/719633 for this as well. Original issue's description: > Linux: Disable DBus auto-launch > > This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading > friendly and causing random hangs when running chrome outside of Linux > desktop environments. > > Background: > ----------- > Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS > environment variable. This variable allows the dbus client library to > directly connect to the existing bus, which is started by the desktop > environment or systemd. > When this variable is missing, the dbus client library will fallback > to auto-launch mode [1], which causes 4 nested fork() + exec() calls. > Doing this has two problems: (i) slows down startup; (ii) can hang > the browser if the fork() happens while another thread is in a malloc() > (Chrome's tcmalloc has no at-fork handlers). > This situation (no env variable) is very common in test scenarios > (browsertests, chromedriver, etc). > > Change introduced by this CL: > ----------------------------- > This CL sets the bus address env variable to "disabled:" if not set. > This effectively shuts down the dbus auto-launch. If necessary, this > behavior can be restored by setting, before launching chrome, > DBUS_SESSION_BUS_ADDRESS="autolaunch:" . > This workaround will be necessary until libdbus and gspawn are fixed > to be multi-threading friendly [2,3] and that fix rolls into the > various distributions. > The change is introduced in the main embedder rather than in the > google-chrome wrapper, as several binaries can be affected by this, > for instance: > - browser tests (http://crbug.com/693668) > - chrome --headless > - webdriver/selenium which seem to directly invoke "chrome" > see SeleniumHQ/docker-selenium#87 > > [1] https://dbus.freedesktop.org/doc/dbus-launch.1.html > [2] https://bugs.freedesktop.org/show_bug.cgi?id=100843 > [3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699 > > BUG=715658,695643,713947 > TEST=strace -ff -o trace chrome; grep dbus-launch trace* > > Review-Url: https://codereview.chromium.org/2861163002 > Cr-Commit-Position: refs/heads/master@{#469987} > Committed: https://chromium.googlesource.com/chromium/src/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0 [email protected],[email protected],[email protected],[email protected],[email protected] # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG=715658,695643,713947 Review-Url: https://codereview.chromium.org/2869843003 Cr-Commit-Position: refs/heads/master@{#470059}
…//codereview.chromium.org/2869843003/ ) Reason for reland: Adding TSan suppression. The race is independent of this CL, see crbug.com/719633 Original issue's description: > Revert of Linux: Disable DBus auto-launch (patchset #1 id:1 of https://codereview.chromium.org/2861163002/ ) > > Reason for revert: > Speculative revert -- the TSan bots have been reporting a data race when setting Envvars (in this case, appending to the python path to start a websocket server). The race appeared immediately after this patch landed, so it may be legitimate. Reverting this to see if it clears the failures up; if so, we'll probably just have to serialize the calls to setenv. > > Filed crbug.com/719633 for this as well. > > Original issue's description: > > Linux: Disable DBus auto-launch > > > > This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading > > friendly and causing random hangs when running chrome outside of Linux > > desktop environments. > > > > Background: > > ----------- > > Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS > > environment variable. This variable allows the dbus client library to > > directly connect to the existing bus, which is started by the desktop > > environment or systemd. > > When this variable is missing, the dbus client library will fallback > > to auto-launch mode [1], which causes 4 nested fork() + exec() calls. > > Doing this has two problems: (i) slows down startup; (ii) can hang > > the browser if the fork() happens while another thread is in a malloc() > > (Chrome's tcmalloc has no at-fork handlers). > > This situation (no env variable) is very common in test scenarios > > (browsertests, chromedriver, etc). > > > > Change introduced by this CL: > > ----------------------------- > > This CL sets the bus address env variable to "disabled:" if not set. > > This effectively shuts down the dbus auto-launch. If necessary, this > > behavior can be restored by setting, before launching chrome, > > DBUS_SESSION_BUS_ADDRESS="autolaunch:" . > > This workaround will be necessary until libdbus and gspawn are fixed > > to be multi-threading friendly [2,3] and that fix rolls into the > > various distributions. > > The change is introduced in the main embedder rather than in the > > google-chrome wrapper, as several binaries can be affected by this, > > for instance: > > - browser tests (http://crbug.com/693668) > > - chrome --headless > > - webdriver/selenium which seem to directly invoke "chrome" > > see SeleniumHQ/docker-selenium#87 > > > > [1] https://dbus.freedesktop.org/doc/dbus-launch.1.html > > [2] https://bugs.freedesktop.org/show_bug.cgi?id=100843 > > [3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699 > > > > BUG=715658,695643,713947 > > TEST=strace -ff -o trace chrome; grep dbus-launch trace* > > > > Review-Url: https://codereview.chromium.org/2861163002 > > Cr-Commit-Position: refs/heads/master@{#469987} > > Committed: https://chromium.googlesource.com/chromium/src/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0 > Review-Url: https://codereview.chromium.org/2869843003 > Cr-Commit-Position: refs/heads/master@{#470059} > Committed: https://chromium.googlesource.com/chromium/src/+/1e78cb7863da28bb3411286cdbcc4fb4510ce173 BUG=715658,695643,713947,719633 [email protected],[email protected],[email protected] Review-Url: https://codereview.chromium.org/2865283002 Cr-Commit-Position: refs/heads/master@{#470301}
|
Also experiencing this issue in Windows 10 (no container) when using Beta Chrome release 60.0.3112.32, but only when running chrome in the new --headless mode (which chromedriver doesn't ostensibly support yet, admittedly, but for my very simple test case it seems fine when I do get chromedriver launched). Launching chromedriver from python with --headless and --disable-gpu nets me the intermittent error on around 2/30 tries , with or without --no-sandbox (which I've seen suggested a lot from googling). As an aside, adding --remote-debugging-port=9222 breaks it since chromedriver generates its own port to try and access devtools on, but that's on me for using chromedriver for headless chrome before it's supported, and not relevant to my use case nor probably this issue. With --headless and --disable-gpu, it simply silently fails to connect as others have observed: As I understand it that linked patch isn't relevant to windows users. Anyone have any ideas about preventing this in windows? |
|
This seems to be a duplicate of #89 which apparently has a fix. Though this seems to be related to docker I'd like to share my experience with a local Selenium 3.4.0 standalone server without docker. The symptoms were the same: Selenium was stuck when creating a new session. It turns out the issue is no longer happening using Chrome 59. See my MCVE with minimal client code, setup descriptions and observations: https://github.com/tholewebgods/selenium-new-session-freeze-mcve |
Don't hide the test failure if there is a problem capturing a screenshot. Avoid stalled headless chrome nodes (SeleniumHQ/docker-selenium#87).
* merge in new auth token and some qa users * update tests to use new users from config * get current tests to pass with new qa users * change dev to qa for thurloe url * Save screenshot on test failure. * udpate ctmpls * put / in front of chrome path * add pem ctmpl and fix template render stuff * documentation for automation * fix failure screenshot rendering * Added clean-up to registration test and migrated it to the new test users. Added passing of FireCloud-Id to Thurloe calls. Fixed a bug with xpath of checking for an element with text. * don't use ivy cache * add dsde-toolbox pull to runtests script * Added Thurloe service * changing local orch api * use different auth domain * get default auth domain from vault * add host name to test runner * fixed loading of particpants.txt for DataTabSpec * fail whole script if tests fail * Improved reliability of test for creating a billing project. Don't hide the test failure if there is a problem capturing a screenshot. Avoid stalled headless chrome nodes (SeleniumHQ/docker-selenium#87). * Re-throw error when logging. Be a little more quiet about clean-up failures. * Avoid instability with Google sign-in when the popup window automatically closes.
|
All, |
See SeleniumHQ/docker-selenium#87 for more details.
…//codereview.chromium.org/2869843003/ ) Reason for reland: Adding TSan suppression. The race is independent of this CL, see crbug.com/719633 Original issue's description: > Revert of Linux: Disable DBus auto-launch (patchset #1 id:1 of https://codereview.chromium.org/2861163002/ ) > > Reason for revert: > Speculative revert -- the TSan bots have been reporting a data race when setting Envvars (in this case, appending to the python path to start a websocket server). The race appeared immediately after this patch landed, so it may be legitimate. Reverting this to see if it clears the failures up; if so, we'll probably just have to serialize the calls to setenv. > > Filed crbug.com/719633 for this as well. > > Original issue's description: > > Linux: Disable DBus auto-launch > > > > This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading > > friendly and causing random hangs when running chrome outside of Linux > > desktop environments. > > > > Background: > > ----------- > > Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS > > environment variable. This variable allows the dbus client library to > > directly connect to the existing bus, which is started by the desktop > > environment or systemd. > > When this variable is missing, the dbus client library will fallback > > to auto-launch mode [1], which causes 4 nested fork() + exec() calls. > > Doing this has two problems: (i) slows down startup; (ii) can hang > > the browser if the fork() happens while another thread is in a malloc() > > (Chrome's tcmalloc has no at-fork handlers). > > This situation (no env variable) is very common in test scenarios > > (browsertests, chromedriver, etc). > > > > Change introduced by this CL: > > ----------------------------- > > This CL sets the bus address env variable to "disabled:" if not set. > > This effectively shuts down the dbus auto-launch. If necessary, this > > behavior can be restored by setting, before launching chrome, > > DBUS_SESSION_BUS_ADDRESS="autolaunch:" . > > This workaround will be necessary until libdbus and gspawn are fixed > > to be multi-threading friendly [2,3] and that fix rolls into the > > various distributions. > > The change is introduced in the main embedder rather than in the > > google-chrome wrapper, as several binaries can be affected by this, > > for instance: > > - browser tests (http://crbug.com/693668) > > - chrome --headless > > - webdriver/selenium which seem to directly invoke "chrome" > > see SeleniumHQ/docker-selenium#87 > > > > [1] https://dbus.freedesktop.org/doc/dbus-launch.1.html > > [2] https://bugs.freedesktop.org/show_bug.cgi?id=100843 > > [3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699 > > > > BUG=715658,695643,713947 > > TEST=strace -ff -o trace chrome; grep dbus-launch trace* > > > > Review-Url: https://codereview.chromium.org/2861163002 > > Cr-Commit-Position: refs/heads/master@{#469987} > > Committed: https://chromium.googlesource.com/chromium/src/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0 > Review-Url: https://codereview.chromium.org/2869843003 > Cr-Commit-Position: refs/heads/master@{#470059} > Committed: https://chromium.googlesource.com/chromium/src/+/1e78cb7863da28bb3411286cdbcc4fb4510ce173 BUG=715658,695643,713947,719633 [email protected],[email protected],[email protected] Review-Url: https://codereview.chromium.org/2865283002 Cr-Original-Commit-Position: refs/heads/master@{#470301} Cr-Mirrored-From: https://chromium.googlesource.com/chromium/src Cr-Mirrored-Commit: 2fc330d0b93d4bfd7bd04b9fdd3102e529901f91
|
Adding DBUS_SESSION_BUS_ADDRESS=/dev/null as an environment variable in docker-compose totally fixed the problem! Now, I feel that it is running faster than it was! |
chromedriver has a bug that causes it to hang and then everything fails: SeleniumHQ/docker-selenium#87 detect if phantomjs is installed and use it, since that's less painful for local dev workflows. also remove the js Dockerfile; just use 'node' since the JS tests are pretty straightforward
|
Now my question is, how can I set this variable in circleCI :( |
|
@hutber Go to your project settings in Circle CI, click the link on the left for "Environment Variables", then add the key/value pair. |
|
I don't believe this issue is strictly related to Docker and I am getting different results based on the browser driver I use. Invoking NUnit console runner from AWS Run-Command (send-command) on a remote EC2. Selenium will fail to navigate to a url for the first one or two tests (inconsistently). My workaround: I made a separate TestFixture with the order attribute of 1. Fixture contains two tests. The test: driver.navigate().gotourl(app) and then assert(pass). Then all remaining tests run fine. Sometimes geckodriver still fails. Not using Selenium Grid currently, but it is necessary to implement it. |
|
DBUS_SESSION_BUS_ADDRESS=/dev/null In Excel vba (Windows) how it works? |
|
I have a solution: I stopped using selenium and went a different way. |
|
I hope that the developers have read and are looking for a solution to solve the problem, present on both Windows and Linux at all. |
|
HI @PS1Online, I am not sure how this could apply to your context. This env var was necessary many releases ago for the docker-selenium images. You seem to be running in a complete different environment (Excel VBA). Perhaps the most simple way is for you to join https://seleniumhq.herokuapp.com, there are lots of people there that can potentially help you. |
Recently we saw the smokey tests were hanging on production (AWS). On
investigating the issue, we found that chrome was failing to respond to
Chromedriver commands. The interaction between chromedriver and Chrome
is done in the context of a session [1]; we were able to get the active
session ID by doing an 'strace' of the chrome/driver processes.
curl -d '{"url":"https://www.google.com"}' http://localhost:9515/session/27f4262ab044392b05138540055a8fd6/url
This provided some clarity on the reason for the smokey tests hanging,
and lead to the following issue, which suggests the issue is related to
'dbus': SeleniumHQ/docker-selenium#87. While
this seems to be part of the main Chromium distribution [3], it's not
clear if this has made it into Chrome itself.
This trials implementing the suggested fix for the smokey process.
References
==========
[1] https://www.pawangaria.com/post/automation/browser-automation-from-command-line/
[2] https://chromium.googlesource.com/chromium/src/+/2fc330d0b93d4bfd7bd04b9fdd3102e529901f91%5E%21/
[3] https://chromium.googlesource.com/chromium/src/+/refs/heads/master/services/service_manager/embedder/main.cc#274
and others
Using the standalone-chrome or node-chrome images, I often observe a timeout when attempting to launch a new Chrome session.
Steps to reproduce:
docker run -d --name chrome selenium/standalone-chrome:2.47.1docker build -t selenium/test:local ./Test./test-repeat.sh chromeNote:
In order to narrow the focus of the test to launching new sessions, you may want to temporarily modify
Test/smoke-test.jsto omit the part of the test that tries navigating to github.com after starting the session. After modifying the script, make sure to rebuild the selenium/test:local image to pick up the changes.Expected results:
All 50 sessions launch and quit successfully - the test passes.
Actual results:
One of the session launch commands will hang indefinitely.
Docker host: boot2docker v1.7.0 (Tiny Core Linux)
The text was updated successfully, but these errors were encountered: