Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Headless Chrome slows tests by 10x #561

Closed
fatso83 opened this issue Jun 19, 2017 · 28 comments
Closed

Headless Chrome slows tests by 10x #561

fatso83 opened this issue Jun 19, 2017 · 28 comments

Comments

@fatso83
Copy link

fatso83 commented Jun 19, 2017

What are you trying to achieve?

I am trying to run my tests in headless Chrome using the --headless argument. I expect the tests to run in approximately the same amount of time as the usual runs, as running a clean webdriverio script runs faster headless than in a normal browser window.

What do you get instead?

The tests run about ten times as slow, using 44 seconds instead of 4 seconds.

$ time $(npm bin)/codeceptjs run --verbose > headless-output.txt

real	0m44.973s
user	0m0.992s
sys	0m0.068s

$ nvim codecept.json # removing --headless option

developer at developer-XPS-13-9360 in ~/dev/dx-nuflo/tests/acceptance (acceptance-testing)
$ time $(npm bin)/codeceptjs run --verbose > output.txt

real	0m4.879s
user	0m0.768s
sys	0m0.088s
verbose output from headless run
CodeceptJS v0.6.3
Using test root "/home/developer/dev/dx-nuflo/tests/acceptance"

Top level-routes --
   [1] Starting recording promises
   Emitted | suite.before ([object Object])
   [1] Queued | hook WebDriverIO._beforeSuite()
 a page to add a new encounter should be rendered
 > [Session] Starting singleton browser session
   [2] Starting recording promises
   Emitted | test.before
   [2] Queued | hook WebDriverIO._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I am on page "/encounter/new")
   [2] Queued | amOnPage: "/encounter/new"
   Emitted | step.after (I am on page "/encounter/new")
   [2] Queued | return step result
   Emitted | step.before (I see "LOGG INN")
   [2] Queued | see: "LOGG INN"
   Emitted | step.after (I see "LOGG INN")
   [2] Queued | return step result
   [2] Queued | fire test.passed
   [2] Queued | finish test
   Emitted | step.start (I am on page "/encounter/new")
 • I am on page "/encounter/new"
   Emitted | step.start (I see "LOGG INN")
 • I see "LOGG INN"
   Emitted | test.passed ([object Object])
 ✓ OK in 22317ms

   Emitted | test.after
   [2] Queued | hook WebDriverIO._after()
 > [Session] cleaning cookies and localStorage
 fooScen
   [3] Starting recording promises
   Emitted | test.before
   [3] Queued | hook WebDriverIO._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I am on page "/encounter/new")
   [3] Queued | amOnPage: "/encounter/new"
   Emitted | step.after (I am on page "/encounter/new")
   [3] Queued | return step result
   Emitted | step.before (I see "FOOLOGG INN")
   [3] Queued | see: "FOOLOGG INN"
   Emitted | step.after (I see "FOOLOGG INN")
   [3] Queued | return step result
   [3] Queued | fire test.passed
   [3] Queued | finish test
   Emitted | step.start (I am on page "/encounter/new")
 • I am on page "/encounter/new"
   Emitted | step.start (I see "FOOLOGG INN")
 • I see "FOOLOGG INN"
   [3] Error | Error
   [3] Starting <teardown> session
   Emitted | test.failed ([object Object])
   [3] <teardown> Queued | hook WebDriverIO._failed()
   [3] <teardown> Queued | () => done(err)
   [3] <teardown> Stopping recording promises
 > Screenshot has been saved to /home/developer/dev/dx-nuflo/tests/acceptance/output/fooScen.failed.png
 ✖ FAILED in 11599ms

   Emitted | test.after
   [3] <teardown> Queued | hook WebDriverIO._after()
 > [Session] cleaning cookies and localStorage
   [4] Starting recording promises
   Emitted | suite.after ([object Object])
   [4] Queued | hook WebDriverIO._afterSuite()

-- FAILURES:

  1) Top level-routes: fooScen:

      expected web page to include "FOOLOGG INN"
      + expected - actual

      -Innlogging
      -Brukernavn
      -Passord
      -LOGG INN
      +FOOLOGG INN
      
  
  Scenario Steps:
  
  - I.see("FOOLOGG INN") at Test.Scenario (top-level-routes_test.js:18:5)
  - I.amOnPage("/encounter/new") at Test.Scenario (top-level-routes_test.js:14:5)
  
  Error
      at Object.<anonymous> (/home/developer/dev/dx-nuflo/node_modules/codeceptjs/lib/helper/WebDriverIO.js:1129:51)
      at Object.exec (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/helpers/safeExecute.js:28:24)
      at Object.resolve (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:189:29)
      at /home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:538:32
      at _fulfilled (/home/developer/dev/dx-nuflo/node_modules/q/q.js:854:54)
      at self.promiseDispatch.done (/home/developer/dev/dx-nuflo/node_modules/q/q.js:883:30)


  FAIL  | 1 passed, 1 failed   // 34s
   Emitted | global.result ([object Object])


real	0m44.973s
user	0m0.992s
sys	0m0.068s
verbose output from normal (non-headless) run
CodeceptJS v0.6.3
Using test root "/home/developer/dev/dx-nuflo/tests/acceptance"

Top level-routes --
   [1] Starting recording promises
   Emitted | suite.before ([object Object])
   [1] Queued | hook WebDriverIO._beforeSuite()
 a page to add a new encounter should be rendered
 > [Session] Starting singleton browser session
   [2] Starting recording promises
   Emitted | test.before
   [2] Queued | hook WebDriverIO._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I am on page "/encounter/new")
   [2] Queued | amOnPage: "/encounter/new"
   Emitted | step.after (I am on page "/encounter/new")
   [2] Queued | return step result
   Emitted | step.before (I see "LOGG INN")
   [2] Queued | see: "LOGG INN"
   Emitted | step.after (I see "LOGG INN")
   [2] Queued | return step result
   [2] Queued | fire test.passed
   [2] Queued | finish test
   Emitted | step.start (I am on page "/encounter/new")
 • I am on page "/encounter/new"
   Emitted | step.start (I see "LOGG INN")
 • I see "LOGG INN"
   Emitted | test.passed ([object Object])
 ✓ OK in 2748ms

   Emitted | test.after
   [2] Queued | hook WebDriverIO._after()
 > [Session] cleaning cookies and localStorage
 fooScen
   [3] Starting recording promises
   Emitted | test.before
   [3] Queued | hook WebDriverIO._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I am on page "/encounter/new")
   [3] Queued | amOnPage: "/encounter/new"
   Emitted | step.after (I am on page "/encounter/new")
   [3] Queued | return step result
   Emitted | step.before (I see "FOOLOGG INN")
   [3] Queued | see: "FOOLOGG INN"
   Emitted | step.after (I see "FOOLOGG INN")
   [3] Queued | return step result
   [3] Queued | fire test.passed
   [3] Queued | finish test
   Emitted | step.start (I am on page "/encounter/new")
 • I am on page "/encounter/new"
   Emitted | step.start (I see "FOOLOGG INN")
 • I see "FOOLOGG INN"
   [3] Error | Error
   [3] Starting <teardown> session
   Emitted | test.failed ([object Object])
   [3] <teardown> Queued | hook WebDriverIO._failed()
   [3] <teardown> Queued | () => done(err)
   [3] <teardown> Stopping recording promises
 > Screenshot has been saved to /home/developer/dev/dx-nuflo/tests/acceptance/output/fooScen.failed.png
 ✖ FAILED in 1299ms

   Emitted | test.after
   [3] <teardown> Queued | hook WebDriverIO._after()
 > [Session] cleaning cookies and localStorage
   [4] Starting recording promises
   Emitted | suite.after ([object Object])
   [4] Queued | hook WebDriverIO._afterSuite()

-- FAILURES:

  1) Top level-routes: fooScen:

      expected web page to include "FOOLOGG INN"
      + expected - actual

      -Innlogging
      -Brukernavn
      -Passord
      -LOGG INN
      +FOOLOGG INN
      
  
  Scenario Steps:
  
  - I.see("FOOLOGG INN") at Test.Scenario (top-level-routes_test.js:18:5)
  - I.amOnPage("/encounter/new") at Test.Scenario (top-level-routes_test.js:14:5)
  
  Error
      at Object.<anonymous> (/home/developer/dev/dx-nuflo/node_modules/codeceptjs/lib/helper/WebDriverIO.js:1129:51)
      at Object.exec (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/helpers/safeExecute.js:28:24)
      at Object.resolve (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:189:29)
      at /home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:538:32
      at _fulfilled (/home/developer/dev/dx-nuflo/node_modules/q/q.js:854:54)
      at self.promiseDispatch.done (/home/developer/dev/dx-nuflo/node_modules/q/q.js:883:30)


  FAIL  | 1 passed, 1 failed   // 4s
   Emitted | global.result ([object Object])


real	0m4.879s
user	0m0.768s
sys	0m0.088s

Test source code

Feature('Top level-routes');

Scenario('a page to add a new encounter should be rendered', (I) => {
  I.amOnPage('/encounter/new');
  I.see('LOGG INN');
});

// just to see how a failing test looks
Scenario('fooScen', (I) => {
  I.amOnPage('/encounter/new');
  I.see('FOOLOGG INN');
});

Details

  • CodeceptJS version: 0.6.3
  • NodeJS Version: 8.1.2
  • Operating System: Ubuntu 16.04 LTS
  • WebDriverIO: 4.8.0
  • Configuration file:
This is my configuration:
{
  "tests": "./*_test.js",
  "timeout": 5000,
  "output": "./output",
  "helpers": {
    "WebDriverIO" : {
      "browser": "chrome",
      "desiredCapabilities": {
          "browserName": "chrome",
          "chromeOptions": {
              "args": [
                  "headless",
                  "disable-gpu"
              ]
          }
      },
      "windowSize": "320x568",
      "restart": false,
      "timeouts": {
        "script": 10000,
        "page load": 3000,
        "implicit" : 5000
      },
      "url": "http://localhost:23450"
    }
  },
  "include": {
    "I": "./steps_file.js"
  },
  "bootstrap": false,
  "mocha": {},
  "name": "acceptance tests",
  "windowSize": "320x568"
}
@fatso83 fatso83 changed the title Passing headless option to Chrome wdio slows tests by 10x Headless Chrome slows tests by 10x Jun 19, 2017
@laurentpayot
Copy link

@fatso83 Did you try with node < 8 such as 7.10.0?

@fatso83
Copy link
Author

fatso83 commented Jun 27, 2017

Tried with Node 7.4.0. Still a lot slower.

A test run took 26 seconds with the non-headless browser and 1m5s with --headless. Not 10x slow, but at least 4x slower (probably due to more page loading in these tests, obscuring performance).

@DavertMik
Copy link
Contributor

Thanks, I will investigate this issue

@DavertMik
Copy link
Contributor

I reproduced the issue but I still have no clue what's wrong.
Just changing Chrome from headless to window makes test run much faster. Can this be caused because by disable-gpu flag? I don't know. Probably headless Chrome is not so stable enough

@Setitch
Copy link

Setitch commented Oct 11, 2017

I see the same slowness with and without --dissable-gpu so thats not the case. When i connect to the headless i see SO MANY ssl warning and so on - i think its just due to that.

@DavertMik
Copy link
Contributor

Thanks. Maybe this could be the case

@mogryo
Copy link

mogryo commented Dec 7, 2017

Hey. Is there any updates on this issue? We've ran into similar problem on some of our machines. Our tests just timeout because of the enormous time it takes for headless chrome to do what we ask. And if launched not in headless it runs smooth.

We ran into this problem on windows 7. With latest chromedriver 2.33 and latest chrome. But it's very random. On other machines with the same configuration it runs perfectly in headless (well, not exactly perfectly, but only ~1.5x times slower). Perhaps you have an idea what should we check on these machines? to pin point the mystical difference between them.

@emesterhazy
Copy link

Also experiencing extremely slow load times when using Chrome in headless mode in Windows 7 with the latest version of Chromedriver, Chrome, and Selenium. Interestingly I'm not having any issues on Windows 10.

@JamesLMilner
Copy link

I'm coming at this from a different place, but interestingly there's an open issue on ChromeDriver which sounds suspiciously related: https://bugs.chromium.org/p/chromedriver/issues/detail?id=2054&q=jenkins&colspec=ID%20Status%20Pri%20Owner%20Summary

@laskeym
Copy link

laskeym commented Feb 23, 2018

I'm in the same boat as @emesterhazy . I currently have Chrome Version 63.0.3239.132 and ChromeDriver 2.35 on Windows 7 and headless hangs up on retrieving the sites. When I come home from work, I'll mess around with this on my Debian 9 and Windows 10 machine and report the results.

Simple test of retrieving the Facebook home page and grabbing login forms:
Windows 7:

  • Headless = 111.667s
  • Non-Headless = 3.018s

Also, I have Google Chrome Canary downloaded, but I haven't attempted to configure the driver to use it. Not sure if using Canary over standard Chrome would make a difference.

@oligee80
Copy link

Try these two ChromeOptions:

options.addArguments("--proxy-server='direct://'");
options.addArguments("--proxy-bypass-list=*");

I dunno in wich forum it was but this did the trick for me

@hYpac0w
Copy link

hYpac0w commented Mar 22, 2018

@oligee80 THANK YOU SO MUCH!
These options finally got headless chrome working for me :)

@Resh5
Copy link

Resh5 commented Apr 9, 2018

@oligee80 Thanks much!!! It worked for me as well.

@yashwanth2804
Copy link

yashwanth2804 commented Apr 17, 2018

@oligee80 Really thank you buddy, I have been breaking my head for that issue, finally got worked out

@rahul-konduru
Copy link

I am trying this on win 10. After adding the arguments that @oligee80 has mentioned, I do not see any difference. Is there something else that I should be doing?

@Resh5
Copy link

Resh5 commented Apr 25, 2018

@krrahulmanikanta Is your Chrome version compatible?

@rahul-konduru
Copy link

@Resh5 Yes, I am using chromedriver v2.38 and my chrome is of v66

@Resh5
Copy link

Resh5 commented Apr 25, 2018

@krrahulmanikanta It workes fine for me . But I am in Windows7
System.setProperty("path to driver);
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--headless");
chromeOptions.addArguments("window-sized1200,600");
chromeOptions.addArguments("--proxy-server='direct://'");
chromeOptions.addArguments("--proxy-bypass-list=*");
ChromeDriver driver = new ChromeDriver(chromeOptions);

@Robula
Copy link

Robula commented May 17, 2018

I'm seeing a similar issue using Puppeteer and Chromium (no WebDriver), the proxy arguments do not seem to make a difference in my case. This may be an issue related to Chromium.

macOS: 10.13.4
Node: 8.10.0
headless: true results in 7747.209ms
headless: false results in 4775.771ms

timorantalaiho added a commit to Opetushallitus/omatsivut that referenced this issue Sep 17, 2018
And decrease single test timeout, optimistically.

See codeceptjs/CodeceptJS#561 (comment)
timorantalaiho added a commit to Opetushallitus/omatsivut that referenced this issue Sep 18, 2018
And decrease single test timeout, optimistically.

See codeceptjs/CodeceptJS#561 (comment)
@oppahansi
Copy link

@oligee80 Thanks

@pardovot
Copy link

Try these two ChromeOptions:

options.addArguments("--proxy-server='direct://'");
options.addArguments("--proxy-bypass-list=*");

I dunno in wich forum it was but this did the trick for me

Don't know if you'll actually see this, but it seems like the following solved the issue for me: options.addArguments("--proxy-server=");
The question is, why? I'm running the simplest code ever which is something like:
new WebDriver, driver.get("https://www.google.com").
This takes around 2 minutes, and solved be that argument, but I really can't understand what exactly it does..
Thanks! :)

@GeorgeDubuque
Copy link

I am having a similar issue on this one still. I tried with the following arguments

options.add_argument("--proxy-server='direct://'")
options.add_argument("--proxy-bypass-list=*")
options.add_argument("--proxy-server=")

I am still not seeing any speed up. 40 seconds to run in headless and 4 seconds to run in windowed mode. I am running ChromeDriver 75.0.3770.140.

Please let me know if anyone solves this one! Could really help me out. All I am doing is calling driver.get("https://www.google.com")

@GeorgeDubuque
Copy link

I am having a similar issue on this one still. I tried with the following arguments

options.add_argument("--proxy-server='direct://'")
options.add_argument("--proxy-bypass-list=*")
options.add_argument("--proxy-server=")

I am still not seeing any speed up. 40 seconds to run in headless and 4 seconds to run in windowed mode. I am running ChromeDriver 75.0.3770.140.

Please let me know if anyone solves this one! Could really help me out. All I am doing is calling driver.get("https://www.google.com")

Ok so after looking into it a bit more it seems that avoiding using headless chrome is the solution for me. Instead I use the pyvirtualdisplay library and set its visibilty to false. This sped up my performance 10x. Here is the code:

from selenium import webdriver
from pyvirtualdisplay import Display
import time

start = time.time()
display = Display(visible=0, size=(800, 600))
display.start()
driver = webdriver.Chrome(
        "/home/george/Desktop/chromedriver")
driver.get("https://www.google.com")
driver.quit()
display.stop()
end = time.time()
print(end-start)

with output:

5.253071069717407

instead of:

42.1179986000061

@HansHagberg
Copy link

HansHagberg commented Aug 14, 2019

Was searching for solution to a similar problem so I'm adding a note for reference:

Running a test scenario with headless Chromium and 100 instances. Machine is an AMD 2950X with 16 phys cores @4 GHz, 128 GB RAM and an Nvidia GTX 1660 GPU on Windows 10. This machine turns into a sluggish mess when running this with GPU-disabled in chromium startup. With GPU enabled, response is acceptable. With 200 clients, machine goes very sluggish (useless) again even with GPU.

With 200 instances, GPU utilization is very low (<10%) and CPU is ~35%
Task manager says 17300 threads and 420000 handles.

It seems to be some windows resource causing degradation because when doing a remote login session with another account while the load is running, the new session has no problems with response. So it looks like Windows and not the hardware is the bottleneck. Contemplating trying with Windows server to see if that helps.

Update: Windows server 2019 made no difference

Running headless instances in tabs instead seems to stress the system a lot less. We can now run 4 - 5 times as many instances with less problems compared to 200 instances before.

@wilsonweb
Copy link

I saw some people get this to work and others do not have any success with it:

chromeOptions.addArguments("--proxy-server='direct://'");
chromeOptions.addArguments("--proxy-bypass-list=*");

I did not have success at first ether, then I looked at my code and realized that I have called setChromeOptions() more than one time and wondered if options are getting over-written. I moved things around to make only one call to setChromeOptions() and it worked.

var $driver = require("selenium-webdriver");
let chrome = require("selenium-webdriver/chrome");

const chromeOptions = new chrome.Options();//.headless();
chromeOptions.addArguments("--headless"); 
chromeOptions.addArguments("--proxy-server='direct://'");
chromeOptions.addArguments("--proxy-bypass-list=*");
chromeOptions.addArguments("window-size=1024,1324");
chromeOptions.addArguments("--blink-settings=imagesEnabled=false"); 

var $browser = new $driver.Builder()
	.forBrowser("chrome")
	.setChromeOptions(chromeOptions) 
	.withCapabilities(capabilities)
	//.forBrowser('firefox')
	.build();

You can ignore the lines dealing with hiding images, windows size etc. if you choose.

@zjhgx
Copy link

zjhgx commented Nov 24, 2020

in my case, add the following argument solve my problem
chrome_options.add_argument('blink-settings=imagesEnabled=false')

@mikemangas
Copy link

mikemangas commented Jan 24, 2022

options.addArguments("--proxy-server='direct://'");
options.addArguments("--proxy-bypass-list=*");

Where do you insert this?

@oligee80

@mys1erious
Copy link

Only thing that worked for me is adding UserAgent to the options:
options.add_argument("user-agent=Chrome/110.0.5481.77")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests