HTTPS proxying does not work in headless Chrome

[adamniedzielski]

Thanks for the amazing gem, I didn't even suspect that mocking JavaScript requests can be that simple 🎉 🥇.

I'm opening this issue for other users that may have the same problem as I do. I don't think that there's anything that Puffing Billy team can do here to help right now.

I'm trying to use Puffing Billy together with headless Chrome. My configuration is following:

Capybara.register_driver :headless_chrome do |app|
  Capybara::Selenium::Driver.new(
    app,
    browser: :chrome,
    options: Selenium::WebDriver::Chrome::Options.new(
      args: [
        "headless",
        "disable-gpu",
        "proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}"
      ]
    )
  )
end

Proxying HTTP requests works perfectly fine. Proxying HTTPS requests throws a following error in Chrome console:

https://maps.googleapis.com/maps/api/js?key=AIzaSyBukfGfN2Ayd_LgVqjEGEtJneT-D-r4Zv4 - Failed to load resource: net::ERR_INSECURE_RESPONSE

When I remove the headless flag from my Capybara driver configuration, proxying HTTPS requests works fine.

Apparently this is the expected behaviour now for headless Chrome. It just doesn't support ignoring these certificate errors. The most relevant issue that I found about it is https://bugs.chromium.org/p/chromium/issues/detail?id=721739.

[ndbroadbent]

Hi @adamniedzielski, I've been trying to get my tests running on GitLab CI with chromedriver and xvfb, but would love to switch to headless Chrome. Are you also running your tests on a CI service? Would you be able to share your setup script to get everything installed? Or is it as simple as installing the latest Google Chrome, then using the config that you provided?

[adamniedzielski]

Hey @ndbroadbent! We're running our tests on Jenkins with the following setup:

• chromedriver 2.31
• Chrome 61.0
• selenium-webdriver gem 3.5.1

Yes, it should be as simple as installing the dependencies and using the config that I provided above 😄.

[ndbroadbent]

Awesome, thanks! Haha but now I'm running into the same SSL problem. In my case it's for Stripe requests. If it's not possible to ignore SSL errors, maybe it's possible to install the self-signed certificate in Chrome?

I guess for now I'll just use xvfb and the :selenium_chrome_billy driver. Although I've been struggling to figure out some Net::Timeout errors on GitLab that aren't happening locally.

[dentarg]

@adamniedzielski have you tried Chrome 62 and --allow-insecure-localhost? https://bugs.chromium.org/p/chromium/issues/detail?id=721739#c28

[adamniedzielski]

Thanks for the ping @dentarg! I've just updated Chrome to version 62 and checked. Unfortunately, adding --allow-insecure-localhost option doesn't change the situation in my case. I'm still getting:

https://maps.googleapis.com/maps/api/js?key=AIzaSyBukfGfN2Ayd_LgVqjEGEtJneT-D-r4Zv4 - Failed to load resource: net::ERR_INSECURE_RESPONSE

[Jack12816]

@adamniedzielski, @ndbroadbent - I had the same issues (Chrome Headless, SSL) and provided a PR for correct+valid SSL handling on puffing billy which was published yesterday with the 0.11.0 release.

Docs: https://github.com/oesmith/puffing-billy#ssl-usage
Changelog: https://github.com/oesmith/puffing-billy/blob/master/CHANGELOG.md#v0110-2017-11-09

Give it a try! :)

[Jack12816]

If you have to deal with WebSockets its worse right now. Chrome Headless (by 63 beta) does not support PAC handling. See the issue for more details. But you can work around WebSocket domains with 62 stable. (--proxy-bypass-list)

[adamniedzielski]

@Jack12816 thanks for your effort and the work on the PR! 💚

I timeboxed 20 minutes to try to get it working according to the docs, but unfortunately I didn't succeed. I believe that:

ENV['HOME'] = "#{Dir.tmpdir}/chrome-home-#{Time.now.to_i}"

# Clear and recreate the Chrome home directory.
FileUtils.rm_rf(ENV['HOME'])
FileUtils.mkdir_p(ENV['HOME'])

# Setup a new pki certificate database for Chrome
system <<~SCRIPT
cd "#{ENV['HOME']}"
curl -s -k -o "cacert-root.crt" "http://www.cacert.org/certs/root.crt"
curl -s -k -o "cacert-class3.crt" "http://www.cacert.org/certs/class3.crt"
echo > .password
mkdir -p .pki/nssdb
CERT_DIR=sql:$HOME/.pki/nssdb
certutil -N -d .pki/nssdb -f .password
certutil -d ${CERT_DIR}  -A -t TC \
  -n "CAcert.org" -i cacert-root.crt
certutil -d ${CERT_DIR} -A -t TC \
  -n "CAcert.org Class 3" -i cacert-class3.crt
certutil -d sql:$HOME/.pki/nssdb -A \
  -n puffing-billy -t "CT,C,C" -i #{Billy.certificate_authority.cert_file}
SCRIPT

is correctly creating the files, at least certutil doesn't throw any errors, and the files are there. I think that somehow not picking up the files. Maybe it has something to do with differences between Chrome configuration on Linux and Mac OS as I'm running this on a Mac.

[Jack12816]

Yeah, on macOS this would be different also because they handle certificates system-wide. I'm not sure how to run this on macOS yet, but for my colleagues, I have to figure this out as well. I will give some updates here for this.

[AlanFoster]

@Jack12816 Thanks for the SSL work, it definitely seems promising! 🎉
For visibility I tried getting this to work on a mac, and unfortunately had no luck! :(

I first tried to install the root CA cert directly on my mac during one of the tests via:

> Billy.config.certs_path
=> "/var/folders/y5/qm48psks18j_rbwg6jtmnx7r0000gp/T/puffing-billy/certs"

I then directly installed and trusted ca.crt cert on the keychain:

I then tried to see if everything would work as expected by configuring chrome directly to use puffing billy's proxy. I got the current port via:

> Billy.proxy.port
=> 58414

However I still got SSL errors on Chrome directly:

The root CA certificate does indeed seem different than the root CA that i've installed on my mac:

However I'm not quite sure why, as I think there should be one root CA used by puffing billy 🤔

Unfortunately I don't have much experience with SSL certs etc, but I'd love to help get this set-up and working, so let me know if there's anything I can do to help :)

[Jack12816]

@AlanFoster Thanks for trying, but each time your test suite run, a new puffing billy root CA is generated. So this won't work if you import it from your macOS settings dialog. I will look for a programmatic way to do this. Or to use a different certificate store for the new Chrome session.
But unfortunately I didn't found the time for this yet, so stay tuned. :)

[AlanFoster]

@Jack12816 haha, yeah - i was aware of that when i was investigating it! The initial goal was to just get it working for the currently paused rspec session, then I'd automate it and post a solution. In the end i had no luck though!

I'll stay tuned though, let me know if i can do anything to help 👍

[Jack12816]

I had time to implement a solution for macOS.

https://github.com/Jack12816/puffing-billy/blob/6dc27da3a473c04818f2d3eeeac854de666a94fc/README.md#google-chrome-headless-example

/cc @AlanFoster @adamniedzielski

[Aesthetikx]

Seemingly Chrome 65 with driver 2.35 will support the acceptInsecureCerts flag, https://bugs.chromium.org/p/chromium/issues/detail?id=721739#c95

[AlanFoster]

@Aesthetikx Have you had any luck using Chrome 65, chromedriver 2.35, and using both acceptInsecureCerts + Proxy configuration? It seems to hang for me.

I haven't had any luck with the following configuration:

Capybara.register_driver :selenium_headless_chrome_billy do |app|
  options = Selenium::WebDriver::Chrome::Options.new
  options.add_argument('--headless')
  options.add_argument('--disable-gpu')
  options.add_argument('--no-sandbox')
  options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")

  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome
  capabilities['acceptInsecureCerts'] = true

  Capybara::Selenium::Driver.new app,
                                 browser: :chrome,
                                 desired_capabilities: capabilities,
                                 options: options
end

[AlanFoster]

In the mean time, if you want headless firefox - that works pretty nice:

Capybara.register_driver :selenium_headless_firefox_billy do |app|
  options = ::Selenium::WebDriver::Firefox::Options.new
  options.add_argument('--headless')

  capabilities = Selenium::WebDriver::Remote::Capabilities.firefox(
    acceptInsecureCerts: true,
    proxy: {
      http: "#{Billy.proxy.host}:#{Billy.proxy.port}",
      ssl: "#{Billy.proxy.host}:#{Billy.proxy.port}"
    }
  )

  ::Capybara::Selenium::Driver.new(
    app,
    options: options,
    desired_capabilities: capabilities
  )
end

Note that unlike capybara webkit, firefox makes additional requests to see what version it is on etc, I have disabled this as part of my puffing billy set up.

Headless firefox appears to be pretty slow though, from 5 minutes on poltergeist to 20 minutes on headless firefox, and 18 minutes on normal firefox 🤔

cc @ronwsmith Let me know if we should add headless firefox support to puffing billy out of the box or not 👍

[ronwsmith]

@AlanFoster more default drivers, the merrier! Please feel free to add.

[rposborne]

Chrome 65 is now stable.

[lucasdavila]

@AlanFoster what is the problem you had when using chrome?

For me (with Chrome 65 and chromedriver 2.37) with the same configs as you, rspec is getting stuck.

The last thing logged on log/test.log is: puffing-billy: PROXY GET succeeded for 'http://localhost:54396/packs-test/c9cda8678897ac8fb981.js'

If I remove the acceptInsecureCerts option rspec do not get stuck anymore, but the specs break with the problem described on this issue, because it is in headless mode.

Do you guys have any idea about how to make the acceptInsecureCerts work with chrome driver? Thanks.

[urbanautomaton]

I have the same problem @lucasdavila describes (requests to proxied secure URLs hang), and am currently trying to diagnose the issue. I've set up a small reproduction here: https://github.com/urbanautomaton/headless_chrome_ssl_proxy

As far as I can tell, something is going wrong in the TLS handshake. For a proxied request to https://example.net, the following sequence occurs:

Non-headless:

• Chrome -> PB: CONNECT example.net:443 HTTP/1.1
• PB -> Chrome: HTTP/1.0 Connection established
• Chrome -> PB: GET / HTTP/1.1...
• PB -> Chrome: HTTP/1.1 200 OK...

Headless:

• Chrome -> PB: CONNECT example.net:443 HTTP/1.1
• PB -> Chrome: HTTP/1.0 Connection established
• Chrome -> PB: CONNECT example.net:443 HTTP/1.1
• ... (loops infinitely)

I've submitted this as a bug to the chromedriver project (since the behaviour differs between headless and non-headless Chrome). They've asked me to try to reproduce the issue using browsermob-proxy instead of puffing-billy.

Interestingly, browsermob-proxy successfully proxies the request to both headless and non-headless Chrome, so my latest project is to try and see exactly what's going on in the handshake, to see if I can spot what browsermob-proxy is doing differently.

[urbanautomaton]

I've now verified that headless chrome works with proxied secure URLs using both browsermob-proxy and mitmproxy. I've updated my repro script, which now allows you to run a minimal reproduction against the three proxies:

https://github.com/urbanautomaton/headless_chrome_ssl_proxy/tree/proxy-comparison

By inspecting the traffic flows in wireshark I was able to see that the TLS handshake appears to complete successfully for puffing billy and headless chrome, it's just that Chrome then immediately drops the connection and retries. I don't know why this is, or why it would only happen in headless mode.

I've run out of time to investigate this, but thought I'd leave the repro and details here in case someone else is interested in following up.

[AlanFoster]

@urbanautomaton Awesome! Out of interest, how did you connect wireshark to inspect the traffic between puffing billy and headless chrome?

[urbanautomaton]

@AlanFoster I started a wireshark session on the loopback interface:

To show only the chrome <-> proxy traffic I added a display filter for tcp.port == 8081 (or whatever port the proxy was running on - I found it helpful to fix it as 8081 in the puffing billy config so I could more easily look at old traces).

The most useful view I found was the traffic flow - once you've got a recording with a request in go Statistics > Flow Graph, and select "Show: displayed packets" to make it respect the display filter. You should then be looking at the back and forth between chrome and puffing billy.

That's about the limit of my wireshark expertise - good luck, hope you find out what's going on. 👍

[ryan-plated]

I have a feeling this bug may have something to do with event machine #233

[ryansch]

HTTPS proxying is working well for us via an approach inspired by @Jack12816.

spec/feature/support/billy_ssl.rb:

module BillySsl
  def add_authority_to_chrome
    puts 'Adding billy CA to chrome'

    cmd = TTY::Command.new(printer: :quiet)
    cmd.run <<~SCRIPT
      cd "#{ENV['HOME']}"
      curl -s -k -o "cacert-root.crt" "http://www.cacert.org/certs/root.crt"
      curl -s -k -o "cacert-class3.crt" "http://www.cacert.org/certs/class3.crt"
      echo > .password
      mkdir -p .pki/nssdb
      CERT_DIR=sql:$HOME/.pki/nssdb
      certutil -N -d .pki/nssdb -f .password
      certutil -d ${CERT_DIR}  -A -t TC \
        -n "CAcert.org" -i cacert-root.crt
      certutil -d ${CERT_DIR} -A -t TC \
        -n "CAcert.org Class 3" -i cacert-class3.crt
      certutil -d sql:$HOME/.pki/nssdb -A \
        -n puffing-billy -t "CT,C,C" -i #{Billy.certificate_authority.cert_file}
    SCRIPT
  end
  module_function :add_authority_to_chrome
end

In spec/feature/feature_helper.rb:

RSpec.configure do |config|
  config.before :suite do
    BillySsl.add_authority_to_chrome
  end
end

In spec/rails_helper.rb:

  Capybara.register_driver :headless_chrome_billy do |app|
    capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
      # Turn on browser logs
      loggingPrefs: {
        browser: 'ALL'
      }
    )

    options = Selenium::WebDriver::Chrome::Options.new
    options.headless!
    options.add_argument('--disable-gpu')
    options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")
    options.add_argument('--proxy-bypass-list=127.0.0.1')

    Capybara::Selenium::Driver.new app,
      browser: :chrome,
      options: options,
      desired_capabilities: capabilities,
      driver_opts: {
        # log_path: '/srv/chromedriver.log',
        # verbose: true
      }
  end

  Capybara.javascript_driver = :headless_chrome_billy

  Capybara.server_port = 7787

[ndbroadbent]

Thanks for the PR @Jack12816! I finally got around to switching to headless Chrome after running into some weird issues with poltergeist today. I was able to skip all of the SSL certificate stuff, because the acceptInsecureCerts option is working for me.

Here's the versions of everything on my MacBook:

• Mac OS Mojave 10.14
• Ruby 2.5.3p105
• puffing-billy 1.1.2
• capybara 3.10.0
• chromedriver 2.44.609545
• Google Chrome 71.0.3578.80

The specs are also passing on CI (GitLab), running in a Docker container based on Debian jessie. My Dockerfile is based on this one. (Same versions of everything, except chromedriver was slightly updated to 2.44.609551.)

My spec/rails_helper.rb:

Capybara.register_driver :headless_chrome_billy do |app|
    capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
      acceptInsecureCerts: true,
      loggingPrefs: { browser: 'ALL' }
    )
    options = Selenium::WebDriver::Chrome::Options.new
    options.headless!
    options.add_argument('--disable-gpu')
    options.add_argument('--window-size=1280,1000')
    options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")
    options.add_argument('--proxy-bypass-list=127.0.0.1;localhost')

    Capybara::Selenium::Driver.new app,
      browser: :chrome,
      options: options,
      desired_capabilities: capabilities,
      driver_opts: {
        log_path: Rails.root.join('log/chromedriver.log').to_s,
        verbose: true,
      }
  end
  Capybara.javascript_driver = :headless_chrome_billy

I didn't need the --disable-web-security or --no-sandbox options on my Mac. I also read this post:

Note: --no-sandbox is not needed if you properly setup a user in the container.

I set up the the chrome user in my CI Docker image instead of using --no-sandbox, but you might need it: options.add_argument('--no-sandbox').

---

One thing I should mention is that I spent about an hour trying to get this to work, but then it suddenly started working and I don't really know why.

I was printing the browser logs in one of my specs, which was really helpful: puts page.driver.browser.manage.logs.get(:browser).

When I didn't have the acceptInsecureCerts: true option, I was getting this error:

SEVERE 2018-12-07 21:15:15 +0700: https://fonts.googleapis.com/css?family=... 
- Failed to load resource: net::ERR_CERT_AUTHORITY_INVALID

After I added acceptInsecureCerts: true, I started getting this error instead:

SEVERE 2018-12-07 21:15:15 +0700: https://fonts.googleapis.com/css?family=... 
- Failed to load resource: net::ERR_TOO_MANY_RETRIES

Unfortunately I can't remember the exact series of steps that caused the "too many retries" error to disappear, and I can't reproduce it anymore. I think it might have stopped happening after I deleted spec/req_cache, removed the --headless argument, and ran the tests from scratch in a real browser window. Then I added --headless again, and the tests were still passing and using the cached responses. I ran rm -rf spec/req_cache one more time, and the tests were still passing and correctly caching the responses. So if you run into any problems, try removing spec/req_cache and running without the --headless flag.

[ronwsmith]

Thanks everyone for the conversation around this issue. Are there still issues to address that need this issue to remain open? Thanks!

[urbanautomaton]

My repro now works fine with the latest puffing billy and chromedriver, with no changes required. Not sure who fixed this, but whoever it was, thank you very much! ❤️

[puloms]

Hello! I was having the same problem as you all, but with Cucumber and testing an external application, the BillySsl module works fine, but I had a bit of a problems with password when running the certutil commands, mostly when running more than one time, so, I've made some changes, and works like a charm! Here my code in case anyone have the same problem:

module BillySsl
  def add_authority_to_chrome
    puts 'Adding billy CA to chrome'

    cmd = TTY::Command.new(printer: :null)
    cmd.run <<~SCRIPT
      cd "#{ENV['HOME']}"
      curl -s -k -o "cacert-root.crt" "http://www.cacert.org/certs/root.crt"
      curl -s -k -o "cacert-class3.crt" "http://www.cacert.org/certs/class3.crt"
      if ! [ -d .pki/nssdb ]; then
        mkdir -p .pki/nssdb
        certutil -N -d .pki/nssdb --empty-password
      fi
      CERT_DIR=sql:$HOME/.pki/nssdb
      certutil -d ${CERT_DIR}  -A -t TC \
        -n "CAcert.org" -i cacert-root.crt
      certutil -d ${CERT_DIR} -A -t TC \
        -n "CAcert.org Class 3" -i cacert-class3.crt
      certutil -d sql:$HOME/.pki/nssdb -A \
        -n puffing-billy -t "CT,C,C" -i #{Billy.certificate_authority.cert_file}
    SCRIPT
  end
  module_function :add_authority_to_chrome
end

Then in my env.rb file:

BillySsl.add_authority_to_chrome

That's it, works both with or without headless and works fine in a Docker/Alpine!

[raldred]

@puloms where do you get certutil from for MacOS?

[dentarg]

@raldred certutil seems to be something from Mozilla:

• https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS
• https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/tools/NSS_Tools_certutil

It can be installed from Homebrew: brew install nss

(I haven't tested this myself)

[ronwsmith]

Closing, seems the issue is resolved now. Let me know if it's not and we can reopen.