web-ext debugger not accepting connection from Puppeteer or Playwright #1927
Comments
lfilho
changed the title
|
I have also tested with const browserURL = `ws://127.0.0.1:${runningInfo.debuggerPort}`;
// Also tried with "http:" instead of "ws:"
const browser = await browserWrapper.firefox.connect({
wsEndpoint: browserURL,
logger: {
isEnabled: () => true,
log: (name, severity, message, args) => console.log(`${name} ${message}`),
},
});output: |
This was referenced Jun 13, 2020
|
I took a quick look into this and based on the documentation related to the CDP protocol support in Firefox: https://firefox-source-docs.mozilla.org/remote/Usage.html
The debugging port that web-ext is using is not the one the CDP one that puppeteer is supposed to connect to (web-ext is using the Firefox DevTools remote debugging protocol to interact with the Firefox instance, which isn't the same protocol that puppeteer or playwright are using), and so you may need to ask web-ext to pass the additional command line parameters and points the web-ext instance to a Firefox Nightly binary. I haven't tried with Playwright, but Puppeteer seems to be able to connect successfully using something like the following: |
|
Hi @rpl, thanks for looking into this. That The fact that After your message I ran Will try again and report back soon. Thanks again! |
|
Ok, learnings so far: 1. web-ext doesn't seem to work with ESM modules as the documentation suggest:
I'll break down several repro codes and their issues: ESM + Puppeteer only
import child_process from 'child_process';
import webExt from 'web-ext';
import getPort from 'get-port';
import pptr from 'puppeteer-core';
let page;
let browser;
(async () => {
const cdpPort = await getPort();
await webExt.default.cmd.run(
{
sourceDir: `${process.cwd()}/src`,
args: [`--remote-debugger=localhost:${cdpPort}`, `--verbose`],
firefox: '/Applications/Firefox Nightly.app/Contents/MacOS/firefox-bin',
},
{ shouldExitProgram: false }
);
// Needed because `webExt.cmd.run` returns before the DevTools agent starts running.
// Alternative would be to wrap the call to pptr.connect() with some custom retry logic
child_process.execSync('sleep 5');
const browserURL = `http://127.0.0.1:${cdpPort}`;
browser = await pptr.connect({
browserURL,
logger: {
isEnabled: () => true,
log: (name, severity, message, args) => {
console.log(`[${severity}] ${name} ${message}. Args: ${args}`);
},
},
});
page = await browser.newPage();
await page.goto('https://mozilla.org');
})();Result: works. Firefox nightly gets started and i see the page. ESM + Jest + Puppeter (WITHOUT web-ext)
import pptr from 'puppeteer-core';
let page, browser;
beforeAll(async () => {
browser = await pptr.launch({
product: 'firefox',
executablePath:
'/Applications/Firefox Nightly.app/Contents/MacOS/firefox-bin',
headless: false,
});
page = await browser.newPage();
});
afterAll(async () => {
await browser.close();
});
describe('loads mozilla', () => {
it('tests for a random element', async () => {
await page.goto('https://mozilla.org');
const text = await page.evaluate(() => {
return document.querySelector('#home');
});
expect(text).toBeTruthy();
});
});Results: works. Browser opens, loads mozilla.org and test pass. ESM + Jest + Puppeter (WITH web-ext) import child_process from 'child_process';
import webExt from 'web-ext';
import getPort from 'get-port';
import pptr from 'puppeteer-core';
let page, browser;
beforeAll(async () => {
const cdpPort = await getPort();
webExt.default.util.logger.consoleStream.makeVerbose();
await webExt.default.cmd.run(
{
noInput: true,
sourceDir: `${process.cwd()}/src`,
args: [`--remote-debugger=localhost:${cdpPort}`, `--verbose`],
firefox: '/Applications/Firefox Nightly.app/Contents/MacOS/firefox-bin',
},
{ shouldExitProgram: false }
);
child_process.execSync('sleep 15');
const browserURL = `http://127.0.0.1:${cdpPort}`;
browser = await pptr.connect({
browserURL,
logger: {
isEnabled: () => true,
log: (name, severity, message, args) => {
console.log(`[${severity}] ${name} ${message}. Args: ${args}`);
},
},
});
page = await browser.newPage();
});
afterAll(async () => {
await browser.close();
});
describe('loads mozilla', () => {
it('tests for a random element', async () => {
await page.goto('https://mozilla.org');
const text = await page.evaluate(() => {
return document.querySelector('#home');
});
expect(text).toBeTruthy();
});
});Results: does not work! Logs: Not sure what could be the problem here :( Notice I put a high sleep number in order to give plenty of time for the remote debugger to start... It looks like something in the way how web-ext spins a browser up conflicts with how jest handles asynchronous calls? Not sure |
|
I create a simple PR to fix the examples in the docs: #1933 |
|
I digged a little into this while triaging Playwright bug. Here's how to launch firefox with webextension and connect Playwright to it: const path = require('path');
const {firefox} = require('playwright');
const webExt = require('web-ext').default;
(async () => {
// Enable verbose logging and start capturing logs.
webExt.util.logger.consoleStream.makeVerbose();
webExt.util.logger.consoleStream.startCapturing();
// Launch firefox
await webExt.cmd.run({
sourceDir: path.join(__dirname, 'webextension'),
firefox: firefox.executablePath(),
args: [`-juggler=1234`],
}, {
shouldExitProgram: false,
});
// Parse firefox logs and extract juggler endpoint.
const JUGGLER_MESSAGE = `Juggler listening on`;
const message = webExt.util.logger.consoleStream.capturedMessages.find(msg => msg.includes(JUGGLER_MESSAGE));
const wsEndpoint = message.split(JUGGLER_MESSAGE).pop();
// Connect playwright and start driving browser.
const browser = await firefox.connect({ wsEndpoint });
const page = await browser.newPage();
await page.goto('https://mozilla.org');
// .... go on driving ....
})();Hope it helps! |
|
Thank you so much, @aslushnikov . I confirm the above works! And here's the ESM version of it: import path from 'path';
import playwrightWrapper from 'playwright-firefox';
import webExtWrapper from 'web-ext';
const { firefox } = playwrightWrapper;
const webExt = webExtWrapper.default;
(async () => {
// Enable verbose logging and start capturing logs.
webExt.util.logger.consoleStream.makeVerbose();
webExt.util.logger.consoleStream.startCapturing();
// Launch firefox
await webExt.cmd.run(
{
sourceDir: path.join(process.cwd(), 'src'),
firefox: firefox.executablePath(),
args: [`-juggler=1234`],
},
{
shouldExitProgram: false,
}
);
// Parse firefox logs and extract juggler endpoint.
const JUGGLER_MESSAGE = `Juggler listening on`;
const message = webExt.util.logger.consoleStream.capturedMessages.find(
(msg) => msg.includes(JUGGLER_MESSAGE)
);
const wsEndpoint = message.split(JUGGLER_MESSAGE).pop();
// Connect playwright and start driving browser.
const browser = await firefox.connect({ wsEndpoint });
const page = await browser.newPage();
await page.goto('https://mozilla.org');
// .... go on driving ....
})();I'm still having troubles having it work with Jest 26.0 though, but I know this is out of scope for this repo. But for the interested ones, here is my code and the error I'm getting: import { jest, afterAll } from '@jest/globals';
import path from 'path';
import playwrightWrapper from 'playwright-firefox';
import webExtWrapper from 'web-ext';
import getPort from 'get-port';
const { firefox } = playwrightWrapper;
const webExt = webExtWrapper.default;
jest.mock('../../../src/feedback_sender.js');
webExt.util.logger.consoleStream.makeVerbose();
webExt.util.logger.consoleStream.startCapturing();
let browser, page, webExtRunner;
beforeAll(async () => {
webExtRunner = await webExt.cmd.run(
{
sourceDir: path.join(process.cwd(), 'src'),
firefox: firefox.executablePath(),
args: [`-juggler=${getPort()}`],
},
{
shouldExitProgram: false,
}
);
// Parse firefox logs and extract juggler endpoint.
const JUGGLER_MESSAGE = `Juggler listening on`;
const message = webExt.util.logger.consoleStream.capturedMessages.find(
(msg) => msg.includes(JUGGLER_MESSAGE)
);
const wsEndpoint = message.split(JUGGLER_MESSAGE).pop();
// Connect playwright and start driving browser.
browser = await firefox.connect({ wsEndpoint });
});
afterAll(async () => {
await browser.close();
await webExtRunner.exit();
});
beforeEach(async () => {
page = await browser.newPage();
});
afterEach(async () => {
jest.resetModules();
await page.close();
});
describe('Browser Action', () => {
it('button exists', async () => {
await page.goto(`https://mozilla.org`);
const submitButton = await page.$('#fxa-learn-primary');
await expect(submitButton).toBeTruthy();
});
});Error: I'm not sure if it's related to the experimental ESM support Jest has or not. If I have time I will convert it to commonjs to see if it that solves the problem or not. |
|
Any suggestion for the latest Web-Ext, Firefox and Puppeteer? Here's my code // @ts-check
const path = require('path');
const puppeteer = require('puppeteer-core');
const webExt = require('web-ext');
async function openFirefox() {
const runner = await webExt.cmd.run({
sourceDir: path.join(__dirname, 'extension'),
firefox: {
'darwin': '/Applications/Firefox Nightly.app/Contents/MacOS/firefox-bin',
'win32': `${process.env.PROGRAMFILES}\\Firefox Nightly\\firefox.exe`,
}[process.platform],
}, {
shouldExitProgram: false,
});
const {debuggerPort} = runner.extensionRunners[0].runningInfo;
return await puppeteer.connect({
browserWSEndpoint: `ws://localhost:${debuggerPort}`,
});
}
async function test() {
try {
await openFirefox();
console.log('WORKS');
} catch (err) {
console.error(err);
process.exit(13);
}
}
test();that fails with The suggestion by @aslushnikov breaks on |
After some tinkering I came up with the following code that seems to work flawlessly: const path = require('path');
const webExt = require('web-ext');
const puppeteer = require('puppeteer');
(async () => {
const browserFetcher = puppeteer.createBrowserFetcher({ product: 'firefox' });
const revisionInfo = await browserFetcher.download('83.0a1');
const cdpPort = 12345;
const extensionRunner = await webExt.cmd.run({
firefox: revisionInfo.executablePath,
sourceDir: path.resolve(__dirname, '../browser-extension'),
args: [ '--remote-debugging-port', cdpPort ]
}, { shouldExitProgram: false });
const browser = await puppeteer.connect({
browserURL: `http://localhost:${cdpPort}`,
product: 'firefox'
});
// ...
})();Note that I'm passing the port number as an additional array element instead of a single element |
|
Thank you @ech0-de! So, the actual solution is adding a port as a second item in |
|
This has stopped working, right? |
This was referenced Jun 16, 2022
|
For everyone interested in running Puppeteer with Firefox extension - this code sample loads the extension, but hangs on loading the extension page. The page loads but Puppeteer fails to detect that fact and waits until timeout. Playwright behave in the same way. @rpl it would be nice if import puppeteer from 'puppeteer';
import { firefox } from 'playwright-firefox';
import getPort from 'get-port';
import { connect } from './node_modules/web-ext/lib/firefox/remote.js';
import fs from 'node:fs';
import path from 'node:path';
const ADDON_UUID = 'd56a5b99-51b6-4e83-ab23-796216679614';
const ADDON_ID = JSON.parse(fs.readFileSync(path.join('webextension-dummy', 'manifest.json'))).browser_specific_settings.gecko.id;
const rppPort = await getPort();
const browser = await puppeteer.launch({
headless: false,
product: 'firefox',
args: [
`--start-debugger-server=${rppPort}`,
],
extraPrefsFirefox: {
'devtools.chrome.enabled': true,
'devtools.debugger.prompt-connection': false,
'devtools.debugger.remote-enabled': true,
'toolkit.telemetry.reportingpolicy.firstRun': false,
'extensions.webextensions.uuids': `{"${ADDON_ID}": "${ADDON_UUID}"}`,
},
executablePath: firefox.executablePath(),
});
const rdp = await connect(rppPort);
await rdp.installTemporaryAddon(path.resolve('webextension-dummy'));
const page = await browser.newPage();
await page.goto(`moz-extension://${ADDON_UUID}/index.html`);
await page.click('button');
await browser.close();Linked puppeteer issue puppeteer/puppeteer#9548 |
I see, so in your snippet you are starting the Firefox instance through puppeteer helpers and playwright's firefox build and then connecting to that instance through Firefox Remote Debugging Protocol and installing the extension. @chrmod now that web-ext is ES modules based, it should be just a matter of adding one or two additional exports from the package.json here: Lines 7 to 11 in 6659079 Personally I wouldn't be against adding firefox/index.js and firefox/remote.js to the modules that we provides "exports" shortcuts for. |
Loading/accessing extension pages works with protocol: 'webDriverBiDi' in puppeteers.launch🙌🏼 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is this a feature request or a bug?
Bug
Problem
Ultimately what I want is to programatically test a WebExtension i'm writing with something like
jest. Since sideloading doesn't work since FF 73 (https://blog.mozilla.org/addons/2019/10/31/firefox-to-discontinue-sideloaded-extensions/). I can't see a way of automatically installing an extension in my dev environment.web-extseems to be the only option. Hence I need to connectpuppeteer(orplaywright) to the FF thatweb-extsping up, so finally I could use it with jest to run my automated tests.I'm trying to use puppeteer to connect to web-ext's Firefox as suggested in puppeteer/puppeteer#5532 (comment) . Didn't work.
Steps to reproduce
Tell us about your environment:
What steps will reproduce the problem?
Save the below to a file an run
node my-file.js:What is the expected result?
What happens instead?
Error:
What else I tried:
ws://127.0.0.1:${runningInfo.debuggerPort}instead ofhttp://127.0.0.1:${runningInfo.debuggerPort}browserWSEndpointas a parameter toconnect()instead ofbrowserURLFull log for the above script:
The text was updated successfully, but these errors were encountered: