Issue building vscode against >=electron-21

[PF4Public]

I've posted this into vscode repo, but maybe you guys have any idea what may be causing this?

The error tree_sitter_tsx_external_scanner_create comes from

vscode-l10n/l10n-dev/src/ast/analyzer.ts

Lines 41 to 52 in b941cad

	static #tsGrammar: Promise<Parser.Language> = (async () => {
	await initParser;
	return await Parser.Language.load(
	path.resolve(__dirname, 'tree-sitter-typescript.wasm')
	);
	})();
	static #tsxGrammar: Promise<Parser.Language> = (async () => {
	await initParser;
	return await Parser.Language.load(
	path.resolve(__dirname, 'tree-sitter-tsx.wasm')
	);
	})();

Both wasm files come pre-built from @vscode/l10n-dev.
Changing the dependency to version 0.0.24 changes the error (although that may be a race condition between the two):

Unhandled Rejection at: Promise Promise {
  <rejected> Error: bad export type for `tree_sitter_typescript_external_scanner_create`: undefined
      at reportUndefinedSymbols (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:19748)
      at postInstantiation (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:17027)
      at /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:17752
      at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:270:10
      at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:259:22
} reason: Error: bad export type for `tree_sitter_typescript_external_scanner_create`: undefined
    at reportUndefinedSymbols (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:19748)
    at postInstantiation (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:17027)
    at /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:17752
    at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:270:10
    at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:259:22

Lines 259 and 270 from the error look as follows:

__privateAdd(ScriptAnalyzer, _tsxParser, (async () => {
  await initParser;
  const parser = new import_web_tree_sitter.default();
  parser.setLanguage(await __privateGet(_ScriptAnalyzer, _tsxGrammar));
  return parser;
})());

__privateAdd(ScriptAnalyzer, _tsxGrammar, (async () => {
  await initParser;
  return await import_web_tree_sitter.default.Language.load(
    path.resolve(__dirname, "tree-sitter-tsx.wasm")
  );
})());

In this case tree_sitter_typescript_external_scanner_create is absent from tree-sitter-tsx.wasm, but present in tree-sitter-typescript.wasm.

I have no idea, why is it trying to load a function that is absent from wasm file. Broken wasm files in @vscode/l10n-dev?

It is interesting that electron-19 and electron-20 show no such behaviour. See fresh logs here.

[TylerLeonhardt]

I've not heard any other reports of this. We've been on Electron 21 for a decent amount of time now... and no one on the team has mentioned this and our localization pipelines are still running fine post-Electron 21.

Maybe you should start by telling me how you are building VS Code.

[PF4Public]

I've not heard any other reports of this.

I realize that!

Maybe you should start by telling me how you are building VS Code.

Sure thing. Here it is: https://github.com/PF4Public/gentoo-overlay/blob/master/app-editors/vscode/vscode-9999.ebuild

Most interesting parts might be following:

• Installing node_modules
  https://github.com/PF4Public/gentoo-overlay/blob/ef81533a567293098ac20607528055f101b7e842/app-editors/vscode/vscode-9999.ebuild#L237-L264

	#TODO: should work starting with electron-22
	if use electron-20 || use electron-21 || use electron-22 || use electron-23 || use electron-24 ; then
		CPPFLAGS="${CPPFLAGS} -std=c++17";
		use build-online || eerror "build-online should be enabled for nan substitution to work" || die;
		sed -i 's$"resolutions": {$"resolutions": {"nan": "^2.17.0",$' package.json || die;
	fi

	ebegin "Installing node_modules"
	# yarn config set yarn-offline-mirror ${T}/yarn_cache || die
	OLD_PATH=$PATH
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}/node_modules/npm/bin/node-gyp-bin:$PATH"
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}/node_modules/npm/bin:$PATH"
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}:$PATH"
	export PATH
	export CFLAGS="${CFLAGS} -I/usr/include/electron-${ELECTRON_SLOT}/node"
	export CPPFLAGS="${CPPFLAGS} -I/usr/include/electron-${ELECTRON_SLOT}/node"
	export ELECTRON_SKIP_BINARY_DOWNLOAD=1
	export PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
	# echo "$PATH"
	yarn config set disable-self-update-check true || die
	yarn config set nodedir /usr/include/electron-${ELECTRON_SLOT}/node || die
	if ! use build-online
	then
		ONLINE_OFFLINE="--offline"
		yarn config set yarn-offline-mirror "${DISTDIR}" || die
	fi
	yarn install --frozen-lockfile ${ONLINE_OFFLINE} \
		--arch=${VSCODE_ARCH} --no-progress || die

• Building everything
  https://github.com/PF4Public/gentoo-overlay/blob/ef81533a567293098ac20607528055f101b7e842/app-editors/vscode/vscode-9999.ebuild#L362-L370

	OLD_PATH=$PATH
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}/node_modules/npm/bin/node-gyp-bin:$PATH"
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}/node_modules/npm/bin:$PATH"
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}:$PATH"
	export PATH

	node --max_old_space_size=8192 node_modules/gulp/bin/gulp.js vscode-linux-${VSCODE_ARCH}-min || die

	export PATH=${OLD_PATH}

• Run Debian packaging task and just copy needed files one by one
  https://github.com/PF4Public/gentoo-overlay/blob/ef81533a567293098ac20607528055f101b7e842/app-editors/vscode/vscode-9999.ebuild#L374-L380

	OLD_PATH=$PATH
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}/node_modules/npm/bin/node-gyp-bin:$PATH"
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}/node_modules/npm/bin:$PATH"
	PATH="/usr/$(get_libdir)/electron-${ELECTRON_SLOT}:$PATH"
	export PATH

	YARN_CACHE_FOLDER="${T}/.yarn-cache" node node_modules/gulp/bin/gulp.js vscode-linux-${VSCODE_ARCH}-prepare-deb || die

Interesting observation: in GitHub actions it fails at the preparing Debian deb file. On a real hardware it fails at api-proposal-names

I tried following:
https://github.com/PF4Public/gentoo-overlay/blob/ef81533a567293098ac20607528055f101b7e842/app-editors/vscode/vscode-9999.ebuild#L342-L344

	#! Although this allows the build to continue, it renders vscode unusable
	einfo "Fixing l10n-dev"
	sed -i 's/return await import_web_tree_sitter/return null; await import_web_tree_sitter/' node_modules/@vscode/l10n-dev/dist/main.js || die

It solves the issue, but the resulting vscode is obviously unusable.

Like I've mentioned in vscode issue, I've bisected it to microsoft/vscode@0510da8 and indeed microsoft/vscode@88c8146 is the last functioning commit for me.

[TylerLeonhardt]

What version of node are you using to build vscode?

[PF4Public]

I'm using ELECTRON_RUN_AS_NODE=1 electron for node.

[TylerLeonhardt]

You're using electron as the node used to build VS Code instead of using regular node to build VS Code?

[PF4Public]

Exactly.

[TylerLeonhardt]

Why are you doing this instead of the recommended way to build VS Code which is to use actual Node.js? I don't think we can expect this to work.

[PF4Public]

I use it mainly to lower the number of dependencies that are required to build vscode. I must admit this must be indeed a very rare edge-case. Still it is curious how it worked flawlessly until electron 21.

Thank you for suggesting to try the real node. Using it just for the failing steps seems to resolve the issue.

[TylerLeonhardt]

Really, we should get an issue going over here:

https://github.com/tree-sitter/tree-sitter-typescript

Because clearly the error comes from there.

Would you like to get that going? @PF4Public maybe we can get some interesting info from the real experts.

[PF4Public]

Would you like to get that going?

Of course! Even though this could be fixed by using real node, I'm still curious why it stopped working with electron acting as node.

Would you like me to open issue there?

[TylerLeonhardt]

That'd be great! Feel free to tag me.

[TylerLeonhardt]

Created this to track bumping our version of tree-sitter when the day comes #114