Module util could not be found

[MaTiAtSIE]

Steps to reproduce

Setup:
Typescript == 4.9.5
node == 20.0.0
theia == 1.45.0
@dsnp/parquetjs == 1.6.0
Webpack == 5.90.3

I develop a theia application where I added @dsnp/parquetjs to a theia extension with

yarn add @dsnp/parquetjs

After that, @dsnp/parquetjs version 1.6.0 was added. I implemented the ParquetReader example in the backend. The build was conducted without any error. However, at runtime, I get an error in the backend saying that the module util could not be imported from wasm_brotli_nodejs.
To fix this, we make the following changes in wasm_brotli_nodejs.js:

// old
// const { TextDecoder } = require(String.raw`util`);
// this fixes the import error
const { TextDecoder } = require(`util`);

Subsequently, we get another runtime error that wasm_brotli_nodejs_bg.wasm could not be found in applications\theia-browser\lib\backend. This error could be solved by copying the file into the directory. As an alternative, it was also possible to solve this issue with some modifications in wasm_brotli_nodejs_bg.js.

The question is: What has to be done to consume @dsnp/parquetjs without these modifications from our node app with webpack 5?

[wilwade]

Hi @MaTiAtSIE

I'm not sure how theia changes things, but here's some notes about using webpack: #116 (comment)

[mpoferl]

I have the same error. I followed #116. Locally it works, but when I invoke my AWS Lambda, it fails. After updating the webpack, I receive this error:

{
  "errorType": "TypeError",
  "errorMessage": "Cannot read properties of undefined (reading 'webkitRequestFileSystem')",
  "trace": [
    "TypeError: Cannot read properties of undefined (reading 'webkitRequestFileSystem')",
    "    at Object.<anonymous> (/var/task/bundle.js:2:596096)",
    "    at Object.<anonymous> (/var/task/bundle.js:2:696642)",
    "    at n (/var/task/bundle.js:2:533097)",
    "    at /var/task/bundle.js:2:533175",
    "    at a (/var/task/bundle.js:2:533180)",
    "    at /var/task/bundle.js:2:777728",
    "    at /var/task/bundle.js:2:327727",
    "    at 5039 (/var/task/bundle.js:2:1067488)",
    "    at n (/var/task/bundle.js:2:1279807)",
    "    at 3481 (/var/task/bundle.js:2:1269084)"
  ]
}

[MaTiAtSIE]

@wilwade Thanks for the hint. I extended my webpack.config.js script with

const path = require('path');
// Function to update the configuration
function updateWebpackConfig(config) {
    // Ensure the resolve.alias object exists
    config.resolve = config.resolve || {};
    config.resolve.alias = config.resolve.alias || {};
  
    // Add your alias
    const aliasKey = '@dsnp/parquetjs';
    const aliasValue = path.resolve('..', '..', 'node_modules', '@dsnp/parquetjs', 'dist', 'browser', 'parquet.cjs.js');
  
    // Check if the alias already exists
    if (!config.resolve.alias[aliasKey]) {
      // Add the alias if it doesn't exist
      config.resolve.alias[aliasKey] = aliasValue;
      console.log('added alias');
    } else {
      // Handle the case where the alias already exists
      console.warn(`Alias "${aliasKey}" already exists.`);
    }
  
    return config;
  }
nodeConfig.config = updateWebpackConfig(nodeConfig.config);

which compiles. After starting my application, I get directly the same error as @mpoferl and others in #116 : 'TypeError: Cannot read properties of undefined (reading 'webkitRequestFileSystem')'.

[MaTiAtSIE]

We tested another alternative: Using brotli-wasm v2.0.1 as drop in replacement of wasm-brotli.

• yarn add brotli-wasm
• Replace wasm-brotli with brotli-wasm
  • in package.json of @dsnp/parquetjs; change version, too
  • in compression.js in node_modules/@dsnp/parquetjs/dist/lib

This worked for us. No changes in the webpack configuration were required.
@wilwade: Is it possible that you use brotli-wasm as alternative to wasm-brotli in @dsnp/parquetjs?

[wilwade]

@MaTiAtSIE that looks like a clean update to me as well. Made a PR #123 and try to get it merged and a release out today.

[wilwade]

@MaTiAtSIE v1.6.2 is out with the suggested fix.