Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS.mount() throws an error when trying to mount local files with WORKERFS #328

Closed
ColinFay opened this issue Dec 6, 2023 · 5 comments
Closed
Labels
bug Something isn't working

Comments

@ColinFay
Copy link
Contributor

ColinFay commented Dec 6, 2023

Following the example from https://docs.r-wasm.org/webr/latest/mounting.html#javascript-api

Here is a reprex.

Step 1, using the ghcr.io/r-wasm/webr image :

docker run -it -v /Users/colin/webr-mount-reprex:/root/output ghcr.io/r-wasm/webr R
> dir.create("test")
> setwd("test")
> write.csv(iris, "iris.csv")
> rwasm::file_packager(".", out_dir = ".", out_name = "iris")
Packaging: iris.data
> list.files()
[1] "iris.csv"         "iris.data"        "iris.js.metadata"
> file.copy("iris.data", "/root/output/iris.data")
[1] TRUE
> file.copy("iris.js.metadata", "/root/output/iris.js.metadata")
[1] TRUE

Step 2, move to a node js script and try to load from disk

const fs = require('fs');
const path = require('path');
const { WebR } = require('webr');
const webR = new WebR();

(async () => {

  await webR.init();

  const data = new Blob(
    fs.readFileSync(
      path.join(__dirname, 'iris.data')
    )
  );

  console.log(data)

  const metadata = JSON.parse(
    fs.readFileSync(
      path.join(__dirname, 'iris.js.metadata')
    )
  );

  console.log(metadata)

  await webR.FS.mkdir('/data');

  const options = {
    packages: [{
      blob: data,
      metadata: metadata,
    }],
  }
  await webR.FS.mount("WORKERFS", options, '/data');

})();

Even if both data and metadata are console.loged, the process throws an error :

node index.js 
Blob { size: 10727, type: '' }
{
  files: [ { filename: '/iris.csv', start: 0, end: 4821 } ],
  remote_package_size: 4821
}
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

We [RuntimeError]: unreachable
    at wasm:/wasm/0169de02:wasm-function[4726]:0x2ec63b
    at abort (/Users/colin/webr-mount-reprex/node_modules/webr/dist/R.bin.js:1846:31535)
    at assert (/Users/colin/webr-mount-reprex/node_modules/webr/dist/R.bin.js:1846:28223)
    at Object.mount (/Users/colin/webr-mount-reprex/node_modules/webr/dist/R.bin.js:1846:57014)
    at Object.mount (/Users/colin/webr-mount-reprex/node_modules/webr/dist/R.bin.js:1846:66786)
    at SharedBufferChannelWorker.dispatch (/Users/colin/webr-mount-reprex/node_modules/webr/dist/webr-worker.js:4218:24)
    at SharedBufferChannelWorker.inputOrDispatch (/Users/colin/webr-mount-reprex/node_modules/webr/dist/webr-worker.js:2613:37)
    at Object.readConsole (/Users/colin/webr-mount-reprex/node_modules/webr/dist/webr-worker.js:4821:19)
    at 1105043 (/Users/colin/webr-mount-reprex/node_modules/webr/dist/R.bin.js:1846:34821)
    at runEmAsmFunction (/Users/colin/webr-mount-reprex/node_modules/webr/dist/R.bin.js:1846:318358)

Node.js v18.17.1

I've tried using

  const options = {
    packages: [{
      blob: await data,
      metadata: await metadata,
    }],
  }

But get the same result

@ColinFay
Copy link
Contributor Author

ColinFay commented Dec 6, 2023

@ColinFay
Copy link
Contributor Author

ColinFay commented Dec 6, 2023

Same happens if I try to reproduce the code from the documentation, using fetch :

const { WebR } = require('webr');
const webR = new WebR();

(async () => {

  await webR.init();

  // Create mountpoint
  await webR.FS.mkdir('/data')

  // Download image data
  const data = await fetch('https://raw.githubusercontent.com/ColinFay/webr-mount-reprex/main/iris.data');
  const metadata = await fetch('https://raw.githubusercontent.com/ColinFay/webr-mount-reprex/main/iris.js.metadata');

  // Mount image data
  const options = {
    packages: [{
      blob: await data.blob(),
      metadata: await metadata.json(),
    }],
  }
  await webR.FS.mount("WORKERFS", options, '/data');

})();
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

We [RuntimeError]: unreachable
    at wasm://wasm/0169de02:wasm-function[4726]:0x2ec63b

@georgestagg
Copy link
Member

Thank you for reporting this and for the example, I can reproduce the bug at my end.

It looks like the issue is specific to using the WORKERFS Emscripten filesystem type under Node.js. The same issue does not seem to occur when running webR in a web browser, which is the use case those examples in the documentation are largely targeting. I will investigate further and report back.

In the meantime, I know it might not be quite what you had in mind, but you should be able to use the alternative NODEFS filesystem type to mount local directories directly without having to package the contents of the directory as an Emscripten filesystem image.

First, write the data you're interested in to disk in a new directory,

$ mkdir data
$ Rscript -e 'write.csv(iris, "data/iris.csv")'

Then, mount the new directory with NODEFS in index.js:

const { WebR } = require('webr');
const webR = new WebR();

(async () => {
  await webR.init();
  await webR.FS.mkdir('/data');
  await webR.FS.mount("NODEFS", { root: './data' }, '/data');
  await webR.evalRVoid("print(head(read.csv('/data/iris.csv')))")

  process.exit(1);
})();
$ node index.js
  X Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1 1          5.1         3.5          1.4         0.2  setosa
2 2          4.9         3.0          1.4         0.2  setosa
3 3          4.7         3.2          1.3         0.2  setosa
4 4          4.6         3.1          1.5         0.2  setosa
5 5          5.0         3.6          1.4         0.2  setosa
6 6          5.4         3.9          1.7         0.4  setosa

@georgestagg georgestagg added the bug Something isn't working label Dec 6, 2023
@ColinFay
Copy link
Contributor Author

ColinFay commented Dec 6, 2023

Yeah this is what I've been doing so far (see https://colinfay.me/rethinking-packages-and-functions-preloading-in-webr-0.2.2/ for ex) but I want to package some stuff into a node module, and I feel like it would be better to upload the .data and .js.metadata files to npm than to upload a whole folder with many files :)

georgestagg added a commit that referenced this issue Sep 3, 2024
Instead, use our own `mountImageData()` function to create VFS nodes for
each file in the VFS metadata package.

TODO: This currently handles only metadata given in the form of the
`packages` property. Emscripten supports additional `files` and `blobs`
properties, and in the future we should also support those here.

Fixes #328.
@georgestagg
Copy link
Member

As of 4655e96, to be included in the next release of webR, mounting WORKERFS images under Node.js should now work.

Note that in your original test script, the argument of new Blob(...) should be in the form of an array:

  const data = new Blob([
    fs.readFileSync(
      path.join(__dirname, 'iris.data')
    )
  ]);

It should also work when passing the Buffer returned by fs.readFileSync() directly for the blob key.

Here is essentially the test script I've been using. Once we cut a new release of webR and it hits npm you should be able to try it out for yourself.

const fs = require('fs');
const path = require('path');
const { WebR } = require('webr');
const webR = new WebR();

(async () => {
  await webR.init();

  const data = new Blob([
    fs.readFileSync(
      path.join(__dirname, 'iris.data')
    )
  ])
  const metadata = JSON.parse(
    fs.readFileSync(
      path.join(__dirname, 'iris.js.metadata')
    )
  );

  await webR.FS.mkdir('/data');

  const options = {
    packages: [{
      blob: data,
      metadata: metadata,
    }],
  }
  await webR.FS.mount("WORKERFS", options, '/data');

  await webR.evalRVoid("print(file.info('/data/iris.csv'))")
  await webR.evalRVoid("print(head(read.csv('/data/iris.csv')))")
  
  process.exit(1);
})();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants