Out of memory or out of time when compiling large JS files

[jeaye]

Do you want to request a feature or report a bug?
Bug.

What is the current behavior?
Metro will either run out of memory or it will give up after 5 minutes of waiting to transform the JS file.

This was first reported here facebook/react-native#8475 and then again here facebook/react-native#12590

If the current behavior is a bug, please provide the steps to reproduce and a minimal repository on GitHub that we can yarn install and yarn test.
I have a repro repo here https://github.com/jeaye/react-native-packager-bug -- it's not in the yarn format, since I don't know what that is.

What is the expected behavior?
The transformation completes in a timely fashion or metro waits long enough for the transformation to finish in however long it needs. In order to achieve the latter, my production build script currently contains the following:

metro=../node_modules/metro-bundler/src/JSTransformer/index.js
sed -i 's/\(TRANSFORM_TIMEOUT_INTERVAL\) = .*;/\1 = 901000;/g' "$metro" || true

Please provide your exact Metro configuration and mention your Metro, node, yarn/npm version and operating system.

React Native: 0.50.4
babel-preset-react-native: 4.0.0
Metro (comes with RN): 0.20.3
Node: 9.3.0
OS: Arch Linux x86_64

[jeaye]

Happy new year! Any update on this?

[rafeca]

Hey @jeaye ! Happy new year and apologies for the delay.

The timeout on transformations was indeed [removed some time ago]( (d4dcd4c), so from metro v0.21.0 there is no maximum time to transform a file.

Out of curiosity, can you provide more details about the file that you're trying to compile? How big is it? Why is it so big? Metro is heavily optimized to build bundles with a really big amount of small files, but cannot run that fast when trying to compile a small amount of huge files (it's impossible to parallelize, the caching system is not efficient, etc(.

[jeaye]

@rafeca In production, my index.android.js is 1.2MB; these out-of-memory and timeout issues happen even before the 1MB mark though. Why is it that big? Well, it's a mid-sized application, about 10K LOC of ClojureScript, compiled with Google Closure.

As linked above, this issue is affect basically everyone using ClojureScript with RN, likely due to the CLJS stdlib and everything else included in the compiled file. Note, this is going through Google Closure, so it's not like there's a lot of waste here which we can trim down; all dead code is removed and all remaining code is optimized and minified.

Closing this issue because the timeout is removed is ok, but, really, waiting over 5 minutes to transform these files is pretty awful. Trying to do this with a non-production build (which doesn't have Google Closure optimizations applied) is a fool's errand, since it'll take more like 30 minutes, if it doesn't run out of memory before that.

[rafeca]

@jeaye : thanks for the explanation! it makes complete sense 😄

In this case, since your index.android.js file has already been transpiled, I suggest you to try to disable babel transformation for that file and check how fast the transformation is (spoiler: it should be way faster). You can do that via the ignore param in your babel config.

Anyways, this flow that you describe is still going to be slow and painful in development mode (as you describe, every time you do a change you have to recompile the whole bundle from clojurescript to javascript, and then metro will see the modified file so it won't be able to leverage its cache and will do the whole bundling).

If possible, a much better flow would be to integrate the clojurescript compiler with metro, so metro would call the compiler on a per-file basis and you will benefit from the speed/parallelization/caching of metro.

I'm not familiar with clojurescript and its compiler, so I don't know if it can even work on a per-file basis, But if it can the integration should not be super-complex:

1. You need to create you own transformer that transforms a clojurescript file to JS (more specifically, to AST). To get an example, this is our basic transformer
2. You'll need to configure metro to use the new transformer that you've created. There is the getTransformModulePath option for that (example).

Let me know if this helps

[jeaye]

@rafeca Thanks for the detailed reply. The ignoring approach was mentioned in the first referenced issue here, but it leads to other issues. When I try it in my production build, rather than my test case repository, I run into issues because my JS actually needs to be transformed. That is, I have some NPM dependencies using newer JS features which need to be transformed for RN to consume them.

The failure looks something like this:

...
:app:bundleReleaseJsAndAssets
Scanning folders for symlinks in my-app/node_modules (4ms)
Scanning folders for symlinks in my-app/node_modules (21ms)
Loading dependency graph, done.
warning: the transform cache was reset.

Export statement may only appear at top level in file "my-app/node_modules/react-native-material-ui/src/RippleFeedback/index.android.js" at line 1:114

:app:bundleReleaseJsAndAssets FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:bundleReleaseJsAndAssets'.
> Process 'command 'node'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

[rafeca]

@jeaye it seems that there're other files in dependencies called index.android.js, so if you set the ignore param to this, it'll ignore all of them which is not correct.

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

[jeaye]

@rafeca That's a very good idea; the babel docs don't mention anything of the sort, as far as limiting matches to the top-level directory. I've tried various patterns, from ^index.android.js and ./index.android.js to /index.android.js. Are you aware of how to accomplish limiting the ignoring to only the top-level index.android.js?

My research into how babel looks this up seems like any JS file within node_modules will look for a .babelrc and will continue upward toward / in search of one. Should a catch-all for node_modules be placed at node_modules/.babelrc then, which doesn't have the ignore?

[HikingDev]

Why is this issue closed? I have a json file with 17.4MB.
If i require it the bundler crashes and if comment it out everything works fine.
This seems to be a common problem. Is there a solution for this issue?

"FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory"

[jeaye]

@roesneb It certainly shouldn't be closed. It was ignored when it was reported, multiple times, in the react native repo, while people suggested it be brought up here. Now that it's brought up here, it's promptly closed and no workarounds are available (for people using ClojureScript).

For your problem, you might try ignoring that JSON file using your babelrc. Alas, if it has a common name, then all JSONs with that name will be ignored, which is the last problem about which I asked @rafeca.

[HikingDev]

@jeaye i've tried the ignoring option already, without success. I could split up my JSON in multiple files but I don't think that should be the way to do it.

[VincentJousse]

Any news on this "not solved" issue ?

[jeaye]

@VincentFTS If this is possible:

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

Then the problem can be resolved for ClojureScript users. No documentation I've found has shown it to be possible though. Instead, renaming your app's index.android.js to myapp.index.android.js or something, may do the trick.

[VincentJousse]

The problem is that the "big file" is a javascript file that uses flow, it permits to check the format of a big table.
Hence I cannot add it to the ignore section of the babelrc file.

[ghost]

Hello @jeaye

Did you end up solving this issue for CLJS? Our iOS build is functioning fine, but our android build not so well. On a previous version of react native, We did all sorts of monkey patching to make the transformers not run for our ClojureScript files it was horrible, but it worked. After upgrading to the latest RN we were hoping to avoid the money patching if possible.

[jeaye]

@ChadHarrisVerr Nope, this is still very much an issue. metro doesn't handle large files well and has a non-linear time growth with respect to file size. So far, there's been no official way to speed things up and this ticket, just like many others here and in the react-native repo, has been closed with little help (read the full discussion here to understand more, if you haven't).

I have documented my current workaround here: https://github.com/cljsrn/cljsrn-org/wiki/Avoiding-Metro-timeouts In short, I patch metro to not give up after several minutes of transforming and I just wait a long time for my prod builds. As a result of this, I also can't reasonably use something like jest for testing, since it wants a bundled package as well.

[sundbry]

In order to fix this, you'll need to use a custom Metro transformer to skip loading the entire compiled Clojurescript AST, and also load its dependencies properly. I've documented how to do it here:

https://www.arctype.co/blog/fixing-react-native-compiler-clojurescript-timeouts

cc @bonlemuel @ChadHarrisVerr

[boogie666]

Hi @sundbry,

The article from arctype is great and all :P but i can't get it to work...

i keep getting TypeError: Cannot read property 'transformFile' of undefined
in /node_modules/metro/src/Bundler.js:77:34

Can you help with that?

(btw i'm trying to solve a different problem, where the bundler is eating some functions from clojurescript core causing a wierd error, and no, it's not an externs thing :P)

[sundbry]

@boogie666 Metro changed the interface here in v0.47.0:

f8cfe20

Try reverting your metro to 0.45.4

[boogie666]

@sundbry Lifesaver :* thx a lot :*

[sundbry]

FYI including @boogie666 the RN cljs transformer has been updated to work with the new interfaces in current metro 0.49 : https://gist.github.com/sundbry/55bb902b66a39c0ff83629d9a8015ca4

[familyfriendlymikey]

The ignore param doesn't seem to be working. I only need to ignore one file in the app's root dir, yet the transform will still hang when node crashes at 1.43GB of memory usage. If I remove the JSON file, the app builds for release config just fine, so I know even though I'm ignoring the file, babel is still parsing it.

babel.config.js

module.exports = {
  presets: ["module:metro-react-native-babel-preset"],
  ignore: [
    "filename.json"
  ]
}

Any ideas? Can't seem to find a solution anywhere. Nothing has had an impact, including attempting to change node's max_old_space_size.

[leasontou]

The ignore param doesn't seem to be working. I only need to ignore one file in the app's root dir, yet the transform will still hang when node crashes at 1.43GB of memory usage. If I remove the JSON file, the app builds for release config just fine, so I know even though I'm ignoring the file, babel is still parsing it.

babel.config.js

module.exports = {
  presets: ["module:metro-react-native-babel-preset"],
  ignore: [
    "filename.json"
  ]
}

Any ideas? Can't seem to find a solution anywhere. Nothing has had an impact, including attempting to change node's max_old_space_size.

I have this problem too, how did you solve it?

[familyfriendlymikey]

Never really did, I ended up having to just download the whole database from within the app on first startup and show a loading screen.

Either that or depending on your use case you might be able to make the database smaller by changing long keys to something shorter. That got me under the threshold of size that would actually build, pretty big difference. I changed every key to be one character long.

Hopefully this gets fixed some time, wishing you luck.

[leasontou]

Never really did, I ended up having to just download the whole database from within the app on first startup and show a loading screen.

Either that or depending on your use case you might be able to make the database smaller by changing long keys to something shorter. That got me under the threshold of size that would actually build, pretty big difference. I changed every key to be one character long.

Hopefully this gets fixed some time, wishing you luck.

Thank you, you are a genius, but it doesn't suit my situation.😟

[leasontou]

I have solved it.
There are some large js files in my project, I increase the memory limit for Node.js, and it works for me.

Add following content in ~/.bash_profile

export NODE_OPTIONS=--max-old-space-size=4096

I'm not sure it can suit you, but maybe it could help someone else.