How to prevent JavaScript freezing in readStream

[stefan-feltmann]

I'm having an issue streaming in a file with readStream. While onData is being called the JS thread freezes for about nine seconds opening a 16.6 MB file. I've tried increasing the buffer to 20000000 from default and that did speed it up from its old time of about forty five seconds, but further increase didn't change anything and if the buffer is set too high, the app crashes. Is there something straightforward I'm missing, or is this expected behavior?

[wkh237]

@stefan-feltmann , thank you for point out this problem. Could you let me know which encoding you use (utf8, base64, ascii)? Does this happen on both Android and IOS ? I'd like to know more information, thank you.

[wkh237]

@stefan-feltmann , after some investigation, I think the problem is result from high frequency event dispatching, however, thank to this issue, I've also found a bug in IOS readStream implementation and have it fixed.

I've changed the code and try to reduce the overhead when handling RCT bridge event, though the problem may still exists, I'm able to read a 22MB file in about 3 seconds (on iPhone 6) by increase the buffer size to 102400(25x of default value). Please try the latest version 0.9.4-beta.4 and leave any feedback, I'll keep look into this issue, thank you 👍

[stefan-feltmann]

I'm using utf8. Currently it's only happening on Android. iOS is having issues which may or may not be related to readStream. More testing is required on our side.

I will try 0.9.4-beta.4 and leave feedback as soon as I am able to.

[stefan-feltmann]

I tried both 0.9.4-beta.4 and 0.9.4-beta.5 on Android and the problem still persists.

I'll get back to you with iOS information as soon as I can.

[wkh237]

@stefan-feltmann , great thanks for the assistance 👍 I've published 0.9.4-beta.7 which contains the following changes :

On IOS when the buffer size is large (about ~1000k) the app crashes due to IOS thread stack size limit (1MB). In this version the stream buffer is moved to heap instead of in heap, so that it's managed to use a large buffer and it's working seemly on iPhone.

As for Android, I've found a possible solution, we were using AsyncTask instead of thread pool when read data using filestream (most of fs and network API are using thread pool since 0.8.0, perhaps I've just missed readStream), in this version fs.readStream is moved to thread pool, I hope it helps ! I'm testing on Google Nexus 5.0 and it's able to show progress on GUI when reading a large file with 1MB buffer (bufferSize = 1024000).

[wkh237]

FYI, the latest release 0.9.4 I've added a new argument interval so that you can limit the read stream event frequency. Here's an example :

fs
// read file with 400kb buffer and with an interval 100ms
.readStream(PATH_TO_FILE, 'utf8', 409600, 100)
.then((stream) => {
  // open file stream
  stream.open()
  stream.onData((chunk) => { /* do something with the chunk */ })
  stream.onEnd(() => { /*  do something */ })
})

Please try if this solve the problem 😄

[tsemerad]

I tried 0.9.4 using a 100 ms interval, and my iOS app is now pretty responsive while the download is in progress. This is definitely a major improvement! Thank you 😄

One other thing that you might want to consider. I came across this article about breaking up heavy processing in React Native. I'm not sure if it's possible with this library, but if it's possible to load each chunk immediately after each UI repaint, maybe that will elegantly solve the freezing problem.

But like I said, the new interval parameter is great so far, and probably good enough. Thanks again!

[wkh237]

@tsemerad , great to hear that 😄

From my understanding, the root cause of this issue is that native module dispatch event in a very high frequency, as such there is a large overhead converting native object to JS context which makes the JS VM very busy. Therefore I tried to reduce the frequency by adding extra option to the API.

However I've never think about requestAnimationFrame could be a solution, thanks for the advice and the reference, I'll look into it and try if it's possible. Also, you're always welcomed to make PR 😏

[stefan-feltmann]

0.9.4 seems to have solved the issue. It took some configuring, but after setting it to use big chunks (20000000) and longer intervals (10000), everything is fine and it isn't freezing anymore.

Thanks a lot for looking into this.