"Cannot perform Construct on a detached ArrayBuffer" error when using msc_basis_transcoder

[Popov72]

One of our user has occasional problems when loading a .glb file containing .ktx2 pictures:

https://playground.babylonjs.com/#CF7E5Q#17

Open the browser console to see the exception. Sometimes it works, so just hit F5 or the Play button to start it again until it fails. On my computer using Chrome, it fails fairly quickly, either the first time I browse the page or after 2 or 3 F5 refreshes. Note that you won't see the full stack traces I dumped below because I used locally a modified version of our ktx2Decoder component to get the stack traces (I will update the production site so that we can see the full traces in the Playground too).

I tried to dig a bit and the errors are thrown from inside the msc_basis_transcoder component, either from the transcodeImage or decodePalettes call:

(transcodeImage):

TypeError: Cannot perform Construct on a detached ArrayBuffer
    at new Uint8Array (<anonymous>)
    at emval_allocator_3 (eval at craftEmvalAllocator (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:13:11)
    at __emval_new (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:8:58552)
    at <anonymous>:wasm-function[246]:0x1d2c0
    at <anonymous>:wasm-function[488]:0x3585e
    at ha (<anonymous>:wasm-function[296]:0x2cc1e)
    at Module.dynCall_iiiiiiii (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:8:63561)
    at dynCall_iiiiiiii_1 (eval at makeDynCaller (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:4:12)
    at BasisLzEtc1sImageTranscoder$transcodeImage [as transcodeImage] (eval at new_ (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:13:10)
    at http://localhost:1338/dist/preview%20release/babylon.ktx2Decoder.js:1:16911

or (decodePalettes):

TypeError: Cannot perform Construct on a detached ArrayBuffer
    at new Uint8Array (<anonymous>)
    at emval_allocator_3 (eval at craftEmvalAllocator (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:13:11)
    at __emval_new (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:8:58552)
    at <anonymous>:wasm-function[261]:0x2a776
    at <anonymous>:wasm-function[503]:0x36857
    at ga (<anonymous>:wasm-function[297]:0x2cc39)
    at Module.dynCall_iiiiiii (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:8:63452)
    at dynCall_iiiiiii_1 (eval at makeDynCaller (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:4:12)
    at BasisLzEtc1sImageTranscoder$decodePalettes [as decodePalettes] (eval at new_ (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:12:10)
    at http://localhost:1338/dist/preview%20release/babylon.ktx2Decoder.js:1:16615

Note that those stack traces have been produced using the latest version of the msc component (https://github.com/KhronosGroup/KTX-Software/releases/tag/v4.0.0-beta6). The version used in production in the Playground above is a little older but the exceptions we get are the same.

Also, in the playground above I have set the number of worker threads to 1 to rule out a possible problem coming from multiple threads instancing the msc component.

So, maybe we are not doing the right things when using the component(?), even if I think the implementation is quite straightforward:
https://github.com/BabylonJS/Babylon.js/blob/master/ktx2Decoder/src/Transcoders/mscTranscoder.ts

Note that the .glb file is using only the ETC1S path.

Could you have a look on your side and tell us what you think? One thing that would help I think would be to know what is this buffer that is detached and not valid anymore: is it a buffer from our side (I don't think so, though, as sometimes the error comes from a call to decodePalettes and we don't pass any buffer to this function)?

Thanks in advance!

[MarkCallow]

@bghgary you have stated in another forum that this problem is "due to the browser cache. Can you try running it in a incognito (in-private) browser window or clear the browser cache?". People have reported success after doing so. Did you discover this after you thumbed-up this issue or is there still something I need to understand and fix?

[Popov72]

Update:

It seems I can't make it fail anymore with the first PG I have provided (where size of worker pool = 1). What changed is that we now use the latest version of the msc transcoder.

However, if using a pool of worker threads > 1, it's still failing sometimes. Here's the corresponding PG:

https://playground.babylonjs.com/#CF7E5Q#18

Even in incognito mode, the PG above fails sometimes.

[bghgary]

@bghgary you have stated in another forum that this problem is "due to the browser cache. Can you try running it in a incognito (in-private) browser window or clear the browser cache?". People have reported success after doing so. Did you discover this after you thumbed-up this issue or is there still something I need to understand and fix?

I was telling people this because we updated the KTX decoder and the way we set up our CDN, it doesn't automatically update immediately. I ran into an issue also and clearing the cache also "fixed" it. It could be that clearing the cache didn't actually do anything and we just got lucky that it worked afterwards. I haven't investigated specifically the call stack for this failure.

[MarkCallow]

Please try just released Release Candidate 1. The web modules in this release have been built with the latest emsdk (2.0.16).

[Popov72]

Still the same error:

TypeError: Cannot perform Construct on a detached ArrayBuffer
    at new Uint8Array (<anonymous>)
    at emval_allocator_3 (eval at craftEmvalAllocator (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:13:11)
    at __emval_new (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:9:57290)
    at <anonymous>:wasm-function[182]:0x24a6a
    at <anonymous>:wasm-function[176]:0x245d6
    at BasisLzEtc1sImageTranscoder$transcodeImage [as transcodeImage] (eval at new_ (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:13:10)
    at http://localhost:1338/dist/preview%20release/babylon.ktx2Decoder.js:1:17031
    at async Promise.all (index 0)

or:

TypeError: Cannot perform Construct on a detached ArrayBuffer
    at new Uint8Array (<anonymous>)
    at emval_allocator_3 (eval at craftEmvalAllocator (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:13:11)
    at __emval_new (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:9:57290)
    at <anonymous>:wasm-function[207]:0x25c95
    at <anonymous>:wasm-function[201]:0x2587e
    at BasisLzEtc1sImageTranscoder$decodePalettes [as decodePalettes] (eval at new_ (http://localhost:1338/dist/preview%20release/ktx2Transcoders/msc_basis_transcoder.js:1:1), <anonymous>:12:10)
    at http://localhost:1338/dist/preview%20release/babylon.ktx2Decoder.js:1:16735
    at async Promise.all (index 5)

[MarkCallow]

Thanks. I'll try to investigate but my knowledge in this area is limited. Any help you can offer will be very welcome.

[MarkCallow]

Some questions:

1. How can I replace your version of msc_basis_transcoder in the playground with one I'm testing?
2. How often does the error happen, e.g. 1 in 5 times?
3. I'm seeing a TypeError from ServiceWorker script but I can't find a way to get more details of the error. Is this the error you are talking about?

I think I have not yet managed to reproduce the error.

Both of the failing functions have code like the following, but so does decode_tables which isn't failing, to create a view into the memory allocated on the Emscripten heap for the incoming slices (in the case of transcodeImage) or the incoming endpoints and selectors (in the case of decode_palettes):

            val memory = val::module_property("HEAP8")["buffer"]; // The Emscripten heap
            std::vector <uint8_t> deflatedSlices;
            uint32_t deflatedSlicesByteLength
                                     = jsInSlices["byteLength"].as<uint32_t>();
            deflatedSlices.resize(deflatedSlicesByteLength);
            val memoryView = jsInSlices["constructor"].new_(memory,
                             reinterpret_cast<uintptr_t>(deflatedSlices.data()),
                             deflatedSlices.size());
            memoryView.call<void>("set", jsInSlices);

val is an emval. This is the only emval allocation I can see. I do not, as yet, know what detached ArrayBuffer means but the functions are making these views on the ArrayBuffer you pass in with the slices, endpoints and selectors. Maybe these are becoming "detached" at your end before you pass the handles to the transcoder.

[lexaknyazev]

ArrayBuffer becomes "detached" when the underlying WebAssembly.Memory object grows. Here's a simple JS-only demo:

let m = new WebAssembly.Memory({initial: 1});
let b = m.buffer;
m.grow(1);
b == m.buffer; // false
new Uint8Array(b) // TypeError: Cannot perform Construct on a detached ArrayBuffer

[lexaknyazev]

In the C++ code above, it may be possible that deflatedSlices.resize(deflatedSlicesByteLength) causes the heap to grow. To fix the issue, the memory reference should be refreshed.

[MarkCallow]

Thanks @lexaknyazev. @Popov72 please try moving the line

            val memory = val::module_property("HEAP8")["buffer"]; // The Emscripten heap

and the equivalent in decodePalettes and decodeTables to after the resize and see if that fixes your problem.

[MarkCallow]

@Popov72 I've made a new branch, issue371, with a potential fix. Please give it a try. You can now easily build the web modules using Docker. See the instructions in BUILDING.md.

[Popov72]

@MarkCallow Sorry, I haven't been able to generate the files myself (on Windows). docker is not installed in my virtual ubuntu:

• if I update the build_wasm_docker.sh file to use docker.exe instead of docker so that my Windows installation of docker is used, I get this output:
  
• if I install docker on the ubuntu system (following https://docs.docker.com/engine/install/ubuntu/), I get:
  
  Even if I run first: sudo service docker start

Would you mind generating the two files for me so that I can test? I'm not sure I will be able to generate those files in a timely fashion (meaning in the remaining of my lifetime, as each time I try to build something from sources it fails miserably).

Thanks!

[MarkCallow]

Here are the files packaged in a .zip to keep GitHub happy. Ignore the rc1 in the file name. This is the latest with the potential fix.

KTX-Software-4.0.0-rc1-Web-msc_basis_transcoder.zip

The first error you got is either because the line

  docker run -dit --name emscripten -v $(pwd):/src emscripten/emsdk bash

in the script didn't work, i.e. $(pwd) returned something strange, or because your KTX-Software directory is not populated, i.e. does not have a CMakeLists.txt. The latter seems unlikely as you have the script in this workarea. I'm surprised you have to change docker to docker.exe. I thought the various unix-like shells on Windows would automatically try foo.exe when not finding foo. What shell were you using?

I know nothing about docker on Ubuntu so I can't help with that error.

[Popov72]

What shell were you using?

I'm using the ubuntu package from the Microsoft Store. CMakeLists.txt is in the root directoy of the KTX-Software directory.

Anyway, thanks for the files, it seems to have fixed the bug!

With the old files, I had 3 crashes over 16 runs, with the new files no crash over 20 runs.

Can I use those files to replace the old ones in the Babylon distrib or are you going to create a new official release with the fix?

[MarkCallow]

I'm glad the issue is fixed. I'll be releasing v4.0.0 with this in a few minutes. I suggest you use the files from that release.