Uncaught RuntimeError: memory access out of bounds

[felipecrs]

I'm aware of #257 and #189 (comment), but I'm opening this issue because I started to get this very frequently after upgrading to v3.0.0-rc.1, so I wonder if something has changed or anything like that.

Here is the stack trace:

Uncaught RuntimeError: memory access out of bounds
    at wasm://wasm/00025b02
    at wasm://wasm/00025b02
    at wasm://wasm/00025b02
    at MPEG1WASM.initializeWasmDecoder (frigate-hass-card.js?hacstag=3940825523001:10893:35)
    at MPEG1WASM.bufferWrite (frigate-hass-card.js?hacstag=3940825523001:10923:12)
    at MPEG1WASM.write (frigate-hass-card.js?hacstag=3940825523001:9225:31)
    at MPEG1WASM.write (frigate-hass-card.js?hacstag=3940825523001:10945:33)
    at TS.packetComplete (frigate-hass-card.js?hacstag=3940825523001:9155:20)
    at TS.parsePacket (frigate-hass-card.js?hacstag=3940825523001:9095:18)
    at TS.write (frigate-hass-card.js?hacstag=3940825523001:8987:44)

And the spinner just keeps spinning:

If I reload the browser a couple of times, the issue is gone.

Also, I guess it's important to mention that such problem never happens with a single camera. If I have more than one camera in the dashboard displaying at the same time, either all of them load correctly, or they all don't.

If nothing has changed and I'm just unlucky, I believe it would be nice if the card could catch the issue from JSMPEG, and display a more helpful message like the JSMPEG has reported an error instead of just keep spinning forever, which gives the impression that it's still loading.

EDIT: I just realized that it's not feasible to catch and treat the error, by reading through the second mentioned issue.

[felipecrs]

I found this:

phoboslab/jsmpeg#268 (comment)

I didn't find in Frigate what are the arguments used to encode the JSMPEG stream. Any hints?

[felipecrs]

Also, if it turns out to be indeed an issue in upstream, it would be nice if we have one open there to track the progress, or at least to have it reported. I'm not sure if the existing ones, like the mentioned above, provides enough information for the upstream project to handle.

[felipecrs]

I must say that I'm receiving this much less frequently since I upgrade Frigate to 0.10.0.

Actually I don't recall if I ever had this problem since.

I will monitor for some days, and shall return here.

[felipecrs]

I was able to catch it again.

Test.mp4

And I noticed that:

live:
  jsmpeg:
    options:
      disableWebAssembly: true

Is indeed effective, although it at least triples the initial loading time.

[felipecrs]

Probably nothing new arised in this matter, therefore I'm closing this issue as duplicate of #189, given the conclusion at #189 (comment).

[felipecrs]

@dermotduffy this never happens when I'm browsing my cameras through Frigate itself. And it happens so frequently when I'm using the card:

YJ3wKEqxTC.mp4

Also, tweaking the ffmpeg feed resolution and quality does not seem to help either, as mentioned in blakeblackshear/frigate#2911 (comment).

I really do not want to bother you about this issue, but if you have any idea/suggestion/anything it would be super appreciated.

I'm starting to think as well that this may come from the core HA frontend. Some kind of incompatibility or conflict during the frontend initialization or so.

[felipecrs]

@dermotduffy I was able to catch the error at:

https://github.com/cycjimmy/jsmpeg-player/blob/1d04cd5b1589e7481207ca4c45d4a39eddbd673c/src/lib/decoder.js#L45

But this is not the card's code, it comes from the jsmpeg player. Do you think there is anything we can do? Like patching the jsmpeg to add this new try catch and then somehow being able to propagate to frigate-hass-card in a way that it could do something to remedy (like destroying the jsmpeg player and recreating it)? Or at least to show a message: jsmpeg player failed to load.

[dermotduffy]

this never happens when I'm browsing my cameras through Frigate itself.

Huh. I have no idea why this would be, they are both using the same player. Are you interacting with them in any different way? (e.g. leaving Lovelace open for hours, but only opening the Frigate UI occasionally?).

Is this happening even with the tricks mentioned earlier in this bug (from here).

But this is not the card's code, it comes from the jsmpeg player. Do you think there is anything we can do?

Frankly, in its current state of maintenance, I am not confident at getting fixes from the JSMPEG player itself ... have you tried a different live provider? (e.g. WebRTC)

Also interesting is that you are getting it immediately upon loading, so even catching the errors and reloading the card somehow would not help.

[blakeblackshear]

I wonder if its the websocket proxy in the integration

[felipecrs]

Are you interacting with them in any different way? (e.g. leaving Lovelace open for hours, but only opening the Frigate UI occasionally?).

I actually should have said that I never observed this problem in Frigate UI. Indeed I browse Frigate UI only occasionally. But I am not doing anything special in lovelace either.

As you can see in the recorded video, the issue happens when I open lovelace (so definitely I'm not leaving it open for hours).

Is this happening even with the tricks mentioned earlier in this bug (from here).

Decreasing or increasing videoBufferSize makes no difference at all. Disabling Web Assembly solves the issue, though. Although it triples the loading time in my browser and on my mobile it gets even slower (not feasible).

WebRTC is nice when it works, but for me, it works 3/10 times (lol just like jsmpeg these days).

I had some good times using WebRTC native, but I haven't considered using it since Frigate does not support it natively (which means I would need to setup direct connections to my cameras). But, I'll give it another go, thanks for reminding me this option.

Frankly, in its current state of maintenance, I am not confident at getting fixes from the JSMPEG player itself

I also share this feeling, unfortunately. But that also means that applying patches from our side are less risky as the upstream code is unlikely to change.

I will take a look in that: perhaps I find a way to at least prevent the log flooding.

I spent some time trying to debug this in low level, but without any success, apart from what I reported in cycjimmy/jsmpeg-player#27 (comment).

Also interesting is that you are getting it immediately upon loading, so even catching the errors and reloading the card somehow would not help.

Yeah. Immediately after loading... Never after (if stream connects, it stays hours connected without any errors).

I wonder if its the websocket proxy in the integration

@blakeblackshear That would explain why it happens in lovelace but not in Frigate UI. I don't receive any error related to websockets in my browser, but I'll check the integration/ha core logs. Maybe there is something there.

Anyway: thank you both very much!

[dermotduffy]

I wonder if its the websocket proxy in the integration

Yeah, that's certainly another difference, but it's not really clear how that would cause memory errors on load for one/some users but not for everyone else.

The options to the player are also not exactly the same, although the differences appear inconsequential.

card:

        {
          // The media carousel automatically pauses when the browser tab is
          // inactive, JSMPEG does not need to also do so independently.
          pauseWhenHidden: false,
          protocols: [],
          audio: false,
          videoBufferSize: 1024 * 1024 * 4,

          // Override with user-specified options.
          ...this.jsmpegConfig?.options,

          // Don't allow the player to internally reconnect, as it may re-use a
          // URL with a (newly) invalid signature, e.g. during a Home Assistant
          // restart.
          reconnectInterval: 0,
          onVideoDecode: () => {
            // This is the only callback that is called after the dimensions
            // are available. It's called on every frame decode, so just
            // ignore any subsequent calls.
            if (!videoDecoded && this._jsmpegCanvasElement) {
              videoDecoded = true;
              dispatchMediaShowEvent(this, this._jsmpegCanvasElement);
              resolve(player);
            }
          },
        },

vs

{ protocols: [],
  audio: false,
  videoBufferSize: 1024*1024*4
}

[dermotduffy]

Disabling Web Assembly solves the issue, though.

Again, suggests an issue with the internals of the player. If it was the websocket proxy, or the options, I wouldn't expect that to be the case -- but I cannot explain why it happens more in Lovelace vs Frigate UI.

[felipecrs]

There are no errors in Core except for the spamming of:

But I noticed one behavior in the Frigate logs:

• I have three cards in my lovelace which displays at the same time, when it connects fine, I receive three logs of this kind:

[2022-03-12 12:08:45] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59966]
[2022-03-12 12:08:45] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59972]
[2022-03-12 12:08:45] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59980]

And then, upon closing the window:

[2022-03-12 12:09:35] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59966]
[2022-03-12 12:09:35] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59980]
[2022-03-12 12:09:35] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59972]

All good. Always three times, when it works.

But this is what happens when it does not work:

[2022-03-12 12:09:41] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60294]
[2022-03-12 12:09:41] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60300]
[2022-03-12 12:09:41] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60306]
[2022-03-12 12:09:41] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60312]
[2022-03-12 12:09:42] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60318]
[2022-03-12 12:09:42] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60324]
[2022-03-12 12:09:42] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60330]
[2022-03-12 12:09:42] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60336]
[2022-03-12 12:09:42] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:60342]

And upon closing, all of them get closed as well. The point is, why so many? If you check the timestamp, they are all at the same time. And the number of WebSocket connections in such case isn't fixed, sometimes more than times, sometimes less. But definitely different than when it works.

PS: This issue happens 99% of the times when I first open lovelace (as I showed in the video demo earlier). From within lovelace, if I refresh the screen, 99% of the times it works.

And I am not considering this for real, but as last resource, when the card detects such a situation, there would be the option to force the window to be refreshed, from within the card.

[felipecrs]

The workaround to reload the page is easy, ugly, but effective (and I mean, it can still be improved obviously):

try {
  this.bytesWritten += this.bufferWrite(buffers);
} catch (error) {
  console.warn('Detected OOB error, reloading window...')
  document.location.reload()
}

chrome_4nuF86mpHS.mp4

[dermotduffy]

The point is, why so many?

I'd guess because the player is running out of memory, then re-trying an operation, which causes another reconnection (?). But I think we'd need to dig into the player code to fully understand why it's opening so many connections in the error case.

And I am not considering this for real, but as last resource, when the card detects such a situation, there would be the option to force the window to be refreshed, from within the card.

Lovelace dashboards are a patchwork of different components written by different people -- a single card refreshing the whole document (i.e. blowing up all components) is probably not going to fly. If we could catch this error 'card-side' and just reset the player, that'd be much better (e.g. does calling FrigateCardLiveJSMPEG._refreshPlayer() work?)