Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state #8250

Open
luke-rogers opened this issue May 16, 2024 · 40 comments
Open

FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state #8250

luke-rogers opened this issue May 16, 2024 · 40 comments

Comments

@luke-rogers
Copy link

luke-rogers commented May 16, 2024

Operating System

iOS 17.4.1

Browser Version

Mobile Safari UI/WKWebView (AppleWebKit/605.1.15)

Firebase SDK Version

10.12.0

Firebase SDK Product:

Firestore

Describe your project's tooling

Angular web app running on iOS via Capacitor
Firestore offline persistence enabled with unlimited cache size

Describe the problem

For a long time now we have been plagued by this iOS-only issue and have received many reports from users that it's causing the app to stop working and forcing them to restart.

We have captured some Firebase debug logs of this error which I am happy to share privately.

This seems similar to the following issues:

Example Errors

  • UnknownError: Attempt to get a record from database without an in-progress transaction
  • UnknownError: Attempt to get all index records from database without an in-progress transaction
  • UnknownError: Connection to Indexed Database server lost. Refresh the page to try again
  • FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state

Example Stack Trace

Error: FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state
  at fail(node_modules/@firebase/firestore/dist/index.esm2017.js:197:36)
  at __PRIVATE_AsyncQueueImpl.Pu(node_modules/@firebase/firestore/dist/index.esm2017.js:18775:16)
  at __PRIVATE_AsyncQueueImpl.enqueue(node_modules/@firebase/firestore/dist/index.esm2017.js:18691:14)
  at __PRIVATE_AsyncQueueImpl.enqueueAndForget(node_modules/@firebase/firestore/dist/index.esm2017.js:18676:10)
  at n(node_modules/@firebase/firestore/dist/index.esm2017.js:14564:15)
  at <anonymous>(node_modules/@firebase/firestore/dist/index.esm2017.js:14540:7)
  at __PRIVATE_StreamBridge.yo(node_modules/@firebase/firestore/dist/index.esm2017.js:13897:10)
  at n(node_modules/@firebase/firestore/dist/index.esm2017.js:14145:81)
  at <anonymous>(node_modules/@firebase/firestore/dist/index.esm2017.js:14094:13)
  at ab(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:548:22)
  at F(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:516:9)
  at Z.prototype.ta(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:2162:5)
  at Rb(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1035:129)
  at M.prototype.Y(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:904:50)
  at M.prototype.ca(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:832:35)
  at ab(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:548:22)
  at F(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:516:9)
  at Wc(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1582:128)
  at h.bb(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1579:5)
  at h.Ea(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1576:61)
  at Lc(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1474:50)
  at h.Pa(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1440:27)
  at {_ZoneImpl#2}.run(node_modules/zone.js/fesm2015/zone.js:111:37)
  at <anonymous>(node_modules/zone.js/fesm2015/zone.js:2299:30)
  at {_ZoneImpl#2}.runTask(node_modules/zone.js/fesm2015/zone.js:157:39)
  at drainMicroTaskQueue(node_modules/zone.js/fesm2015/zone.js:529:23)

Steps and code to reproduce issue

Steps currently unknown but possibly related to the app going between the background and the foreground over time.

@luke-rogers luke-rogers added new A new issue that hasn't be categoirzed as question, bug or feature request question labels May 16, 2024
@jbalidiong jbalidiong added needs-attention and removed new A new issue that hasn't be categoirzed as question, bug or feature request labels May 16, 2024
@wu-hui wu-hui self-assigned this May 16, 2024
@luke-rogers
Copy link
Author

Looking into the Firebase logs, the error seems to occur after updateClientMetadataAndTryBecomePrimary is called.

Our app uses persistentSingleTabManager(undefined) for the tabManager config in the Capacitor environments as there can never be more than 1 tab. I assume this tab manager is the cause of the many PUT clientMetadata and SimpleDb GET owner owner logs.

I wonder if there would be any legs in not having a tab manager if that will result in no calls to updateClientMetadataAndTryBecomePrimary etc.

This is also similar to what is described by another GitHub user here

I don't know a lot about the internals of the tab management so might be completely wrong here.

Sample Log

[2024-05-16T13:03:08.238Z] Firestore (10.12.0): SimpleDb PUT clientMetadata <auto-key> {"clientId":"jtQjaM2I0v6CbepACBf7","updateTimeMs":1715864588238,"networkEnabled":true,"inForeground":false}
[2024-05-16T13:03:08.242Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864584009}
[2024-05-16T13:03:08.244Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864584009}
[2024-05-16T13:03:08.244Z] Firestore (10.12.0): IndexedDbPersistence Client 'jtQjaM2I0v6CbepACBf7' is not zombied in LocalStorage
[2024-05-16T13:03:08.244Z] Firestore (10.12.0): SimpleDb PUT owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864588244}
[2024-05-16T13:03:25.878Z] Firestore (10.12.0): IndexedDbPersistence Starting transaction: updateClientMetadataAndTryBecomePrimary
[2024-05-16T13:03:48.787Z] Firestore (10.12.0): SimpleDb Aborting transaction: Attempt to get a record from database without an in-progress transaction
[2024-05-16T13:03:48.787Z] Firestore (10.12.0): SimpleDb Transaction failed with error: Attempt to get a record from database without an in-progress transaction Retrying: true
[2024-05-16T13:03:48.787Z] Firestore (10.12.0): SimpleDb Opening database: firestore/[DEFAULT]/insite-teams/main
[2024-05-16T13:03:48.802Z] Firestore (10.12.0): WebChannelConnection RPC 'Listen' stream 0x7d7f4c65 received: {"targetChange":{"resumeToken":"CgkIic7l5p2ShgM=","readTime":"2024-05-16T13:03:34.823689Z"}}
[2024-05-16T13:03:56.239Z] Firestore (10.12.0): SimpleDb Aborting transaction: Attempt to get a record from database without an in-progress transaction
[2024-05-16T13:03:56.239Z] Firestore (10.12.0): SimpleDb Transaction failed with error: Attempt to get a record from database without an in-progress transaction Retrying: true
[2024-05-16T13:03:56.239Z] Firestore (10.12.0): SimpleDb Opening database: firestore/[DEFAULT]/insite-teams/main
[2024-05-16T13:03:56.243Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864588244}
[2024-05-16T13:03:56.243Z] Firestore (10.12.0): SimpleDb PUT clientMetadata <auto-key> {"clientId":"jtQjaM2I0v6CbepACBf7","updateTimeMs":1715864636243,"networkEnabled":true,"inForeground":false}
[2024-05-16T13:03:56.244Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864588244}
[2024-05-16T13:03:56.245Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864588244}
[2024-05-16T13:04:02.786Z] Firestore (10.12.0): SimpleDb Aborting transaction: Attempt to get records from database without an in-progress transaction
[2024-05-16T13:04:02.786Z] Firestore (10.12.0): SimpleDb Transaction failed with error: Attempt to get records from database without an in-progress transaction Retrying: false
[2024-05-16T13:04:02.788Z] Firestore (10.12.0): INTERNAL UNHANDLED ERROR:  Attempt to get records from database without an in-progress transaction
[2024-05-16T13:04:19.979Z] Firestore (10.12.0): WebChannelConnection RPC 'Listen' stream 0x7d7f4c65 received: {"targetChange":{"resumeToken":"CgkIrpqg/J2ShgM=","readTime":"2024-05-16T13:04:19.823918Z"}}
[2024-05-16T13:04:19.989Z] Firestore (10.12.0): FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state

@wu-hui I would be Happy to share the full log with you privately if you could let me know the best way to do that.

@vojto
Copy link

vojto commented May 17, 2024

This has gotten really bad for us over the past few days. We didn't deploy anything.

Edit: This is on desktop.

@luke-rogers
Copy link
Author

@vojto Perhaps that's a different issue, this has always been limited to just iOS for us.

@wu-hui
Copy link
Contributor

wu-hui commented May 21, 2024

I think the multitab log might be unrelated, this seems to stem from a underlying webchannal message.

@luke-rogers
Copy link
Author

Fair enough, not an area I've dug too deep into.

@wu-hui what steps can I take to help try and move this forward?

@MarkDuckworth MarkDuckworth self-assigned this May 24, 2024
@luke-rogers
Copy link
Author

Still seeing this issue on the latest version. I have logs I can share if that would be useful?

@MarkDuckworth
Copy link
Contributor

@luke-rogers I would like to review the logs. You can share the logs through a private GitHub repo (shared with me), through a Cloud support ticket, or through a Firebase support ticket. If creating a ticket with either Google support channel, please reference this GitHub issue so the support staff knows to route the ticket to my team. Thanks!

@SamLoys
Copy link

SamLoys commented Jun 10, 2024

I encounter the same problem.
For me this happens in my webapp running in a webview through Flutter.

Usually this is combined with
QuotaExceededError __PRIVATE_WebStorageSharedClientState.setItem(@firebase/firestore/dist/index.esm2017)

Failed to execute 'setItem' on 'Storage': Setting the value of 'firestore_clients_firestore/[DEFAULT]/kekkle/_jLkv2Vnt89NL5Xx3UYtN' exceeded the quota.

The problem is always resolved by loggin out (clearing local storage).
So my first guess is that the quota exception is not handled correcrly

EDIT: I created a new issue, as I am not sure if the underlying problem is the same.

@luke-rogers
Copy link
Author

@MarkDuckworth I've shared the logs through a Firebase support ticket and via a private repo

Let me know if you need anything more from me.

@luke-rogers
Copy link
Author

@MarkDuckworth did you get access to the logs? Support said they forwarded them on.

@MarkDuckworth
Copy link
Contributor

@luke-rogers, I received the logs. Thank you for sharing. On initial review, I don't see an obvious cause, but we will give it some deeper investigation as soon as we can.

I've also started work on a PR to help us get error codes from these internal assertion failed messages.

Do you also have a reproducible example that you can share with us?

@luke-rogers
Copy link
Author

@MarkDuckworth Thanks for the update.

PR looks good. Hopefully, you will be able to merge that. We would be happy to test a pre-release version of this if that would still be helpful?

Unfortunately, we do not have a way of reproducing this ourselves at the minute.

I have added some links to stack traces from Sentry to the readme of private repo I shared with you.

@tom-andersen
Copy link
Contributor

@luke-rogers Mark is out of office and asked I take a look at logs. Could you grant access to private repo, to myself @tom-andersen and maybe some other members of our team such as @wu-hui ? Thanks.

@luke-rogers
Copy link
Author

@tom-andersen added yourself and @wu-hui to the repo.

Still struggling with a reproduction but if there is anything else I can try and help with please let me know.

@tom-andersen
Copy link
Contributor

Update:

The error with text database without an in-progress transaction is from within the IndexedDB implementation. This becomes a fatal error when Firestore settings disable multi-tab support (as far as I can tell from reading code).

The workaround might be to enable multi-tab support, since Firestore SDK will then handle this error by entering a non-primary role until IndexedDB resolves the problem, instead of treating this like a fatal error.

That this happens at all, is a little concerning, and might have more to do with the AppleWebKit.

@tom-andersen
Copy link
Contributor

@luke-rogers Are you able to enable multi-tab and see if that resolves the problem?

@luke-rogers
Copy link
Author

@tom-andersen we will enable multi-tab and report back. Hopefully doing a release today so should have some initial feedback by the end of the week.

@google-oss-bot
Copy link
Contributor

Hey @luke-rogers. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

@luke-rogers
Copy link
Author

We released yesterday with the change to enable multi-tab. We will have more of an update within the next few days as to whether it's made a difference.

@luke-rogers
Copy link
Author

luke-rogers commented Aug 1, 2024

Unfortunately enabling multi-tab has not resolved this issue. I am still trying to access the impact on the end user experience but we have managed to capture some logs for the error. I've added those to the GitHub repo previously shared.

The new log files for this error with Firebase version 10.12.4 and multi-tab enabled are:

  1. 2024-8-1_17-6-47.log
  2. 2024-8-1_8-42-34.log

@luke-rogers
Copy link
Author

Added an additional log file for Firebase version 10.12.5: 2024-8-12_12-53-14.log

@braincomb
Copy link

braincomb commented Aug 20, 2024

Just wanted to chime in and mention that we also started seeing the same error in our Capacitor app, running on Android.

One notable change we did that started showing this error is when we introduced enableNetwork() in an attempt to fix hanging onSnapshot listeners when app is backgrounded. I think we are still seeing that issue, but we sporadically started seeing this error too, although it's very hard to replicate. Still uncertain of the impact it has on the user experience.

Some details:

  • Firebase JS SDK v10.13.0
  • Not running indexedDbPersistence, just the memoryLocalCache which is default I believe

@anderecc
Copy link

Hi, sorry about the English, it's not my mother tongue

I was getting the error “@firebase/firestore: Firestore (10.13.0): FIRESTORE (10.13.0) INTERNAL ASSERTION FAILED: Unexpected state”, when I went to investigate I saw that the cause would be the error ‘ERR_BUFFER_OUT_OF_BOUNDS’, what helped me to solve this error was this article in Stackoverflow, I downgraded nodejs from version 22.7.0 to 22.5.0 and it solved my problem, I hope it helps someone.

@ElianCordoba
Copy link

I my particular case the solution was to downgrade to the version 10.6.0, 10.7.0 is where the issue start. Here is the diff. If I got time I'll try to debug the SDK locally to see if I can figure out what the error is

@braincomb
Copy link

I want to add on an interesting observation, since I've seen a few issues pop up relating to Capacitor which means it's a mobile app. My friend was able to coincidentally reproduce it where he had a very bad internet connection. And he saw our handled Firestore errors, but Sentry reported them as ASSERTION FAILED.

Interesting ....

@braincomb
Copy link

braincomb commented Oct 6, 2024

@luke-rogers @vojto curious if you've been able to fix this or find out the cause of it? These errors are plaguing our Sentry from our Capacitor app.

@shaffy9633
Copy link

@anderecc would it work with nodejs v20.11.0? I see some of my users getting FIRESTORE (10.12.4) INTERNAL ASSERTION FAILED: Unexpected state

@anderecc
Copy link

@shaffy9633, Hello, I believe it will work, I'm using nodejs 22.9.0 and firestore 10.13.0 and not having any problems at the moment related to this 'error', try updating firestore

@TimmNL
Copy link

TimmNL commented Oct 22, 2024

I have a similar issue (#8383) and tried to implement the new version of the sdk released yesterday, 11.0.0. Sadly, the problem is not fixed, as I still get @firebase/firestore: Firestore (11.0.0): FIRESTORE (11.0.0) INTERNAL ASSERTION FAILED: Unexpected state.

@basvandorst
Copy link

We are experiencing similar issues on several devices and it looks we were able to find the cause of this error (at least in our case):

Short timeline:

  • 2014-11-07: documentMutations = 0, estimated size = 75 MB
  • 2014-11-10: documentMutations = 0, estimated size = 75 MB
  • for days the mutations are going up / down
  • 2014-11-11 22:00: documentMutations = 3682, estimated size = 90 MB
  • 2014-11-11 23:00: documentMutations = 13000, estimated size = 102 MB
  • slightly increase in nr of mutaitons
  • 2024-11-13 18:00: documentMutations = 30000, estimated size = 140 MB
  • 2024-11-13 20:00: documentMutations = 35000, estimated size = 150 MB
  • 150 MB is exactly the moment when we see INTERNAL

All devices with this error have a high number of documentMutations and the same estimated IndexedDB size is ~150MB

It's also in line with the error message we see during / around the time of INTERNAL ASSERTION FAILED: Unexpected state-error:

DOMException: Failed to execute 'setItem' on 'Storage': Setting the value of 'firestore_targets_firestore/[DEFAULT]/xxxxxxxx/_4472' exceeded the quota.

our config:

{
  localCache: persistentLocalCache({
    cacheSizeBytes: 10485760,
    tabManager: persistentMultipleTabManager()
  }
}

most devices are running on iOS

{
  "quota": 4831838208,
  "quotaFormatted": "4.5 GB",
  "usage": 961174873,
  "avgLogSizeFormatted": "97.59 KB",
  "avgLogSize": 99935,
  "guestimated": 77678815,
  "totalEntries": 9618,
  "usageFormatted": "916.65 MB",
  "dbStats": [
    {
      "store": "logItems",
      "database": "LOG_ITEMS_DB_3VR6NV8FMH",
      "count": 5
    },
    {
      "store": "logItems",
      "count": 2,
      "database": "LOG_ITEMS_DB_ALL8M9663N"
    },
    {
      "database": "LOG_ITEMS_DB_JAF7879PKR",
      "count": 2,
      "store": "logItems"
    },
    {
      "count": 21,
      "database": "LOG_ITEMS_DB_JXY674C9NX",
      "store": "logItems"
    },
    {
      "store": "logItems",
      "count": 2,
      "database": "LOG_ITEMS_DB_X49UPXDWML"
    },
    {
      "database": "cmLogItems",
      "store": "logItems",
      "count": 5
    },
    {
      "database": "firebase-heartbeat-database",
      "count": 1,
      "store": "firebase-heartbeat-store"
    },
    {
      "count": 0,
      "database": "firebaseLocalStorageDb",
      "store": "firebaseLocalStorage"
    },
    {
      "store": "bundles",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0
    },
    {
      "store": "clientMetadata",
      "count": 1,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 128,
      "store": "collectionParents"
    },
    {
      "store": "documentMutations",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "documentOverlays",
      "count": 0
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "globals",
      "count": 0
    },
    {
      "count": 0,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "indexConfiguration"
    },
    {
      "store": "indexEntries",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0,
      "store": "indexState"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "mutationQueues",
      "count": 0
    },
    {
      "count": 0,
      "store": "mutations",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "namedQueries",
      "count": 0
    },
    {
      "count": 1,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "owner"
    },
    {
      "store": "remoteDocumentGlobal",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 1
    },
    {
      "store": "remoteDocumentsV14",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 2184
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "targetDocuments",
      "count": 4793
    },
    {
      "count": 1,
      "store": "targetGlobal",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "store": "targets",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 2471
    }
  ],
  "guestimatedFormatted": "74.08 MB"
}


{
  "totalEntries": 15649,
  "usage": 1213044879,
  "guestimatedFormatted": "102.53 MB",
  "quota": 4831838208,
  "usageFormatted": "1.13 GB",
  "quotaFormatted": "4.5 GB",
  "avgLogSizeFormatted": "75.7 KB",
  "guestimated": 107511759,
  "dbStats": [
    {
      "store": "logItems",
      "database": "LOG_ITEMS_DB_3VR6NV8FMH",
      "count": 5
    },
    {
      "count": 2,
      "database": "LOG_ITEMS_DB_ALL8M9663N",
      "store": "logItems"
    },
    {
      "count": 2,
      "database": "LOG_ITEMS_DB_JAF7879PKR",
      "store": "logItems"
    },
    {
      "store": "logItems",
      "count": 21,
      "database": "LOG_ITEMS_DB_JXY674C9NX"
    },
    {
      "count": 2,
      "database": "LOG_ITEMS_DB_X49UPXDWML",
      "store": "logItems"
    },
    {
      "count": 5,
      "store": "logItems",
      "database": "cmLogItems"
    },
    {
      "store": "firebase-heartbeat-store",
      "count": 1,
      "database": "firebase-heartbeat-database"
    },
    {
      "count": 0,
      "store": "firebaseLocalStorage",
      "database": "firebaseLocalStorageDb"
    },
    {
      "store": "bundles",
      "count": 0,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "clientMetadata",
      "count": 1
    },
    {
      "store": "collectionParents",
      "count": 128,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "store": "documentMutations",
      "count": 3682,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 1,
      "store": "documentOverlays"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "globals",
      "count": 0
    },
    {
      "store": "indexConfiguration",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0
    },
    {
      "count": 0,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "indexEntries"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "indexState",
      "count": 0
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0,
      "store": "mutationQueues"
    },
    {
      "store": "mutations",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 3682
    },
    {
      "store": "namedQueries",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0
    },
    {
      "store": "owner",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 1
    },
    {
      "count": 1,
      "store": "remoteDocumentGlobal",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "count": 1918,
      "store": "remoteDocumentsV14",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "store": "targetDocuments",
      "count": 3996,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "store": "targetGlobal",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 1
    },
    {
      "store": "targets",
      "count": 2200,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    }
  ],
  "avgLogSize": 77515
}


{
  "quota": 4831838208,
  "usageFormatted": "1.38 GB",
  "guestimated": 157982000,
  "avgLogSize": 18781,
  "totalEntries": 78991,
  "quotaFormatted": "4.5 GB",
  "usage": 1483599673,
  "dbStats": [
    {
      "count": 5,
      "store": "logItems",
      "database": "LOG_ITEMS_DB_3VR6NV8FMH"
    },
    {
      "count": 4,
      "database": "LOG_ITEMS_DB_ALL8M9663N",
      "store": "logItems"
    },
    {
      "count": 2,
      "store": "logItems",
      "database": "LOG_ITEMS_DB_JAF7879PKR"
    },
    {
      "store": "logItems",
      "count": 21,
      "database": "LOG_ITEMS_DB_JXY674C9NX"
    },
    {
      "database": "LOG_ITEMS_DB_X49UPXDWML",
      "store": "logItems",
      "count": 2
    },
    {
      "store": "logItems",
      "count": 5,
      "database": "cmLogItems"
    },
    {
      "store": "firebase-heartbeat-store",
      "database": "firebase-heartbeat-database",
      "count": 1
    },
    {
      "count": 0,
      "database": "firebaseLocalStorageDb",
      "store": "firebaseLocalStorage"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0,
      "store": "bundles"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "clientMetadata",
      "count": 1
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 128,
      "store": "collectionParents"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "documentMutations",
      "count": 35352
    },
    {
      "count": 1,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "documentOverlays"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "globals",
      "count": 0
    },
    {
      "count": 0,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "indexConfiguration"
    },
    {
      "count": 0,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "indexEntries"
    },
    {
      "count": 0,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "indexState"
    },
    {
      "store": "mutationQueues",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 35352,
      "store": "mutations"
    },
    {
      "store": "namedQueries",
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 0
    },
    {
      "count": 1,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "owner"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "remoteDocumentGlobal",
      "count": 1
    },
    {
      "store": "remoteDocumentsV14",
      "count": 1918,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    },
    {
      "count": 3996,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "store": "targetDocuments"
    },
    {
      "database": "firestore/[DEFAULT]/xxxxxxxx/main",
      "count": 1,
      "store": "targetGlobal"
    },
    {
      "store": "targets",
      "count": 2200,
      "database": "firestore/[DEFAULT]/xxxxxxxx/main"
    }
  ],
  "avgLogSizeFormatted": "18.34 KB",
  "guestimatedFormatted": "150.66 MB"
}

#8383 #7884 #7712 #6684 #4451 #2710

@basvandorst
Copy link

To validate if you guys have similar issues, run this script in the console:

(function showIndexedDbSize() {
  "use strict";

  const dbName = "firestore/[DEFAULT]/your-project-name/main";
  let db;
  const storeSizes = [];

  function openDatabase() {
    return new Promise((resolve, reject) => {
      const request = indexedDB.open(dbName);
      request.onsuccess = (event) => {
        db = event.target.result;
        resolve(db.objectStoreNames);
      };
      request.onerror = (event) => reject(event.target.error);
    });
  }

  function getObjectStoreData(storeName) {
    return new Promise((resolve, reject) => {
      const transaction = db.transaction(storeName, "readonly");
      const store = transaction.objectStore(storeName);
      const items = [];

      transaction.oncomplete = () => {
        const sizeInBytes = calculateSize(items);
        const sizeInMB = (sizeInBytes / 1024 / 1024).toFixed(2);

        storeSizes.push({
          'Store Name': storeName,
          'Items': items.length,
          'Size': `${sizeInMB}MB (${sizeInBytes} bytes)`
        });

        resolve();
      };

      store.openCursor().onsuccess = (event) => {
        const cursor = event.target.result;
        if (cursor) {
          items.push(cursor.value);
          cursor.continue();
        }
      };
    });
  }

  function calculateSize(items) {
    return items.reduce((size, item) => {
      return size + (JSON.stringify(item).length * 2);
    }, 0);
  }

  openDatabase()
    .then(stores => {
      const storePromises = Array.from(stores).map(getObjectStoreData);
      return Promise.all(storePromises);
    })
    .then(() => {
      console.table(storeSizes);
    })
    .catch(error => {
      console.error('Error:', error);
    });
})();

This will output the size of each object store, including the documentMutations (.. in this example it's 0, but we have also seen 33k items with ~130 MB)

(index) Store Name Items Size
0 'bundles' 0 '0.00MB (0 bytes)'
1 'documentMutations' 0 '0.00MB (0 bytes)'
2 'documentOverlays' 0 '0.00MB (0 bytes)'
3 'globals' 0 '0.00MB (0 bytes)'
4 'indexConfiguration' 0 '0.00MB (0 bytes)'
5 'indexEntries' 0 '0.00MB (0 bytes)'
6 'indexState' 0 '0.00MB (0 bytes)'
7 'mutationQueues' 0 '0.00MB (0 bytes)'
8 'mutations' 0 '0.00MB (0 bytes)'
9 'namedQueries' 0 '0.00MB (0 bytes)'
10 'clientMetadata' 1 '0.00MB (212 bytes)'
11 'owner' 1 '0.00MB (196 bytes)'
12 'remoteDocumentGlobal' 1 '0.00MB (40 bytes)'
13 'targetGlobal' 1 '0.00MB (312 bytes)'
14 'collectionParents' 47 '0.01MB (11466 bytes)'
15 'targets' 1358 '1.63MB (1710700 bytes)'
16 'remoteDocumentsV14' 1445 '5.00MB (5238646 bytes)'
17 'targetDocuments' 3240 '1.05MB (1100666 bytes)'

@luke-rogers
Copy link
Author

Still getting a very large number of reports for this issue.

Added an additional log file for Firebase version 10.14.1: 2024-12-11_10-28-1.log

@wu-hui
Copy link
Contributor

wu-hui commented Jan 2, 2025

@luke-rogers The link you posted does not work for me.

@luke-rogers
Copy link
Author

@wu-hui I've just invited you to the repository.

@wu-hui
Copy link
Contributor

wu-hui commented Jan 2, 2025

I can see the logs now, thanks.

Sorry if it is buried in the thread: are you able to reproduce this yourself? If so, can you share a minimum reproduction?

OTOH, I believe this is the outcome of https://bugs.webkit.org/show_bug.cgi?id=202705

It was reported before #2232 and we made a change to make the error less frequent (#2307). The root cause is still there however. To properly fix this we might need to reconsider how the SDK interacts with indexeddb, and this is going to be a pretty big change that we do not have a timeline for.

We can update here once we have a decision on what to do.

For googlers only: b/387310299

@luke-rogers
Copy link
Author

Unfortunately, we've never found a way to reproduce these errors ourselves. The best I've got is the Firebase debug logs in the repository I added you to.

There seem to be several combinations of the INTERNAL ASSERTION FAILED: Unexpected state error in those logs.

We're seeing thousands of these errors per week across our customers though so it's certainly prevalent.

@wu-hui
Copy link
Contributor

wu-hui commented Jan 3, 2025

Hi Luke,

I understand this is worrying. Unfortunately it might be out of what we can do to completely fix this issue. We will discuss internally to see if we can reduce the frequency this happens.

We are also under capacity constraints recently, so whatever we decided to do, it might take some time.

Sorry for the inconvenience.

@frell
Copy link

frell commented Jan 8, 2025

Hello,

@luke-rogers We have spent the past few months fire fighting this on and off with our app and have finally got it to the point where we have a repeatable easy to recreate (for us) test case.

Can confirm is connected to enable/disabling persistence. We noticed it happened much more often for slow/unstable internet connected devices.

In our test case the best way to reproduce was to;

  1. Setup some internet traffic rules on our office router to cap the internet speed to 1mb for the device being tested (this was important to make it easily repeatable every time -- i don't know why

  2. Have the app downloading files from Firebase storage and updating a FireStrore doc for each successful file downloaded

  3. Works perfect until you sleep the device (press power button) -- Then INTERNAL ASSERTION FAILED: Unexpected state errors fill the console logs

Once this occurs only a window.reload() will get firebase errors to stop.

@wu-hui - Do you and the team already have enough information from what @luke-rogers has provided to investigate or would it be helpful if we produced a Minimal Reproducible Example also?

@Revadike
Copy link

Revadike commented Jan 12, 2025

I my particular case the solution was to downgrade to the version 10.6.0, 10.7.0 is where the issue start. Here is the diff. If I got time I'll try to debug the SDK locally to see if I can figure out what the error is

Thank you! Downgrading worked for me. This should be emphasized as the best current workaround.

I'll look forward to the actual fix. Best of luck!

@Revadike
Copy link

I my particular case the solution was to downgrade to the version 10.6.0, 10.7.0 is where the issue start. Here is the diff. If I got time I'll try to debug the SDK locally to see if I can figure out what the error is

Thank you! Downgrading worked for me. This should be emphasized as the best current workaround.

I'll look forward to the actual fix. Best of luck!

Oops, nevermind...
Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests