FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state

[luke-rogers]

Operating System

iOS 17.4.1

Browser Version

Mobile Safari UI/WKWebView (AppleWebKit/605.1.15)

Firebase SDK Version

10.12.0

Firebase SDK Product:

Firestore

Describe your project's tooling

Angular web app running on iOS via Capacitor
Firestore offline persistence enabled with unlimited cache size

Describe the problem

For a long time now we have been plagued by this iOS-only issue and have received many reports from users that it's causing the app to stop working and forcing them to restart.

We have captured some Firebase debug logs of this error which I am happy to share privately.

This seems similar to the following issues:

• Snapshot stops working after Capacitor app being minimized for 5m #8068
• @firebase/firestore: Firestore (9.23.0): FIRESTORE (9.23.0) INTERNAL ASSERTION FAILED: Unexpected state #7712
• FIRESTORE (8.2.1) INTERNAL ASSERTION FAILED: Unexpected state #4451

Example Errors

• UnknownError: Attempt to get a record from database without an in-progress transaction
• UnknownError: Attempt to get all index records from database without an in-progress transaction
• UnknownError: Connection to Indexed Database server lost. Refresh the page to try again
• FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state

Example Stack Trace

Error: FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state
  at fail(node_modules/@firebase/firestore/dist/index.esm2017.js:197:36)
  at __PRIVATE_AsyncQueueImpl.Pu(node_modules/@firebase/firestore/dist/index.esm2017.js:18775:16)
  at __PRIVATE_AsyncQueueImpl.enqueue(node_modules/@firebase/firestore/dist/index.esm2017.js:18691:14)
  at __PRIVATE_AsyncQueueImpl.enqueueAndForget(node_modules/@firebase/firestore/dist/index.esm2017.js:18676:10)
  at n(node_modules/@firebase/firestore/dist/index.esm2017.js:14564:15)
  at <anonymous>(node_modules/@firebase/firestore/dist/index.esm2017.js:14540:7)
  at __PRIVATE_StreamBridge.yo(node_modules/@firebase/firestore/dist/index.esm2017.js:13897:10)
  at n(node_modules/@firebase/firestore/dist/index.esm2017.js:14145:81)
  at <anonymous>(node_modules/@firebase/firestore/dist/index.esm2017.js:14094:13)
  at ab(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:548:22)
  at F(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:516:9)
  at Z.prototype.ta(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:2162:5)
  at Rb(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1035:129)
  at M.prototype.Y(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:904:50)
  at M.prototype.ca(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:832:35)
  at ab(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:548:22)
  at F(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:516:9)
  at Wc(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1582:128)
  at h.bb(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1579:5)
  at h.Ea(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1576:61)
  at Lc(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1474:50)
  at h.Pa(node_modules/@firebase/webchannel-wrapper/dist/webchannel-blob/esm/webchannel_blob_es2018.js:1440:27)
  at {_ZoneImpl#2}.run(node_modules/zone.js/fesm2015/zone.js:111:37)
  at <anonymous>(node_modules/zone.js/fesm2015/zone.js:2299:30)
  at {_ZoneImpl#2}.runTask(node_modules/zone.js/fesm2015/zone.js:157:39)
  at drainMicroTaskQueue(node_modules/zone.js/fesm2015/zone.js:529:23)

Steps and code to reproduce issue

Steps currently unknown but possibly related to the app going between the background and the foreground over time.

[luke-rogers]

Looking into the Firebase logs, the error seems to occur after updateClientMetadataAndTryBecomePrimary is called.

Our app uses persistentSingleTabManager(undefined) for the tabManager config in the Capacitor environments as there can never be more than 1 tab. I assume this tab manager is the cause of the many PUT clientMetadata and SimpleDb GET owner owner logs.

I wonder if there would be any legs in not having a tab manager if that will result in no calls to updateClientMetadataAndTryBecomePrimary etc.

This is also similar to what is described by another GitHub user here

I don't know a lot about the internals of the tab management so might be completely wrong here.

Sample Log

[2024-05-16T13:03:08.238Z] Firestore (10.12.0): SimpleDb PUT clientMetadata <auto-key> {"clientId":"jtQjaM2I0v6CbepACBf7","updateTimeMs":1715864588238,"networkEnabled":true,"inForeground":false}
[2024-05-16T13:03:08.242Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864584009}
[2024-05-16T13:03:08.244Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864584009}
[2024-05-16T13:03:08.244Z] Firestore (10.12.0): IndexedDbPersistence Client 'jtQjaM2I0v6CbepACBf7' is not zombied in LocalStorage
[2024-05-16T13:03:08.244Z] Firestore (10.12.0): SimpleDb PUT owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864588244}
[2024-05-16T13:03:25.878Z] Firestore (10.12.0): IndexedDbPersistence Starting transaction: updateClientMetadataAndTryBecomePrimary
[2024-05-16T13:03:48.787Z] Firestore (10.12.0): SimpleDb Aborting transaction: Attempt to get a record from database without an in-progress transaction
[2024-05-16T13:03:48.787Z] Firestore (10.12.0): SimpleDb Transaction failed with error: Attempt to get a record from database without an in-progress transaction Retrying: true
[2024-05-16T13:03:48.787Z] Firestore (10.12.0): SimpleDb Opening database: firestore/[DEFAULT]/insite-teams/main
[2024-05-16T13:03:48.802Z] Firestore (10.12.0): WebChannelConnection RPC 'Listen' stream 0x7d7f4c65 received: {"targetChange":{"resumeToken":"CgkIic7l5p2ShgM=","readTime":"2024-05-16T13:03:34.823689Z"}}
[2024-05-16T13:03:56.239Z] Firestore (10.12.0): SimpleDb Aborting transaction: Attempt to get a record from database without an in-progress transaction
[2024-05-16T13:03:56.239Z] Firestore (10.12.0): SimpleDb Transaction failed with error: Attempt to get a record from database without an in-progress transaction Retrying: true
[2024-05-16T13:03:56.239Z] Firestore (10.12.0): SimpleDb Opening database: firestore/[DEFAULT]/insite-teams/main
[2024-05-16T13:03:56.243Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864588244}
[2024-05-16T13:03:56.243Z] Firestore (10.12.0): SimpleDb PUT clientMetadata <auto-key> {"clientId":"jtQjaM2I0v6CbepACBf7","updateTimeMs":1715864636243,"networkEnabled":true,"inForeground":false}
[2024-05-16T13:03:56.244Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864588244}
[2024-05-16T13:03:56.245Z] Firestore (10.12.0): SimpleDb GET owner owner {"ownerId":"jtQjaM2I0v6CbepACBf7","allowTabSynchronization":false,"leaseTimestampMs":1715864588244}
[2024-05-16T13:04:02.786Z] Firestore (10.12.0): SimpleDb Aborting transaction: Attempt to get records from database without an in-progress transaction
[2024-05-16T13:04:02.786Z] Firestore (10.12.0): SimpleDb Transaction failed with error: Attempt to get records from database without an in-progress transaction Retrying: false
[2024-05-16T13:04:02.788Z] Firestore (10.12.0): INTERNAL UNHANDLED ERROR:  Attempt to get records from database without an in-progress transaction
[2024-05-16T13:04:19.979Z] Firestore (10.12.0): WebChannelConnection RPC 'Listen' stream 0x7d7f4c65 received: {"targetChange":{"resumeToken":"CgkIrpqg/J2ShgM=","readTime":"2024-05-16T13:04:19.823918Z"}}
[2024-05-16T13:04:19.989Z] Firestore (10.12.0): FIRESTORE (10.12.0) INTERNAL ASSERTION FAILED: Unexpected state

@wu-hui I would be Happy to share the full log with you privately if you could let me know the best way to do that.

[vojto]

This has gotten really bad for us over the past few days. We didn't deploy anything.

Edit: This is on desktop.

[luke-rogers]

@vojto Perhaps that's a different issue, this has always been limited to just iOS for us.

[wu-hui]

I think the multitab log might be unrelated, this seems to stem from a underlying webchannal message.

[luke-rogers]

Fair enough, not an area I've dug too deep into.

@wu-hui what steps can I take to help try and move this forward?

[luke-rogers]

Still seeing this issue on the latest version. I have logs I can share if that would be useful?

[MarkDuckworth]

@luke-rogers I would like to review the logs. You can share the logs through a private GitHub repo (shared with me), through a Cloud support ticket, or through a Firebase support ticket. If creating a ticket with either Google support channel, please reference this GitHub issue so the support staff knows to route the ticket to my team. Thanks!

[SamLoys]

I encounter the same problem.
For me this happens in my webapp running in a webview through Flutter.

Usually this is combined with
QuotaExceededError __PRIVATE_WebStorageSharedClientState.setItem(@firebase/firestore/dist/index.esm2017)

Failed to execute 'setItem' on 'Storage': Setting the value of 'firestore_clients_firestore/[DEFAULT]/kekkle/_jLkv2Vnt89NL5Xx3UYtN' exceeded the quota.

The problem is always resolved by loggin out (clearing local storage).
So my first guess is that the quota exception is not handled correcrly

EDIT: I created a new issue, as I am not sure if the underlying problem is the same.

[luke-rogers]

@MarkDuckworth I've shared the logs through a Firebase support ticket and via a private repo

Let me know if you need anything more from me.

[luke-rogers]

@MarkDuckworth did you get access to the logs? Support said they forwarded them on.

[MarkDuckworth]

@luke-rogers, I received the logs. Thank you for sharing. On initial review, I don't see an obvious cause, but we will give it some deeper investigation as soon as we can.

I've also started work on a PR to help us get error codes from these internal assertion failed messages.

Do you also have a reproducible example that you can share with us?

[luke-rogers]

@MarkDuckworth Thanks for the update.

PR looks good. Hopefully, you will be able to merge that. We would be happy to test a pre-release version of this if that would still be helpful?

Unfortunately, we do not have a way of reproducing this ourselves at the minute.

I have added some links to stack traces from Sentry to the readme of private repo I shared with you.

[tom-andersen]

@luke-rogers Mark is out of office and asked I take a look at logs. Could you grant access to private repo, to myself @tom-andersen and maybe some other members of our team such as @wu-hui ? Thanks.

[luke-rogers]

@tom-andersen added yourself and @wu-hui to the repo.

Still struggling with a reproduction but if there is anything else I can try and help with please let me know.

[tom-andersen]

Update:

The error with text database without an in-progress transaction is from within the IndexedDB implementation. This becomes a fatal error when Firestore settings disable multi-tab support (as far as I can tell from reading code).

The workaround might be to enable multi-tab support, since Firestore SDK will then handle this error by entering a non-primary role until IndexedDB resolves the problem, instead of treating this like a fatal error.

That this happens at all, is a little concerning, and might have more to do with the AppleWebKit.

[tom-andersen]

@luke-rogers Are you able to enable multi-tab and see if that resolves the problem?

[luke-rogers]

@tom-andersen we will enable multi-tab and report back. Hopefully doing a release today so should have some initial feedback by the end of the week.

[google-oss-bot]

Hey @luke-rogers. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[luke-rogers]

We released yesterday with the change to enable multi-tab. We will have more of an update within the next few days as to whether it's made a difference.

[luke-rogers]

Unfortunately enabling multi-tab has not resolved this issue. I am still trying to access the impact on the end user experience but we have managed to capture some logs for the error. I've added those to the GitHub repo previously shared.

The new log files for this error with Firebase version 10.12.4 and multi-tab enabled are:

1. 2024-8-1_17-6-47.log
2. 2024-8-1_8-42-34.log