-
Notifications
You must be signed in to change notification settings - Fork 912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
App Check works at first but fails to reconnect after sleep/background #6708
Comments
I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight. |
Ok, this is a lot of information but it seems like the core of it is that you get 403 attestation errors while the tab is in the background or the computer is asleep. I don't think the Firestore "terminate" thing is related. 403 attestation errors indicate that the ReCAPTCHA token sent to the AppCheck endpoint is bad, which could be for various reasons: it wasn't able to get any ReCAPTCHA token, or it got one and you got a bad ReCAPTCHA score (it thinks you are a bot or something), or the project API key doesn't match. The backoff for this type of error is set to 1 day. This is because it was assumed a ReCAPTCHA failure was a more or less long term problem that wouldn't somehow fix itself in a few seconds, unlike other errors such as a 503 (server too busy). It might be possible to revisit this backoff policy and change it in the specific case of it being a 403 AND it being an attestation error (there's other types of 403s), to use a shorter exponential backoff, in case the ReCAPTCHA check failed due to the tab being in the background? (I'm guessing that's what happened.) This will be somewhat difficult because you can't tell on the client side what the ReCAPTCHA score was (as intended, to avoid exploitation). Before I do that I want to do some more research into what happens to ReCAPTCHA when the tab is backgrounded, whether it's simply unable to reach the endpoint or whether it's actually giving you a failed score. |
A bit more info that might help: I left two tabs open yesterday. This time they got the 403 after ~23 & ~24 hours (while in the background but the laptop was not asleep). I filtered the network tab for Update: Retrying the experiment with two fresh tabs the next day.
But since the first request succeeded on both tabs, this time both of them survived. |
As mentioned above, there is a 1 day backoff when the response is a 403. I will try to look into some kind of solution for your situation that lets us maintain a long backoff for the cases it was intended for. |
I'm getting the same issue but with out leaving the tab open, I just get the error right away when trying to interact with firebase services. I'm setting up a contact form for my app and am hoping to use app check to block potential bots from submitting the form. I've set up app check with my other firebase services. But when trying to add a document to firestore, it gets blocked due to app check (which is supposed to happen) but with the app check 403 error (which I have no clue why is showing up). |
same error here! anyone found a solution?? |
Just to be clear, a 403 error is not a bug in itself. It means that App Check failed attestation. That should happen when the app is not allowed to use App Check. That could mean reCAPTCHA failed or the app making the request is providing the wrong app ID, or there's a typo in your reCAPTCHA site key. This is what should happen in these cases. Further requests to the App Check backend are then throttled for 24 hours, and the reasoning behind this is that if any of these things are the case, it won't be fixed by retrying (whereas a 50x error might be), so it's throttled to not overwhelm your quota. If you're getting this when you expect access to be blocked, that's normal. If you're testing things out in development you can clear your local indexedDB cache to get it to retry again without waiting 24h. If you're expecting access not to be blocked, make sure your reCAPTCHA account is set up properly and registered in the Firebase console. If you feel like a 24h throttle is not appropriate in some cases (which seems like it might be the case), let us know the details, we are looking into what we can do to make it more granular somehow. |
Hi, everyone. I had the same problem, but after carefully checking it out based on @hsubox76 advice, I was able to solve it. ✅
First (and I don't know if this is truly relevant), I decided to generate a non-test Enterprise Key and use that. The code is as follows: import { initializeApp } from 'firebase/app'
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from 'firebase/app-check'
const firebaseConfig = {
// hide
}
const app = initializeApp(firebaseConfig)
initializeAppCheck(app, {
provider: new ReCaptchaEnterpriseProvider(import.meta.env.VITE_APP_RECAPTCHA_SITE_KEY),
isTokenAutoRefreshEnabled: true,
}) Next, I reconfigured the Finally, just to be safe, I disabled That's all there is to it. Incidentally, there was no need to wait 24 hours after this. Best wishes for success. |
I also started getting this error today, even though it's been working fine for the past year. Did not change any configuration. |
I got the solution. The problem it's not you. It's the system. AppCheck to work without that error needs that the computer IP Address is the same when you register to AppCheck. My Router changed address my PC and I got a lot of errors that's the only one that remains until Changed the IP Address Back. I hope that helps. 😊 |
I am not getting this error when using normal browsing (Chrome), but when trying on incognito I get the error. A coworker of mine reported this issue to me today when he tried accessing our staging site and he gets this regardless of normal or incognito. We we're both able to access the site fine just yesterday. The only assumption I can think of based on what you said is recaptcha thinks that I'm a "bad" user when using incognito. Is there a way to verify this? And are there any ways of telling appcheck to allow the user anyways, or perhaps lower the score threshold? Or allow additional verification should it be needed. From reading the recaptcha v3 features, this seems to be supported. |
I made an error deploying an app with a wrong recaptcha key. OK , I understand it but now I have to tell my manager I can't work for 1d to fix this? |
Hi guys, I just want to clarify that this issue is not about configuring AppCheck or Recaptcha for the first time. If you have a different issue, please open a separate issue to keep the discussion here focused on the main issue.
|
This is an interesting problem and worth looking into, but as mentioned above, can you open a new issue for this? |
I think the focus of this issue should probably be cases where the currently configured 1d backoff (will not retry) after hitting a 403 error is inappropriate. Cases it was intended for:
Retries in those cases will just blow out your quota for no good reason. Cases where the 1d backoff might be inappropriate:
The last case I think is the initial problem brought up in this issue. The problem is that we have to figure out what state is causing the failure (offline? background tab? sleep?) and if there is a way to detect the browser being in that state, or detect coming out of that state. That part is going to be pretty tricky. If anyone has any of the 2 cases above ("Cases where the 1d backoff might be inappropriate") and has any more insight into them, or can think of an additional case, please add in this issue. If it's not related to this, please open a separate issue. |
Thanks for the solution |
Hey @hsubox76, Is there any update on this at all? We're seeing this issue a lot in our logs and it's causing a lot of error noise. A lot of the time it's happening after the user comes back from being idle as previously mentioned in this thread. It's hard to know exactly what experience the user is getting when this happens. Would love to know if there's a fix in the works or a temporary workaround? We're even starting to consider moving away from AppCheck as we're concerned it may be impacting conversions if we have no visibility on how it effects users. |
Working on a PR that may address the problem if the issue is that reCAPTCHA is failing and not being caught, and the App Check endpoint is then being called unnecessarily: #7203 There still may be a lot of errors but they will say "recaptcha error" and a request won't be sent to the App Check exchange endpoint, which means you won't get a 403, which means further requests won't be throttled for 1 day. That means if the reCAPTCHA situation corrects itself (computer wakes up? tab is foregrounded?) the SDK will be free to send a new request to the App Check exchange endpoint that should work. It seems like there may be multiple things going on in this situation so I'm not going to tag that PR as "fixes" this issue (which would close it) because there may be more to address. |
Yep that's the issue @hsubox76 :) Agree there's a few other issues going on here but that's the main one for us. Looking forward to the release and happy to test on this end when something is ready. I've subscribed to the PR so will keep track there. Thanks for the quick response. |
Release 9.20.0 went out yesterday and should have the change described above. |
This problem is still happening for us, and we've been on 9.20.0 for a few days now. |
Hi @google-oss-bot, the issue is happening less frequently after @hsubox76's improvements - thanks a lot! But I've encountered it at least once after all the updates (i.e., AppCheck errors after returning to an idle tab, leading to Firestore disconnecting and the app stops working). I'll share more details the next time I observe this. |
We are also still experiencing this issue, on v9.22.2 |
Update: This issue is occurring less frequently (thanks to @hsubox76's improvements), but unfortunately it still occurs. Example console log with 403 crashBasically:
My analysis of what's happeningThere are two types of token requests:
Observations
Reproduction steps
Suggested fixAssuming my analysis is correct (that the success of |
Great reproduction write up @anisabboud I was hoping it might mean a bit of action on this issue but its been 3 weeks. We have had to disable app check as it makes our site unusable for a significant portion of our users. Now investigating alternatives. Assuming firebaseappcheck is failing because the recaptcha token has expired. Do you still have access to the response from the firebaseappcheck request? Im wondering if there is an error code or something that this js sdk could use to know it has to get a new recaptcha token. Also from what i can see using the getToken(forceRefresh = true) does not force refresh the recaptcha token it only does the firebaseappcheck exchange using the same busted recapcha token. |
Please help on firebaseappcheck alternative, as it was working for almost all users , but now only 30% users are able to use. Updated to latest version, but issue still persists. |
I'm commenting here since the issue I opened #7116 was closed and locked. This error happened less often but still occurs. |
Please open a new issue if it is not related to failures after waking from sleep/background. Edit: Also, for more info on a general ReCAPTCHA error, please see this comment above: #6708 (comment) |
Thanks for the additional info @anisabboud but the "securetoken" endpoint is an auth endpoint and not related to App Check requests. I think from previous posts you've made, you may simultaneously be having two different issues upon sleep/wake, one with App Check, and one with Auth, which can get mixed up in the logs. Regarding this issue, the lack of further solutions is not because it's not important to us, it's because we are unable to reproduce it. All of our previous fixes have been just guessing what might be going wrong on your machines and taking our best guess at addressing that. One thing that might help us reproduce it faster is if it doesn't take 2 24 hour periods to attempt one reproduction. I would bet 24 hours is not some kind of hardcoded timeline but the TTL set in the console. Can you change it to 30 minutes and reproduce the same behavior over 2 30 minute periods instead? That seems like it would give us a better chance at reproduction. Here's where you would change the App Check TTL in the console: ![]() Additionally, if anyone has any other ways to quickly reproduce the error, especially if it's guaranteed to always get the same result, that's what would be most helpful in solving this issue. |
Hey @anisabboud. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically. If you have more information that will help us get to the bottom of this, just add a comment! |
Since there haven't been any recent updates here, I am going to close this issue. @anisabboud if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this. |
I am experiencing a similar issue with Firebase AppCheck. When a tab with my website remains inactive in the background, returning to it often results in the 403 error. I suspect the root cause might be linked to reCAPTCHA Enterprise's inability to gather sufficient user interaction data in inactive tabs, leading to failed human verification checks (false positives). A potential solution could be allowing the option to disable or limit attestation request throttling in the SDK. This way, if an attestation fails, a new attempt could be made promptly, potentially resolving the issue. This could mitigate issues arising from false positives. For context, I recently integrated Firebase AppCheck into my staging environment, a production-like setup (using a live Firebase project on a deployed website) restricted to internal testing. The occurrence of this issue in a live environment would be highly detrimental to user experience. Is there any update from the Firebase team regarding a solution to this problem? @hsubox76 |
I've successfully replicated the problem across different devices and networks (different IP addresses). Here's a description of the steps I followed to reproduce the error:
Additional scenarios where the error occurs:
Context for my testing environment:
|
Sorry, I didn't see the additional comments after the issue had been closed. I've just created a new issue and pasted in the last two comments to try to get a fresh start with this bug as this one contains a lot of extraneous info and is hard to parse. However, I've linked back to this one from the new one in case there's any important info here we need to refer to. |
Hi Gabriel, unfortunately I have not been able to successfully reproduce the issue following the steps you provided. I would really like to be able to reproduce this- it would help me if you could provide the following information:
Or, preferably, could you share a minimal Next.js app that you have been able to successfully reproduce the issue with? |
Summary
Using latest Firebase v9.12 (through latest AngularFire 7.4.1) in production, with AppCheck, Firestore, and Auth.
When leaving the webapp tab open and idle, the app often runs into a
403
error in POST request to exchange AppCheck token ("App attestation failed." "PERMISSION DENIED"), followed by a warning:@firebase/app-check: AppCheck: Requests throttled due to 403 error. Attempts allowed again after 01d:00m:00s (appCheck/throttled).
Console Tab
As you can see, the
appCheck/throttled
warning shows up twice, and is preceded by a failing 403 POST request tocontent-firebaseappcheck.googleapis.com/...exchangeRecaptchaV3Token
There was no exponential backoff. Quoting from @hsubox76 in #6373 (comment):
After the failed token exchange & AppCheck throttling, Firestore can no longer load data, thus logging several "
FirebaseError: Missing or insufficient permissions.
" on all active document/collection listeners, rendering the app dysfunctional.In a second tab running the same app, a similar crash was observed, but it was preceded by a

Timeout
error inrecaptcha__en.js
and a@firebase/app-check: FirebaseError: AppCheck: ReCAPTCHA error. (appCheck/recaptcha-error).
Network Tab
As you can see from the network tab, the 403 POST request failed with "App attestation failed." - when is this supposed to occur?

There were no further requests to firebaseappcheck.googleapis.com, but there was a successful

OPTIONS
request200
at the same secondWed, 19 Oct 2022 10:17:35 GMT
:Right before the failing 403 request, during the same second 10:17:35 GMT, there was a POST request to Firestore with query parameter

TYPE=terminate
- could this be related? Sounds suspicious.There were no further requests to firebaseappcheck.googleapis.com but there were successful token requests to

securetoken.googleapis.com/v1/token
every 55 minutes before & after the 403 issue:09:07:52 GMT
10:02:53 GMT
10:57:54 GMT
11:52:55 GMT
12:47:56 GMT
Reproduction
The issue is tricky to reproduce, as it's related to token expiration. I often leave the tab open for several days before it happens.
Today I encountered this issue during the workday, while the computer was on all the time, in two different tabs that were open in the background. The same behavior mentioned above was observed in both tabs, 4 seconds apart:
Wed, 19 Oct 2022 10:17:35 GMT
Wed, 19 Oct 2022 10:17:39 GMT
These 4 seconds could be the difference from when I initially opened or interacted with each tab, hinting at a token expiration issue timeout, possibly initiated by Firestore terminate.
The text was updated successfully, but these errors were encountered: