-
-
Notifications
You must be signed in to change notification settings - Fork 31.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nest Cameras/Doorbell showing too many API calls since latest HA update #130061
Comments
Hey there @allenporter, mind taking a look at this issue as it has been labeled with an integration ( Code owner commandsCode owners of
(message by CodeOwnersMention) nest documentation |
How long was the stream open here? i'm seeing these counters: {
"command": {
"sdm.devices.commands.CameraLiveStream.GenerateWebRtcStream_count": 52,
"sdm.devices.commands.CameraLiveStream.GenerateWebRtcStream_sum": 31924,
"sdm.devices.commands.CameraLiveStream.StopWebRtcStream_count": 19,
"sdm.devices.commands.CameraLiveStream.StopWebRtcStream_sum": 30070,
"fetch_image_count": 1,
"fetch_image_sum": 496,
"sdm.devices.commands.CameraLiveStream.ExtendWebRtcStream_count": 112180,
"sdm.devices.commands.CameraLiveStream.ExtendWebRtcStream_sum": 23248068
},
} where there were 50 streams created but over 100k streams extended! which seems indeed like a bug. Can you turn on debug logging and capture debug logs? I think we could see what the timings are between extending the stream. |
I usually check the camera through the day but never leave it open for more than 10-15 seconds at a time. I usually check it most of the time on my phone. When the error started happening, it usually doesn't even give me a chance to view the camera, even after several minutes. I've attached some debug logs of when it throws the error. I've attached debug logs, one while it works before it breaks and the other while it throws the error consistently for a long time. Please let me know if you need anything else. |
Thank you the detail is very helpful. I think a problem is that once the stream expires then fails to renew, it repeatedly hammers the API trying to extend the stream. We may not be able to avoid occasionally going over the rate limit, but once this happens it guarantees it won't stop. |
I'm experiencing the same issue immediately after updating from HA 2024.10.4 to HA 2024.11.0. I tried fully cycling the VM running Home Assistant, then allowed for a cooling off period, but it appears that system unconditionally ends up in that API hammering loop. The interesting thing is, if I rollback to a VM snapshot I took just prior to the upgrade, it starts working immediately. A bit surprising, because I expected the API limits to still be in play, but it's apparently only being exhausted by the behavior of 2024.11.0. |
Thanks for the report, it definitely seems consistent with a stream expiration code issue. Working on a fix. |
Thank you for the help so far! I saw that 2024.11.1 came out with the fix related to this issue. I installed it and restarted HA, but still have the same issue unfortunately. Works for a bit then breaks again eventually. |
I'm seeing similar behavior. Working fine for a few minutes, then regressed back to the previous API rate limit. |
I posted about this on Discord, also still seeing the issue 2024.11.1 |
I think I'm getting the same issue. On 2024.11.0, it stopped working completely with the 'Failed to start webRTC stream' error. Now on 2024.11.1, it doesn't stop completely, but I get the following error in the logs every time I try to access a Nest camera/doorbell through Home Assistant:
|
Re-opening as I think there still may be an issue here where too many streams are opened in parallel with the go2rtc fallback change, but i'm going to investigate a little more. |
Hey @allenporter, at least for me I still have this issue. Thanks for looking into this :) |
Part of the issue may be that both types of streams are now opened with |
Is the intended behaviour that the Nest camera should stop streaming when I close the page containing the card in my browser? The state of the camera in the battery camera's own device page says "Streaming" whenever I look. As a battery camera it shouldn't be, unless something is holding it open when not required. Didn't know if this could be connected with the issue. |
@IngeJones nest camera APIs have no state, and so it has to either show up as streaming or off and so it shows up as streaming. Don't worry, it's not draining your battery when not in use. |
Just to add: I have a Nest thermostat as well as a doorbell and this issue is stopping me controlling my thermostat from HA. |
Is there a user-facing configuration that allows one to disable the new go2rtc functionality (globally or perhaps on a more granular basis)? The release notes for 2024.11 don't seem to mention anything. |
I recommend either rolling back home assistant versions or disabling cameras if you need a workaround. |
I'd love the choice per camera, or even per streaming card. Obviously I don't know whether it's feasible or how much work it would entail. |
|
Which HA version? Just disabling the camera entities in HA, or somehow disabling from the GCloud side? |
Folks say this started in 2024.11.0 and I included some fixes in 2024.11.1 that were bugs of trying to refresh the stream way way too much. If someone is experiencing the API rate limitinf in 2024.11.1 or later I would love more data about the API calls you see when debug log is enabled and including diagnostics which includes request count stats. |
I have it in 2024.11.1 What exact logging needs to be enable? |
I have 3 cameras and I turned on debug logging. I can see the 3 API calls go out, and then I see a long list of:
Before finally getting:
This is not consistent, because sometimes after the long list of "cannot handle WebRTC candidate" I will get a response and cameras will work. Cameras: |
I disabled the camera entity and that allowed me to use my thermostat again. |
i have 2 thermostats and 4 cams, my thought process is that if you have the cam or the thermostat on the dashboard, it requests every sec or something like that. In the pub/sub api and the other api enabled, there were almost about 10,000 requests in an hr alone and with 99 percent error rate coming from home assistant. when i remove most of the nest devices and say keep one cam or both thermostats, i am able to access in and out the cams and thermostats without getting the timeout error so quickly but i still do get it so def a bug in the way its requesting |
There is a bug in steam expiration handling in 2024.11.0 fixed in 2024.11.1 that would send many requests. As I mentioned above looking to understand more debug logs from 2024.11.1 if this is still happening. (Pubsub should be unrelated, nothing to do with this and errors there should be ignored as it's part of how streaming pull works.) So far the only report is about 3 requests being sent which is totally normal. |
gotcha, just checked my OS, it is currently |
@allenporter Apologies for not sending logs earlier. The camera has been showing the error for a few days now. It has not let me view the camera at any point until I restart HA and then it works for a little bit only. The logs include me attempting to open up the camera 2-3 times earlier. I am on 2024.11.1. |
ok it looks like some of the fixes in 2024.12 could help namely #130265 -- i can see from your logs once it fails, its sending one request per second to try extending the stream. That shouldn't happen anymore after the next patch release. As a workaround you can either go backwards to 2024.10 or disable camera entities. |
looks like 11.2 fixed the problems. thanks
…On Thu, Nov 14, 2024 at 23:32 Allen Porter ***@***.***> wrote:
ok it looks like some of the fixes in 2024.12 could help namely #130265
<#130265> -- i can see from
your logs once it fails, its sending one request per second to try
extending the stream. That shouldn't happen anymore after the next patch
release.
As a workaround you can either go backwards to 2024.10 or disable camera
entities.
—
Reply to this email directly, view it on GitHub
<#130061 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BM57EUD7GKTRKJKIYKIFTRL2AV2OPAVCNFSM6AAAAABRLR7BEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZXHE2DCMJXG4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Thank you all for your patience in resolving this. Appreciate the time you took in providing debug logs and details. |
11.2 seems to have solved the problem for my Nest cam but now my EZVIZ camera isn't working|! In11.1 the stream was working perfectly, coming up really quickly but now at best I get a snapshot while if I go into the device and click on the camera entity I get "Failed to start WebRTC stream: Unknown command.". I know it's not Nest related but I wonder if the fix for the Nest has caused this? And to add, the PTZ movement buttons work! |
Bad form to reply to yourself I know, sorry! Just to update on my EZVIZ camera, I've restarted HA and it's now working! A bit odd - I'll monitor it over the next few days. |
Nest cameras a lot better in 11.2 for me. I can actually see some live streaming now. I did have to start HA a second time after installing the update before the error message cleared. I'll post again when I've had longer to test. The Reolink cameras were improved right from the first version of this, as in they now stream live rather than jerkily every 5 or so seconds. |
If you have problems with other cameras and webrtc you can file a report with that integration or go2rtc |
At the end of the day the nest API still has rate limits and we can't entirely stop it, but sounds clear the root problems are addressed by fixing the expiration error handling. Thanks. |
Reolink and Nest cameras still stable here after several hours. It seems now to be working as intended. |
Can you confirm what home assistant version? It should be max 1 request every 30 seconds. |
Can you enable debug logging and capture requests being sent? |
Does below give enough information to troubleshoot?
|
These errors are normal to happen from time to time because the API has a rate limit. What we're looking for is evidence that the integration is sending repeated requests that are inappropriate. It should be backing off every time there is an error and refreshing at a slower rate. |
From the log, it does seem to be backing off as expected: Just had a thought - I have the Nest Protect Integration too, which I think uses the same API - perhaps this could be the cause? Will investigate. Edit: OK, seems it does not use Google SDM. |
I'm not aware of any ongoing issue here, so we need more details to investigate. Pleasen reopen a new issue with debug logs and diagnostics since its a different root cause. |
The problem
I am receiving the error error below since the latest HA update for my Nest doorbell camera. It also seems like cameras are also affected based on this thread https://community.home-assistant.io/t/google-nest-cameras-not-working-after-latest-home-assistant-update/791628
Failed to start WebRTC stream: Nest API error: Too Many Requests response from API (429): RESOURCE_EXHAUSTED (429): Rate limited for the ExecuteDeviceCommand API for the user.
I assume it is related to the new camera changes in the 2024.11 update. It was working fine before then.
What version of Home Assistant Core has the issue?
core-2024.11.0
What was the last working version of Home Assistant Core?
core-2024.10.4
What type of installation are you running?
Home Assistant OS
Integration causing the issue
Google Nest
Link to integration documentation on our website
https://www.home-assistant.io/integrations/nest
Diagnostics information
nest-01JB0SMBHCYG2NXA7D104GMJAP-Front door-3f2d7977e8184487d1a66382a9f3eca2.json
Example YAML snippet
No response
Anything in the logs that might be useful for us?
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: