Powerwall gets http 410 response with null data and does not update due to API change

[Viss]

Version of the custom_component

3.17.1 - current as of yesterday

Configuration

Describe the bug

I updated this integration yesterday, and noticed that the automations I have to switch my powerwalls configs around never fired, and upon investigation I found the integration had stopped working:

I am thinking that it just needs to have a token refresh happen, but it appears that I can't update the refresh token without completely removing and reconfiguring my powerwall. Is there a way to update the refresh token without tearing out the integration?

Debug log

2023-10-05 17:53:37.518 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/powerwalls/STE20201009-00492 {}
2023-10-05 17:53:37.780 DEBUG (MainThread) [teslajsonpy.connection] 410: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/powerwalls/1232100-00-E--TG1202500029KL/fullstatus => Gone","error_description":""}
2023-10-05 17:53:37.781 WARNING (MainThread) [teslajsonpy.controller] Unable to get battery data during setup, battery will still be added. 410: UNKNOWN_ERROR_410
2023-10-05 17:53:37.781 DEBUG (MainThread) [teslajsonpy.connection] Token expiration in 2:57:54
2023-10-05 17:53:37.782 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/powerwalls/STE20201009-00492/status {}
2023-10-05 17:53:37.898 DEBUG (MainThread) [teslajsonpy.connection] 410: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/powerwalls/1232100-00-E--TG1202500029KL/status => Gone","error_description":""}
2023-10-05 17:53:39.062 DEBUG (MainThread) [teslajsonpy.connection] Token expiration in 2:57:52
2023-10-05 17:53:39.062 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/powerwalls/STE20201009-00492/status {}
2023-10-05 17:53:39.157 DEBUG (MainThread) [teslajsonpy.connection] 410: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/powerwalls/1232100-00-E--TG1202500029KL/status => Gone","error_description":""}
2023-10-05 17:53:42.013 DEBUG (MainThread) [teslajsonpy.connection] Token expiration in 2:57:49
2023-10-05 17:53:42.015 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/powerwalls/STE20201009-00492/status {}
2023-10-05 17:53:42.120 DEBUG (MainThread) [teslajsonpy.connection] 410: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/powerwalls/1232100-00-E--TG1202500029KL/status => Gone","error_description":""}
2023-10-05 17:53:46.235 DEBUG (MainThread) [teslajsonpy.connection] Token expiration in 2:57:45
2023-10-05 17:53:46.236 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/powerwalls/STE20201009-00492/status {}
2023-10-05 17:53:46.328 DEBUG (MainThread) [teslajsonpy.connection] 410: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/powerwalls/1232100-00-E--TG1202500029KL/status => Gone","error_description":""}
2023-10-05 17:53:52.783 DEBUG (MainThread) [teslajsonpy.connection] Token expiration in 2:57:39
2023-10-05 17:53:52.784 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/powerwalls/STE20201009-00492/status {}
2023-10-05 17:53:53.071 DEBUG (MainThread) [teslajsonpy.connection] 410: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/powerwalls/1232100-00-E--TG1202500029KL/status => Gone","error_description":""}

[akballow]

I reinstalled with a new token and still have the issue. This seems like an API issue as the response comes back but has null info

[Viss]

oh! so its not just me!

…

On October 5, 2023 6:30:02 PM PDT, akballow ***@***.***> wrote: I reinstalled with a new token and still have the issue. This seems like an API issue as the response comes back but has null info -- Reply to this email directly or view it on GitHub: #724 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[Avatar1976]

Add me to the list, same issue, tried same fixes too.

[Avatar1976]

Further information, this specifically relates to Powerwall / Energy Site integrations.
If you delete your configuration from the devices are in Home Assistant, and then add back again but deselecting energy sites and leaving only vehicles selected. This will allow you to at least have cars visible to HA. Sad to see though as I used this to control my Powerwall a lot.

[akballow]

Further information, this specifically relates to Powerwall / Energy Site integrations. If you delete your configuration from the devices are in Home Assistant, and then add back again but deselecting energy sites and leaving only vehicles selected. This will allow you to at least have cars visible to HA. Sad to see though as I used this to control my Powerwall a lot.

Yeah even if you select both you can see in the logs that the car discovery comes in fine it just fails on getting the powerwall info and repeats a few times before it quits

[DUNKENBLITHS]

Ah so this isn't just me ! - usually i have to reboot the router when something isn't connected, but didn't fix it, so....

Just can't access the car through HA, will just use phone now, as I need all PowerWall control

[SalisburyLad]

It just appears to be Powerwall changing operational mode for me. Backup reserve and grid charging are working. I only have Powerwall.

[quietlikeninja]

Mine stopped talking to my powerwall about 6 hours ago. I saw there was an update to kicked that off but now have this issue :-(

I only noticed when I saw I had exported 11kwh of solar at -4c/kWh because my automation thought I needed an extra 5kWh to charge my battery as it was only at 37% :-(

[Viss]

i wonder if theres a way to change the powerwall settings locally, without having to use the api, making this integration not have to use cloud...

…

On October 5, 2023 7:10:14 PM PDT, Avatar1976 ***@***.***> wrote: Add me to the list, same issue, tried same fixes too. -- Reply to this email directly or view it on GitHub: #724 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[Viss]

oh it wrecked the whole thing for me. im using this integration to draw all my charts for consumption and solar - everything. basically my whole electric setup went offline

…

On October 5, 2023 9:19:31 PM PDT, SalisburyLad ***@***.***> wrote: It just appears to be changing operational mode for me. Backup reserve and grid charging are working. -- Reply to this email directly or view it on GitHub: #724 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[DUNKENBLITHS]

With regards to the Power wall.. use the other interface, thats still working fine...

[akballow]

With regards to the Power wall.. use the other interface, thats still working fine...

I use both the official one reads data straight from the gateway. The custom one is the api to the app which gives you the ability to modify the settings which the official one never can do since you can not make those changes if you visit your gateway web ui even.

[akballow]

i wonder if theres a way to change the powerwall settings locally, without having to use the api, making this integration not have to use cloud...
…
On October 5, 2023 7:10:14 PM PDT, Avatar1976 @.> wrote: Add me to the list, same issue, tried same fixes too. -- Reply to this email directly or view it on GitHub: #724 (comment) You are receiving this because you authored the thread. Message ID: @.>

I wish I feel like the app has like 80% success rate to make changes. If it was all local it would be amazing.

[quietlikeninja]

With regards to the Power wall.. use the other interface, thats still working fine...

You are now my hero :-)

Solar management is back online for me

[daniel-simpson]

Same issue for me. Logged on to the powerwall local portal directly to get some diagnostics. I'm assuming it's probably a system update that has knocked it out, so here's my "Customer Version": 23.28.2 27626f98

When looking at the local Powerwall's API it looks like they're spruiking a new "Tesla Pros" app to commission new Powerwall installs... so I'd guess there is a decent chance some breaking changes have occurred in the APIs this code is using 😭

[andrew2408]

Also stopped working for me as of 8.40am Brisbane time.

Interestingly I have a third party 'Powerwall' app on my phone that I can only assume relies on the same cloud API and it is updating fine still. Does anyone have any other set ups that rely on the cloud API to see if these are down to (i.e. is it an issue with the API or the integration?)

[Viss]

i spent a bit of time troubleshooting.
this integration calls a host named "powergate.prd.sn.tesla.services".

All of those subdomains stopped working, only the apex domain 'tesla.services' remains.

So this integration has broken because the host it phones home to so that it can interact with the tesla API is "just gone". We'll have to disassemble the new APK or do a pcap or dns dump of it so that we can fish out the new host it's talking to.

[ChillV2L]

was this after the core update for you all above ? this has also happened to me after the core update but it could just be a coincidence

[bmsrecycle]

This problem has seemed to happen with the Home Assistant 2023.10.0 update, I'm having the same issue.

[NickSills1]

This issue is not related to a Home Assistant update. Same for me and I'm on 2023.7.3.

[Viss]

it happened to me today, I think it was right after I updated this module. though i wager thats just a coincidence, because the debug logs say its trying to poke a host to do an api call, and that host straight up doesnt exist. at first, glance it appears telsa rugged us all.

On 2023-10-06 00:25, NickSills1 wrote: This issue isnot related to a Home Assistant update. Same for me and I'm on 2023.7.3. -- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you authored the thread.Message ID: ***@***.***>

Links: ------ [1] #724 (comment) [2] https://github.com/notifications/unsubscribe-auth/AANMYX6FTZONEDW6SNVYZELX56W7DAVCNFSM6AAAAAA5VCLEBCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQGEYTANJXGA

[JasSmiths]

Another +1 here which is not unexpected if they’ve killed the targets.

Fingers crossed this can sorted as my setup depends on it

[jaymunro]

+1

[Duff-Box1]

Same issue, any fix's would be much appreciated as I also drive many automations from this.

[gkwok1]

My changes have been accepted and it's now merged with the master. https://github.com/zabuldon/teslajsonpy/releases/tag/v3.9.5

@alandtse I am not sure what the process is now for HA to pick it up.

[alandtse]

My changes have been accepted and it's now merged with the master. zabuldon/teslajsonpy@v3.9.5 (release)

@alandtse I am not sure what the process is now for HA to pick it up.

Teslajsonpy must be bumped. For example d704473. You can't really do it in GH interface as we expect the build environment to be bumped too using poetry. I'll get to it eventually; I'm working on other things.

[jaymunro]

The update in HACS is live and working.

Operation mode is now working and the false zero readings on SoC appear gone.

[mcdhome]

My issue is probably something Tesla did. Gkwok1 fix works but have an issue. I have one what I think is called a energy site that is not showing up like it used to. The one that shows up is my 3 solar arrays+batteries combined into the Gateway. I have one of those "gateway" little ethernet devices for it that sends data back to the mothership. But I also have another one for the original solar that was on the house. It shows up in the Tesla app as another entity I scroll to and I can see how much that original solar alone is producing as part of the combined total in the other "site.". It no longer shows up though in HA, but does still show up in my Tesla Phone App. I suspect Tesla has hid the entity under some additional structure maybe has to be queried differently, or it is being hidden from the API somehow. But since it shows up in the Tesla App, my guess is they hide under some new structure in the API.

EDIT: This was fixed in an additional teslajsonpy update gkwow1 did that fixed an issue with solarcity installs with no powerwall.

[JohnKG72]

Now the integration has been updated, should I remove the teslajsonpy folder from root /config?

[d0dge]

All working here. Massive thanks to everyone involved in fixing this so quickly!

[Toolroom1]

A huge thank you to all those that got this working again, very happy, well done.

[PaddyLambert]

Thanks to all who got this sorted so swiftly. Much appreciated.

[JasonRS1969]

Now the integration has been updated, should I remove the teslajsonpy folder from root /config?

Ye you should, it's mentioned in this comment here:

#724 (comment)

Thanks to everyone who assisted in the quick resolution & release here. Get community effort.

[Duff-Box1]

Really impressive effort by everyone thank you, works beautifully again. What was interesting in this thread is all the unique ways everyone is using their power-walls and this integration to do cool things and save energy. Maybe there is a location to share more of this info to help others.
Thanks again contributors to the solution.

[Alona23]

Seeing how everyone is using HS integrations with their powerwalls would be fantastic Alan Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Duff-Box1 ***@***.***> Sent: Sunday, October 8, 2023 10:48:05 AM To: alandtse/tesla ***@***.***> Cc: Alona23 ***@***.***>; Mention ***@***.***> Subject: Re: [alandtse/tesla] Powerwall gets http 410 response with null data and does not update due to API change (Issue #724) Really impressive effort by everyone thank you, works beautifully again. What was interesting in this thread is all the unique ways everyone is using their power-walls and this integration to do cool things and save energy. Maybe there is a location to share more of this info to help others. Thanks again contributors to the solution. — Reply to this email directly, view it on GitHub<#724 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A6BA7KN4CN76NLJTZII4JALX6JZFLAVCNFSM6AAAAAA5VCLEBCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJRHE3TSMZYGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[JasSmiths]

Agreed, just setup this Facebook group for it - All welcome to join
https://www.facebook.com/groups/220993527434751/

[daniel-simpson]

Probably makes more sense to have it on the Home Assistant forums rather than facebook @JasSmiths.

I've created a thread on the Home Assistant forums, and will write up a brief overview of a few of my automations a bit later today:
https://community.home-assistant.io/t/tesla-powerwall-use-cases-and-automation-sharing/623950

Thanks again to everyone involved in sorting this out so quickly, and to the repo maintainer alandtse 💌

[GeorgeIoak]

Has anyone experienced this error in HA:

2023-10-08 13:27:42.889 ERROR (MainThread) [custom_components.tesla_custom] Unexpected error fetching tesla_custom data: 2252262012599371
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 290, in _async_refresh
self.data = await self._async_update_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/tesla_custom/__init__.py", line 440, in _async_update_data
return await self.controller.update(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/teslajsonpy/controller.py", line 825, in update
return any(await asyncio.gather(*tasks))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/teslajsonpy/controller.py", line 716, in _get_and_process_site_summary
self._site_summary[energysite_id].update(response)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
KeyError: 2252262012599371

I just had my inverter replaced from Solar Edge to 2 Tesla Inverters so this is a new configuration. I was able to use pypowerwall from the CL and retrieve data but that doing a direct communication and not going through the cloud and using the Token.

[alandtse]

Oh you could use the wiki.

[GeorgeIoak]

@alandtse I have looked but don't see anything related to this, did I miss something?

[alandtse]

Please do not ping me. I'm not your tech support. And the issue is you jumped in a closed thread with an off topic post. Hence that is why you're confused.

[gkwok1]

@GeorgeIoak Delete your integration and add it back in, you are trying to poll an ID that is not linked to your user.

[GeorgeIoak]

@gkwok1 Thank for not biting my head off and replying with some helpful information. I saw all of the other comments and truly was just curious if this another byproduct of the current problem or if my issue was completely unrelated.

Where is a better place to discuss this if I continue to have issues, here in the discussions or perhaps in the HA community?

[gkwok1]

@GeorgeIoak Discussion would probably be the best place to start.

https://github.com/alandtse/tesla/discussions

I am not an active participant here but wanted to give it a shot to fix the integration since it's something I rely on.

[mcdhome]

The Tesla Powerwall HA plugin was having endpoint issues as well, and the Set Powerwall operation mode was failing due to endpoint urls.
I copied the teslapy (not telsajsonpy) from this repo into my main /root/config/ folder as was done here for teslajsonpy by gwok1, and the set operation mode of my powerwalls is working again. Will comment in the HA app too. When they fix/update the main Tesla Powerwall integration, I will be sure to nuke this folder out of the config dir.
https://github.com/tdorssers/TeslaPy/tree/master

It has an updated enpoints.json file. Copy the inner folder teslapy into the /root/config/, don't copy the whole github repo download, just the inner folder named teslapy.

I probably submitted wrong and likely they already knew, but if it helps. home-assistant/core#102086

[skipishere]

The Tesla Powerwall HA plugin was having endpoint issues as well, and the Set Powerwall operation mode was failing due to endpoint urls.
I copied the teslapy (not telsajsonpy) from this repo into my main /root/config/ folder as was done here for teslajsonpy by gwok1, and the set operation mode of my powerwalls is working again. Will comment in the HA app too. When they fix/update the main Tesla Powerwall integration, I will be sure to nuke this folder out of the config dir.
https://github.com/tdorssers/TeslaPy/tree/master

It has an updated enpoints.json file. Copy the inner folder teslapy into the /root/config/, don't copy the whole github repo download, just the inner folder named teslapy.

I probably submitted wrong and likely they already knew, but if it helps. home-assistant/core#102086

I'd recommend if you're still having issues that you open a new issue, as it's unlikely related to this issue given the API changes and the applied fix working for everyone else.