Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Frequent gaps in InfluxDB and errors in pypowerwall #449

Closed
jgleigh opened this issue Mar 15, 2024 · 84 comments
Closed

Frequent gaps in InfluxDB and errors in pypowerwall #449

jgleigh opened this issue Mar 15, 2024 · 84 comments

Comments

@jgleigh
Copy link

jgleigh commented Mar 15, 2024

I've noticed that lately I'm seeing a lot more missing data in InfluxDB. I run tesla-history periodically to clean things up. I'm guessing this is related to 23.44.0 firmware, but hard to tell. Dashboard up-to-date and running 4.1.1.

Searching InfluxDB for data gaps (power usage)

  • Found data gap: [2024-03-10 20:32:00-07:00] - [2024-03-10 20:49:00-07:00] (0:17:00s)
  • Found data gap: [2024-03-10 23:31:00-07:00] - [2024-03-10 23:52:00-07:00] (0:21:00s)
  • Found data gap: [2024-03-11 00:47:00-07:00] - [2024-03-11 01:07:00-07:00] (0:20:00s)
  • Found data gap: [2024-03-11 03:02:00-07:00] - [2024-03-11 03:12:00-07:00] (0:10:00s)
  • Found data gap: [2024-03-11 21:32:00-07:00] - [2024-03-11 21:43:00-07:00] (0:11:00s)
  • Found data gap: [2024-03-14 01:02:00-07:00] - [2024-03-14 01:39:00-07:00] (0:37:00s)
  • Found data gap: [2024-03-14 08:30:00-07:00] - [2024-03-14 08:39:00-07:00] (0:09:00s)
  • Found data gap: [2024-03-14 18:30:00-07:00] - [2024-03-14 18:39:00-07:00] (0:09:00s)
  • Found data gap: [2024-03-14 20:01:00-07:00] - [2024-03-14 20:08:00-07:00] (0:07:00s)
  • Found data gap: [2024-03-14 21:15:00-07:00] - [2024-03-14 21:34:00-07:00] (0:19:00s)

2024-03-10 15:31:34 03/10/2024 03:31:34 PM [proxy] [INFO] pyPowerwall [0.7.9] Proxy Server [t42] - HTTP Port 8675
2024-03-10 15:31:34 03/10/2024 03:31:34 PM [proxy] [INFO] pyPowerwall Proxy Started
2024-03-10 15:31:36 03/10/2024 03:31:36 PM [proxy] [INFO] pyPowerwall Proxy Server - Local Mode
2024-03-10 15:31:36 03/10/2024 03:31:36 PM [proxy] [INFO] Connected to Energy Gateway 192.168.1.6 (Home Energy Gateway)

Every gap aligns with these messages in the pypowerwall log.

2024-03-10 20:32:10 03/10/2024 08:32:10 PM [proxy] [ERROR] Socket broken sending API response to client [doGET]
... repeats over and over
2024-03-10 20:36:20 03/10/2024 08:36:20 PM [proxy] [ERROR] Socket broken sending API response to client [doGET]
2024-03-10 20:49:30 ----------------------------------------
2024-03-10 20:49:30 Exception occurred during processing of request from ('172.18.0.6', 34336)
2024-03-10 20:49:30 ----------------------------------------
2024-03-10 20:49:30 Exception occurred during processing of request from ('172.18.0.6', 34362)
2024-03-10 20:49:30 Traceback (most recent call last):
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:30 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:30 return self._find_no_duplicates(name)
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:30 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:30 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:30
2024-03-10 20:49:30 During handling of the above exception, another exception occurred:
2024-03-10 20:49:30
2024-03-10 20:49:30 Traceback (most recent call last):
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:30 self.finish_request(request, client_address)
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:30 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:30 self.handle()
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:30 self.handle_one_request()
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:30 method()
2024-03-10 20:49:30 File "/app/server.py", line 307, in do_GET
2024-03-10 20:49:30 d = pw.system_status() or {}
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status
2024-03-10 20:49:30 payload = self.poll('/api/system_status', jsonformat=True)
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:30 self._get_session()
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:30 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:30 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:30 ----------------------------------------
2024-03-10 20:49:30 Traceback (most recent call last):
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:30 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:30 return self._find_no_duplicates(name)
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:30 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:30 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:30
2024-03-10 20:49:30 During handling of the above exception, another exception occurred:
2024-03-10 20:49:30
2024-03-10 20:49:30 Traceback (most recent call last):
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:30 self.finish_request(request, client_address)
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:30 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:30 self.handle()
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:30 self.handle_one_request()
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:30 method()
2024-03-10 20:49:30 File "/app/server.py", line 189, in do_GET
2024-03-10 20:49:30 message = pw.poll('/api/meters/aggregates')
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:30 self._get_session()
2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:30 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:30 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:30 ----------------------------------------
2024-03-10 20:49:31 ----------------------------------------
2024-03-10 20:49:31 Exception occurred during processing of request from ('172.18.0.6', 34364)
2024-03-10 20:49:31 Traceback (most recent call last):
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:31 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:31 return self._find_no_duplicates(name)
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:31 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:31 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:31
2024-03-10 20:49:31 During handling of the above exception, another exception occurred:
2024-03-10 20:49:31
2024-03-10 20:49:31 Traceback (most recent call last):
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:31 self.finish_request(request, client_address)
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:31 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:31 self.handle()
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:31 self.handle_one_request()
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:31 method()
2024-03-10 20:49:31 File "/app/server.py", line 192, in do_GET
2024-03-10 20:49:31 message = pw.poll('/api/system_status/soe')
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:31 self._get_session()
2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:31 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:31 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:31 ----------------------------------------
2024-03-10 20:49:35 ----------------------------------------
2024-03-10 20:49:35 Exception occurred during processing of request from ('172.18.0.6', 38378)
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:35 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:35 return self._find_no_duplicates(name)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:35 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:35 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:35
2024-03-10 20:49:35 During handling of the above exception, another exception occurred:
2024-03-10 20:49:35
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:35 self.finish_request(request, client_address)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:35 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:35 self.handle()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:35 self.handle_one_request()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:35 method()
2024-03-10 20:49:35 File "/app/server.py", line 307, in do_GET
2024-03-10 20:49:35 d = pw.system_status() or {}
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status
2024-03-10 20:49:35 payload = self.poll('/api/system_status', jsonformat=True)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:35 self._get_session()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:35 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:35 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:35 ----------------------------------------
2024-03-10 20:49:35 ----------------------------------------
2024-03-10 20:49:35 Exception occurred during processing of request from ('172.18.0.6', 38372)
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:35 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:35 return self._find_no_duplicates(name)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:35 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:35 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:35
2024-03-10 20:49:35 During handling of the above exception, another exception occurred:
2024-03-10 20:49:35
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 ----------------------------------------
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:35 self.finish_request(request, client_address)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:35 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:35 self.handle()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:35 self.handle_one_request()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:35 method()
2024-03-10 20:49:35 Exception occurred during processing of request from ('172.18.0.6', 38350)
2024-03-10 20:49:35 File "/app/server.py", line 192, in do_GET
2024-03-10 20:49:35 message = pw.poll('/api/system_status/soe')
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:35 self._get_session()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:35 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:35 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:35 ----------------------------------------
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:35 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:35 return self._find_no_duplicates(name)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:35 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:35 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:35
2024-03-10 20:49:35 During handling of the above exception, another exception occurred:
2024-03-10 20:49:35
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:35 self.finish_request(request, client_address)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:35 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:35 self.handle()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:35 self.handle_one_request()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:35 method()
2024-03-10 20:49:35 File "/app/server.py", line 269, in do_GET
2024-03-10 20:49:35 d = pw.system_status() or {}
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status
2024-03-10 20:49:35 payload = self.poll('/api/system_status', jsonformat=True)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:35 self._get_session()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:35 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:35 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:35 ----------------------------------------
2024-03-10 20:49:35 ----------------------------------------
2024-03-10 20:49:35 Exception occurred during processing of request from ('172.18.0.6', 34338)
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 537, in _make_request
2024-03-10 20:49:35 response = conn.getresponse()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 466, in getresponse
2024-03-10 20:49:35 httplib_response = super().getresponse()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/client.py", line 1375, in getresponse
2024-03-10 20:49:35 response.begin()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/client.py", line 318, in begin
2024-03-10 20:49:35 version, status, reason = self._read_status()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/client.py", line 279, in _read_status
2024-03-10 20:49:35 line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socket.py", line 705, in readinto
2024-03-10 20:49:35 return self._sock.recv_into(b)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/ssl.py", line 1307, in recv_into
2024-03-10 20:49:35 return self.read(nbytes, buffer)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/ssl.py", line 1163, in read
2024-03-10 20:49:35 return self._sslobj.read(len, buffer)
2024-03-10 20:49:35 TimeoutError: The read operation timed out
2024-03-10 20:49:35
2024-03-10 20:49:35 The above exception was the direct cause of the following exception:
2024-03-10 20:49:35
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
2024-03-10 20:49:35 resp = conn.urlopen(
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-03-10 20:49:35 retries = retries.increment(
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
2024-03-10 20:49:35 raise reraise(type(error), error, _stacktrace)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
2024-03-10 20:49:35 raise value
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-03-10 20:49:35 response = self._make_request(
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 539, in _make_request
2024-03-10 20:49:35 self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
2024-03-10 20:49:35 raise ReadTimeoutError(
2024-03-10 20:49:35 urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='192.168.1.6', port=443): Read timed out. (read timeout=5)
2024-03-10 20:49:35
2024-03-10 20:49:35 During handling of the above exception, another exception occurred:
2024-03-10 20:49:35
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 197, in _get_session
2024-03-10 20:49:35 r = self.session.post(url,data = pload, verify=False, timeout=self.timeout)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
2024-03-10 20:49:35 return self.request("POST", url, data=data, json=json, **kwargs)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
2024-03-10 20:49:35 resp = self.send(prep, **send_kwargs)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
2024-03-10 20:49:35 r = adapter.send(request, **kwargs)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 532, in send
2024-03-10 20:49:35 raise ReadTimeout(e, request=request)
2024-03-10 20:49:35 requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='192.168.1.6', port=443): Read timed out. (read timeout=5)
2024-03-10 20:49:35
2024-03-10 20:49:35 During handling of the above exception, another exception occurred:
2024-03-10 20:49:35
2024-03-10 20:49:35 Traceback (most recent call last):
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:35 self.finish_request(request, client_address)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:35 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:35 self.handle()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:35 self.handle_one_request()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:35 method()
2024-03-10 20:49:35 File "/app/server.py", line 269, in do_GET
2024-03-10 20:49:35 d = pw.system_status() or {}
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status
2024-03-10 20:49:35 payload = self.poll('/api/system_status', jsonformat=True)
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:35 self._get_session()
2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 202, in _get_session
2024-03-10 20:49:35 raise ConnectionError(err)
2024-03-10 20:49:35 pypowerwall.ConnectionError: Unable to connect to Powerwall at https://192.168.1.6
2024-03-10 20:49:35 ----------------------------------------
2024-03-10 20:49:36 ----------------------------------------
2024-03-10 20:49:36 Exception occurred during processing of request from ('172.18.0.6', 38394)
2024-03-10 20:49:36 Traceback (most recent call last):
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:36 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:36 return self._find_no_duplicates(name)
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:36 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:36 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:36
2024-03-10 20:49:36 During handling of the above exception, another exception occurred:
2024-03-10 20:49:36
2024-03-10 20:49:36 Traceback (most recent call last):
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:36 self.finish_request(request, client_address)
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:36 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:36 self.handle()
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:36 self.handle_one_request()
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:36 method()
2024-03-10 20:49:36 File "/app/server.py", line 189, in do_GET
2024-03-10 20:49:36 message = pw.poll('/api/meters/aggregates')
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:36 self._get_session()
2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:36 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:36 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:36 ----------------------------------------
2024-03-10 20:49:40 ----------------------------------------
2024-03-10 20:49:40 Exception occurred during processing of request from ('172.18.0.6', 38440)
2024-03-10 20:49:40 Traceback (most recent call last):
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:40 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:40 return self._find_no_duplicates(name)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:40 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:40 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:40
2024-03-10 20:49:40 During handling of the above exception, another exception occurred:
2024-03-10 20:49:40
2024-03-10 20:49:40 Traceback (most recent call last):
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:40 self.finish_request(request, client_address)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:40 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:40 self.handle()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:40 self.handle_one_request()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:40 method()
2024-03-10 20:49:40 File "/app/server.py", line 192, in do_GET
2024-03-10 20:49:40 message = pw.poll('/api/system_status/soe')
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:40 self._get_session()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:40 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:40 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:40 ----------------------------------------
2024-03-10 20:49:40 ----------------------------------------
2024-03-10 20:49:40 Exception occurred during processing of request from ('172.18.0.6', 38422)
2024-03-10 20:49:40 Traceback (most recent call last):
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:40 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:40 return self._find_no_duplicates(name)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:40 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:40 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:40
2024-03-10 20:49:40 During handling of the above exception, another exception occurred:
2024-03-10 20:49:40
2024-03-10 20:49:40 Traceback (most recent call last):
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:40 self.finish_request(request, client_address)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:40 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:40 self.handle()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:40 self.handle_one_request()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:40 method()
2024-03-10 20:49:40 File "/app/server.py", line 307, in do_GET
2024-03-10 20:49:40 d = pw.system_status() or {}
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status
2024-03-10 20:49:40 payload = self.poll('/api/system_status', jsonformat=True)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:40 self._get_session()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:40 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:40 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:40 ----------------------------------------
2024-03-10 20:49:40 ----------------------------------------
2024-03-10 20:49:40 Exception occurred during processing of request from ('172.18.0.6', 38400)
2024-03-10 20:49:40 Traceback (most recent call last):
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session
2024-03-10 20:49:40 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem
2024-03-10 20:49:40 return self._find_no_duplicates(name)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates
2024-03-10 20:49:40 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
2024-03-10 20:49:40 KeyError: "name='AuthCookie', domain=None, path=None"
2024-03-10 20:49:40
2024-03-10 20:49:40 During handling of the above exception, another exception occurred:
2024-03-10 20:49:40
2024-03-10 20:49:40 Traceback (most recent call last):
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
2024-03-10 20:49:40 self.finish_request(request, client_address)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
2024-03-10 20:49:40 self.RequestHandlerClass(request, client_address, self)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init
2024-03-10 20:49:40 self.handle()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
2024-03-10 20:49:40 self.handle_one_request()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-03-10 20:49:40 method()
2024-03-10 20:49:40 File "/app/server.py", line 269, in do_GET
2024-03-10 20:49:40 d = pw.system_status() or {}
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status
2024-03-10 20:49:40 payload = self.poll('/api/system_status', jsonformat=True)
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll
2024-03-10 20:49:40 self._get_session()
2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session
2024-03-10 20:49:40 raise LoginError("Invalid Powerwall Login")
2024-03-10 20:49:40 pypowerwall.LoginError: Invalid Powerwall Login
2024-03-10 20:49:40 ----------------------------------------
2024-03-10 20:49:44 03/10/2024 08:49:44 PM [proxy] [ERROR] Socket broken sending API response to client [doGET]

And then everything returns to normal until the next data gap.

@jgleigh
Copy link
Author

jgleigh commented Mar 15, 2024

Potentially unrelated, but I had to reboot my Gateway today. It completely locked up and was offline for an hour before I noticed. Hadn't done that in a very long time (ever since I ditched WiFi).

@jgleigh
Copy link
Author

jgleigh commented Mar 15, 2024

Definitely seems like a flaw in the Gateway design that it doesn't have a watchdog timer to reboot itself when it gets stuck. Gateway could be offline for a considerable amount of time if people don't watch it as closely as we tend to.

@jasonacox
Copy link
Owner

pypowerwall.LoginError: Invalid Powerwall Login

I have seen this during upgrades or when the gateway starts rate limiting you. If it sees too many connections or requests, it will lock you out. However, it will clear within 10 minutes if you stop pypowerwall. I assume you are only using the standard pypowerwall to access the gateway? If so, you may want to adjust the cache or max pool size by adding these lines to pypowerwall.env:

PW_POOL_MAXSIZE=15
PW_CACHE_EXPIRE=5

I would only adjust one at a time. Specifically: Try to decrease the pool size to 12 or 10. If that doesn't help, try increasing the cache to 10 (seconds).

These default settings have been working through many different upgrades but perhaps something happened with 23.44.0 that makes it more sensitive to local API calls.

@jgleigh
Copy link
Author

jgleigh commented Mar 16, 2024

I was using 15 for the pool size, so I'm going to try 10 and see if that helps. I found the old threads where this was actually made the default based on my feedback. :)

#261

@jasonacox
Copy link
Owner

That's right, you're the troublemaker?! 😂

Reducing to 10 could resurrect old errors again (pool cycling) but may reduce the rate limiting (if that is indeed what is happening). The other possibility is that requesting a 404 API endpoint (e.g. /api/device/vitals) trips the rate limit. Right now pypowerwall will continue to retry that API. We could add logic to detect Firmware version or the 404 response and either stop trying or cache it for a longer period.

@jgleigh
Copy link
Author

jgleigh commented Mar 16, 2024

Good thought about vitals. I hadn't been watching it that closely to see when the data dropouts started, but it's very possible they align with 23.44.0.

A little surprised you haven't been upgraded yet.

@jasonacox
Copy link
Owner

@jgleigh If you want to try it, I updated pypowerwall to cache the 404 responses and not retry them for 10 minutes. You can test it by editing the powerwall.yml and updating the version of pypowerwall to: jasonacox/pypowerwall:0.7.10t42

    pypowerwall:
        image: jasonacox/pypowerwall:0.7.10t42
# Restart
./compose-dash.sh up -d

I'l run some test and unless there are issues, push it as v4.1.2 for everyone else.

jasonacox added a commit that referenced this issue Mar 16, 2024
@BuongiornoTexas
Copy link
Contributor

@jasonacox, as an alternative to caching 404, would it be possible to test if version is above 23.44 and skip the call to vitals altogether. I know you don't want to give up on it yet, but right now it looks as though vitals is not coming back, so I'm not sure what we get from caching the results of a call we know is failing?

I've just checked my logs and I'm seeing the same intermittent broken sockets as @jgleigh - I've turned on debug to see if I get more on what is happening at this point.

@jasonacox
Copy link
Owner

@BuongiornoTexas excellent point. I'll add that. Let me know what you see in debug mode. I'm also going to add detection logic for a 429 "Too Many Requests" HTTP response and build in backoff logic similar to the 404 response. That would help during high CPU events (e.g. during Firmware upgrades) as well.

@BuongiornoTexas
Copy link
Contributor

Here are the broken socket errors from the last couple of hours and a couple of lines each side. I'm not seeing a specific pattern in URLs - arguably it's a bit concerning that it is not tied to any specific url aside from vitals (which we already know is not working as it used to).

I also haven't seen an Exception yet, so I'll leave it running to see if I can get more info (there were definitely some Exceptions in the log before I restarted in debug mode).

03/17/2024 08:44:05 AM [proxy] [DEBUG] 172.18.0.4 "GET /api/system_status HTTP/1.1" 200 -
03/17/2024 08:44:05 AM [proxy] [DEBUG] 172.18.0.4 "GET /freq HTTP/1.1" 200 -
03/17/2024 08:44:05 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/devices/vitals
03/17/2024 08:44:05 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 08:44:05 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 08:44:05 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 08:44:06 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
--
03/17/2024 08:44:11 AM [proxy] [DEBUG] 172.18.0.4 "GET /freq HTTP/1.1" 200 -
03/17/2024 08:44:11 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/system_status/grid_status
03/17/2024 08:44:11 AM [pypowerwall] [DEBUG] ERROR unable to parse payload for grid_status: None
03/17/2024 08:44:11 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 08:44:11 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 08:44:11 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 08:44:15 AM [proxy] [DEBUG] 172.18.0.4 "GET /strings HTTP/1.1" 200 -
--
03/17/2024 09:05:00 AM [proxy] [DEBUG] 172.18.0.4 "GET /alerts/pw HTTP/1.1" 200 -
03/17/2024 09:05:00 AM [proxy] [DEBUG] 172.18.0.4 "GET /api/meters/solar HTTP/1.1" 200 -
03/17/2024 09:05:00 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/meters/site
03/17/2024 09:05:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:05:00 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:05:00 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:05:00 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
--
03/17/2024 09:05:05 AM [proxy] [DEBUG] 172.18.0.4 "GET /api/system_status HTTP/1.1" 200 -
03/17/2024 09:05:05 AM [proxy] [DEBUG] 172.18.0.4 "GET /api/meters/solar HTTP/1.1" 200 -
03/17/2024 09:05:05 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/devices/vitals
03/17/2024 09:05:05 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:05:05 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:05:05 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:05:05 AM [proxy] [DEBUG] 172.18.0.4 "GET /api/meters/site HTTP/1.1" 200 -
--
03/17/2024 09:08:30 AM [proxy] [DEBUG] 172.18.0.4 "GET /api/meters/site HTTP/1.1" 200 -
03/17/2024 09:08:30 AM [proxy] [DEBUG] 172.18.0.4 "GET /pod HTTP/1.1" 200 -
03/17/2024 09:08:30 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/system_status/soe
03/17/2024 09:08:30 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:08:30 AM [proxy] [DEBUG] 172.18.0.4 "GET /temps/pw HTTP/1.1" 200 -
03/17/2024 09:08:30 AM [proxy] [DEBUG] 172.18.0.4 "GET /aggregates HTTP/1.1" 200 -
03/17/2024 09:08:30 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
--
03/17/2024 09:17:50 AM [proxy] [DEBUG] 172.18.0.4 "GET /strings HTTP/1.1" 200 -
03/17/2024 09:17:50 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/devices/vitals
03/17/2024 09:17:50 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:17:50 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:17:50 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:17:50 AM [proxy] [DEBUG] 172.18.0.1 "GET /api/sitemaster HTTP/1.1" 200 -
03/17/2024 09:17:50 AM [proxy] [DEBUG] 172.18.0.1 "GET /api/system_status/soe HTTP/1.1" 200 -
--
03/17/2024 09:24:40 AM [proxy] [DEBUG] 172.18.0.4 "GET /alerts/pw HTTP/1.1" 200 -
03/17/2024 09:24:40 AM [proxy] [DEBUG] 172.18.0.4 "GET /aggregates HTTP/1.1" 200 -
03/17/2024 09:24:40 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/devices/vitals
03/17/2024 09:24:40 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:24:40 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:24:40 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:24:40 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
--
03/17/2024 09:46:30 AM [proxy] [DEBUG] 172.18.0.4 "GET /strings HTTP/1.1" 200 -
03/17/2024 09:46:30 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/meters/site
03/17/2024 09:46:30 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:46:30 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:46:30 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:46:30 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:46:30 AM [proxy] [DEBUG] 172.18.0.1 "GET /api/sitemaster HTTP/1.1" 200 -
--
03/17/2024 09:50:51 AM [proxy] [DEBUG] 172.18.0.4 "GET /temps/pw HTTP/1.1" 200 -
03/17/2024 09:50:51 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:50:51 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/operation
03/17/2024 09:50:51 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:50:53 AM [proxy] [DEBUG] 172.18.0.1 "GET /api/sitemaster HTTP/1.1" 200 -
03/17/2024 09:50:53 AM [proxy] [DEBUG] 172.18.0.1 "GET /api/meters/aggregates HTTP/1.1" 200 -
03/17/2024 09:50:53 AM [proxy] [DEBUG] 172.18.0.1 "GET /api/system_status/soe HTTP/1.1" 200 -
--
03/17/2024 09:50:55 AM [proxy] [DEBUG] 172.18.0.4 "GET /freq HTTP/1.1" 200 -
03/17/2024 09:50:55 AM [proxy] [DEBUG] 172.18.0.4 "GET /api/meters/solar HTTP/1.1" 200 -
03/17/2024 09:50:55 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/devices/vitals
03/17/2024 09:50:55 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:50:55 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:50:55 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:50:55 AM [proxy] [DEBUG] 172.18.0.4 "GET /api/meters/site HTTP/1.1" 200 -
--
03/17/2024 09:54:15 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/system_status
03/17/2024 09:54:15 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:54:15 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:54:15 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 09:54:15 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 09:54:15 AM [proxy] [DEBUG] 172.18.0.1 "GET /api/meters/aggregates HTTP/1.1" 200 -
03/17/2024 09:54:15 AM [proxy] [DEBUG] 172.18.0.1 "GET /api/sitemaster HTTP/1.1" 200 -
--
03/17/2024 10:01:55 AM [proxy] [DEBUG] 172.18.0.4 "GET /alerts/pw HTTP/1.1" 200 -
03/17/2024 10:01:55 AM [proxy] [DEBUG] 172.18.0.4 "GET /aggregates HTTP/1.1" 200 -
03/17/2024 10:01:55 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/meters/aggregates
03/17/2024 10:01:55 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 10:01:55 AM [proxy] [DEBUG] 172.18.0.4 "GET /temps/pw HTTP/1.1" 200 -
03/17/2024 10:01:55 AM [proxy] [DEBUG] 172.18.0.4 "GET /freq HTTP/1.1" 200 -
03/17/2024 10:01:55 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
--
03/17/2024 10:04:00 AM [proxy] [DEBUG] 172.18.0.4 "GET /freq HTTP/1.1" 200 -
03/17/2024 10:04:00 AM [proxy] [DEBUG] 172.18.0.4 "GET /pod HTTP/1.1" 200 -
03/17/2024 10:04:00 AM [pypowerwall] [DEBUG] ERROR Timeout waiting for Powerwall API https://192.168.25.11/api/devices/vitals
03/17/2024 10:04:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]
03/17/2024 10:04:00 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 10:04:00 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals
03/17/2024 10:04:00 AM [pypowerwall] [DEBUG] 404 Powerwall API not found at https://192.168.25.11/api/devices/vitals

@jasonacox
Copy link
Owner

Thanks @BuongiornoTexas - Did you switch to jasonacox/pypowerwall:0.7.10t42 ?

@BuongiornoTexas
Copy link
Contributor

| Did you switch to jasonacox/pypowerwall:0.7.10t42 ?

Not yet - I thought it was worth checking if I could get more details on the cause/trigger for the auth Exception @jgleigh is seeing before moving on to the new container.

@jgleigh
Copy link
Author

jgleigh commented Mar 16, 2024

I've got 0.7.10t42 running. All good so far.

@jasonacox
Copy link
Owner

I updated pypowerwall to detect the Firmware version > 23.44 which will cause it to disable API calls to vitals. I also added logic to detect rate limiting and activate a cooldown period (5m). Instead of serving up cached data during that cooldown, I'm sending back None to create a true representation of the state. It will likely still result in the proxy throwing some errors, but it should help the Powerwall gateway recover more quickly and reduce the timeouts.

You can test it by editing the powerwall.yml and updating the version of pypowerwall to: jasonacox/pypowerwall:0.7.11t42

    pypowerwall:
        image: jasonacox/pypowerwall:0.7.11t42
# Restart
./compose-dash.sh up -d

# Check logs - you should see a Firmware detection a vitals disabled notice if all goes well
docker logs pypowerwall -f

@jgleigh
Copy link
Author

jgleigh commented Mar 17, 2024

Here's the output:

2024-03-16 18:10:50 03/16/2024 06:10:50 PM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/devices/vitals
2024-03-16 18:10:50 03/16/2024 06:10:50 PM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/devices/vitals
2024-03-16 18:10:50 03/16/2024 06:10:50 PM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/devices/vitals
2024-03-16 18:10:50 03/16/2024 06:10:50 PM [pypowerwall] [ERROR] Firmware 234400 detected - Does not support vitals API - disabling.
2024-03-16 18:10:50 03/16/2024 06:10:50 PM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/devices/vitals
2024-03-16 18:10:50 03/16/2024 06:10:50 PM [pypowerwall] [ERROR] Firmware 234400 detected - Does not support vitals API - disabling.
2024-03-16 18:10:50 03/16/2024 06:10:50 PM [pypowerwall] [ERROR] Firmware 234400 detected - Does not support vitals API - disabling.
2024-03-16 18:10:50 03/16/2024 06:10:50 PM [pypowerwall] [ERROR] Firmware 234400 detected - Does not support vitals API - disabling.

@BuongiornoTexas
Copy link
Contributor

@jgleigh Can you check if the broken sockets are still occurring?

docker logs pypowerwall 2>&1 >/dev/null | grep doGET

@jasonacox
Copy link
Owner

Thanks @jgleigh ! As you see, each thread discovers the 404 and detects the new firmware, causing it to disable vitals.

@BuongiornoTexas , I'm still being a bit optimistic by looking for a 404 before disabling, but the effect is the same and should significantly reduce the load on the gateway. It may not 100% solve the doGET errors (since that is client side connection) but it should help.

Please let me know what you both observe. I'll run it for a while and submit it as v4.1.2 later today.

Thanks for your help!!! 🙏

@BuongiornoTexas
Copy link
Contributor

| It may not 100% solve the doGET errors (since that is client side connection)

I was mostly interested in the doGETs because only about half were on vitals. The log above shows doGET errors triggered by timeouts on grid_status, soe, aggregates and site meters. I was/am curious if these are resolved when the vitals calls are bypassed.

jasonacox added a commit that referenced this issue Mar 17, 2024
@jasonacox
Copy link
Owner

Agree.

I just merged v4.1.2 - Use upgrade.sh to update to latest.

@BuongiornoTexas
Copy link
Contributor

I've just run the upgrade - I'm still seeing plenty of socket broken errors, but nothing on the vitals anymore. I'll check if any exceptions throw overnight.

@jgleigh
Copy link
Author

jgleigh commented Mar 17, 2024

My socket errors were pretty random. I also reduced my pool size to 10. Going to let it run for a few days and see how things look with 4.1.2.

FYI, looks like another part of the API got dropped:

2024-03-16 22:52:50 03/16/2024 10:52:50 PM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/troubleshooting/problems

@BuongiornoTexas
Copy link
Contributor

Like @jgleigh, I'm seeing plenty of the api/troubleshooting/problems errors, and regular doGET errors still (had debug off overnight, so will need to check for more detail). Also had a series of auth errors lasting about 10 seconds in the middle of the night. This triggering error occured 6 times in a row, and then things went back to normal.

  File "/usr/local/lib/python3.10/site-packages/pypowerwall/__init__.py", line 212, in _get_session
    self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']}

@jgleigh
Copy link
Author

jgleigh commented Mar 18, 2024

No broken sockets here. Just the missing api/troubleshooting/problems.

2024-03-16 22:52:50 03/16/2024 10:52:50 PM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/devices/vitals
2024-03-16 22:52:50 03/16/2024 10:52:50 PM [pypowerwall] [ERROR] Firmware 234400 detected - Does not support vitals API - disabling.
2024-03-16 22:52:50 03/16/2024 10:52:50 PM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/troubleshooting/problems
2024-03-17 08:03:53 03/17/2024 08:03:53 AM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/troubleshooting/problems
2024-03-17 09:44:30 03/17/2024 09:44:30 AM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/troubleshooting/problems
2024-03-17 10:07:51 03/17/2024 10:07:51 AM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/troubleshooting/problems
2024-03-17 20:13:31 03/17/2024 08:13:31 PM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/troubleshooting/problems
2024-03-18 05:23:11 03/18/2024 05:23:11 AM [pypowerwall] [ERROR] 404 Powerwall API not found at https://192.168.1.6/api/troubleshooting/problems

@BuongiornoTexas
Copy link
Contributor

Hmm - OK, may be a local issue to my site. Will need to dig a bit more.

@jasonacox
Copy link
Owner

Thanks! The /api/troubleshooting/problems 404 is new. I'm glad we have the 404 logic to cache it so at lest it will be less load. The power flow animation was designed to use that so it must have been removed. I'll stub it out.

FYI It would be this in the past, I assume this was a way to communicate banner notices of problems in the portal:

{
"problems": []
}

@jasonacox
Copy link
Owner

Ok, if you want to try it - this quick update stubs out /api/troubleshooting/problems so related 404's should go away.

jasonacox/pypowerwall:0.7.12t44

@jgleigh
Copy link
Author

jgleigh commented May 20, 2024

Just had a thought...I have the Plume Mesh WiFi. It constantly optimizes itself to manage all the mesh nodes. I'm wondering if that's injecting errors into the packets as the Gateway has to change channels/bands. The Gateway has shown it doesn't have the most robust networking logic. I'm planning out a switch of the whole system to UniFi so it'll be interesting to see how the stability looks once that's all up and running.

But no reboots since 5/15, so I think I'm on the right track.

@jasonacox
Copy link
Owner

2024-05-19 06:30:00 05/19/2024 06:30:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe

This is the result of a client (most likely telegraf but could be a browser) timing out and disconnecting from pypowerwall. I say telegraf because it has a 4 second timeout so if the Powerwall does not resupond within 4s, it will disconnect and you will get this error. Technically, we can increase this timeout in the telegraf.conf file to 5s as that is the default for pypowerwall but can also cause some overlap (telegraf samples every 5s). These errors generally aren't an issue as it just means one data point isn't fetched during that 5s sample, but if you see a lot of them, that indicates the Powerwall gateway is having trouble responding within 4s.

Just had a thought...I have the Plume Mesh WiFi. It constantly optimizes itself to manage all the mesh nodes. I'm wondering if that's injecting errors into the packets as the Gateway has to change channels/bands.

Interesting theory. It will be good to see if anything changes when you swap it out.

@jgleigh
Copy link
Author

jgleigh commented May 21, 2024

A lot of these have to be telegraf because they're in the middle of night and no browser dashboard is loaded.

@BuongiornoTexas
Copy link
Contributor

Not sure if it is the same thing you are seeing, but I'm on wifi only, and I get periodic socket broken errors in pypowerwall and a continuous stream of context deadline exceeded errors in my telgraf log.

Some of it may be api calls that are no longer valid in telegraf.conf and local.conf, but I suspect a broader problem as well. Haven't had time to investigate properly yet.

@jgleigh
Copy link
Author

jgleigh commented May 21, 2024

@BuongiornoTexas mine got exponentially worse on Ethernet (probably a bad cable or network card), so I turned WiFi back on. Could also be something local to my network that's making the issue worse for me.

Same telegraf errors for me:
context deadline exceeded (Client.Timeout exceeded while awaiting headers)

@cwagz
Copy link
Contributor

cwagz commented May 21, 2024

I have been meaning to reply here for several weeks now. I started seeing very large data gaps appearing over the last several weeks. Sometime the gaps would be for multiple hours. My HAProxy would show that the TEG was not responding. The issue would resolve itself and I was using the history script to fill in. This Sunday I noticed the batteries had not charged and that I was pulling from the grid during peak due to the batteries not being used. I logged in to the old TEG interface and there was no error. I used the reset button on the gateway and the batteries have been working again since. I have turned my Powerwall dashboard off to see if the issue returns. Maybe my gateway is going bad or maybe this firmware is very sensitive to the polling from HAProxy and pypowerwall causing lock ups? Or maybe the latest pfSense version has something going on with HAProxy causing the TEG to have issues? There are a lot of variables recently. I have been using HAproxy with the TEG for over a year now without issue. Been on Ethernet the whole time as well.

@jasonacox
Copy link
Owner

Hi @cwagz - Yikes! What is your Firmware version and what version of Dashboard are you using?

It is hard for me to believe that local API usage would cause problems with the Gateway and batteries. It is all read-only and It typically will respond with rate limiting errors if you hit it too hard. I have Firmware 24.4.0 and 3 different Dashboards (all current Firmware) polling my gateway without any issue. I'm also using current base install (not HAProxy).

While you wait to see if it returns, you can also switch your Dashboard to cloud mode so at least you have current state if you want access to it. The downsize to cloud mode is that they have removed the Powerwall capacity data points (sadly also gone from FleetAPI) which means the only way to get Powerwall degradation data is via local API (/api/system_status).

@cwagz
Copy link
Contributor

cwagz commented May 22, 2024

Hi @jasonacox - I was on the May 15th version of the dashboard and firmware 24.4.0. I have a bad feeling that something must be going wrong with my gateway. I was going to give it a couple weeks to see if any weirdness happens again. I trust what you are saying regarding the negligible effect the dashboard should have.

@jgleigh
Copy link
Author

jgleigh commented May 26, 2024

I had 10 days with no issues and then 2 reboots in the last 12 hours. The randomness is just crazy!

@cwagz I've also got a thread going on TMC: https://teslamotorsclub.com/tmc/threads/24-4-0-gateway-crashes.326493/

@jgleigh
Copy link
Author

jgleigh commented May 28, 2024

Back to Ethernet I guess...what a crappy network module these things have!!!

Screenshot 2024-05-28 at 10 00 14 AM

@jasonacox
Copy link
Owner

Ugh. Sorry to hear that @jgleigh

My system has weak WiFi even with a mesh node close enough to give it a strong signal. I would see 5m spans where all packets would drop, but nothing like you are seeing. All of that went away when I hardwired in ethernet. Your gateway has me puzzled. I've not been able to throw enough traffic, or bad traffic, at my Gateway to get it to reboot. It will throttle me with 429 HTTP rate limit errors, but it never goes offline.

When I run ethernet cable, I always use a testing tool to ensure I don't short (or long) any of the wires during crimping. I'm not an expert crimper, so it is required. 😂

🤞 that this time goes better for you.

@cwagz
Copy link
Contributor

cwagz commented May 29, 2024

@jgleigh, I have HAProxy still monitoring my TEG and it is still dropping out on Ethernet as well every few days. I have not had to reset it again though. No effect with production or charging / discharging again. I will keep an eye on your TMC post as well. Does seem coincidental with the 24.4.0 firmware. I will get my dashboard back up and running. Tired of feeling blind.

@jasonacox
Copy link
Owner

@cwagz how long does it go offline? Also, @jgleigh does it seems that everything reconnects when it comes back online?

If there was only a way to get Vitals data... 🤷

@cwagz
Copy link
Contributor

cwagz commented May 29, 2024

@jasonacox - I will get a screenshot of it on the dashboard once it happens again. It would just flatline all the data and pick right back up after sometimes hours. The data would strangely load from the cloud though if I ran the history script. Then the one time my powerwalls stopped charging / discharging and I had to use the reset button to get them working again. I have not had this happen again yet and although I am seeing drops in the haproxy stats there is no data missing in the Tesla app. Probably because it is using the cloud data which I agree must buffer inside the gateway somehow.

@BuongiornoTexas
Copy link
Contributor

@jgleigh - A long shot here (definitely mentioned elsewhere but not in this thread yet) - my powerwall disappears from my network if I don't use a static ARP. I've had to set this up on both my dashboard box and my windows box. If I don't do this, my powerwall stops responding to TCP traffic on the local network.

The way I used to get around it before figuring this out was going into the app and forcing it to reconnect to the local network.

@jasonacox
Copy link
Owner

Thanks @cwagz. The local data drop is consistent with my tests where I overload the API (as I mentioned, getting 429 HTTP errors) and it will take 5-10m to "cool down" and let me back in. but, I have never seen the System stop functioning (charge/discharge). Maybe I'm just lucky and there is a bug in 24.4.0 that I just haven't hit yet. It would be extremely noisy, but you could activate DEBUG mode in pypowerwall.env just in case there is some clue as to what is happening. But to be fair, I would think your HAProxy would have just as much info (if not more).

I seriously doubt this applies to anyone else, but I run multiple copies of Powerwall-Dashboad across Linux, RPI, Windows, Mac hosts. I originally started seeing the "data gaps" and 429 errors after the removal of vitals. It was due to the retry logic in pypowerwall trying to fetch vitals. Even most recently, I had a rogue system that was still running the old code and would eventually cause the gateway to rate limit local traffic. As I say, I doubt that scenario would apply for anyone else in the community but putting it out there just in case.

Thanks @BuongiornoTexas for your note as well.

I'm sure they have their reasons, but I'm disappointed in what we are seeing in the firmware updates. The removal of vitals and not providing a local API for the PW3 feel like we are going backwards. If it is related to the system failures you and @jgleigh have raised, that is even more concerning. Maybe the next release will improve things. In the meantime, I'm working on some things... 😁

@jgleigh
Copy link
Author

jgleigh commented May 29, 2024

@BuongiornoTexas Both Ethernet and WiFi are assigned fixed addresses so they don't move around. When the WiFi locked up yesterday, the Ethernet was still up and functional. I did reboot the system last night so both interfaces are up and running.

@jasonacox The total system shutdown is fairly rare. Was much more common when I first got the system installed three years ago. Once switching to Ethernet it was pretty rock solid until around the time they started removing stuff from the API (23.44.x). I'm keeping both interfaces up again to see what happens and polling from the Ethernet one. I've also been messing with my home network a lot so nothing is truly stable at the moment. Still planning on rebuilding my WiFi APs, but I'm not quite there yet (wife is poking me about all the money I'm spending). I should also run a cable test on the Ethernet line since it spans the entire length of my house from the switch to the Gateway, but haven't gotten around to that yet. Got plenty of critters around here that like to chew on things!

Very unfortunate they've locked down the PW3s so much. None of us would have any knowledge about our systems or how to troubleshoot them without getting into the API. Sounds like they may have blocked people from using Tesla One on PW3s as well. Imagine trying to get Tesla tech support on the line and doing something useful instead of just rebooting the whole system. I would have been down for months at a time if I couldn't restart the system locally from the installer interface.

@BuongiornoTexas
Copy link
Contributor

| Both Ethernet and WiFi are assigned fixed addresses

@jgleigh - Just to be clear we are on the same page, I'm talking about ARP - not static IP addresses. It's part of the protocol layer that links IP address to MAC address. This fix shouldn't be required - for almost everything ARP should just work without user intervention (to the point I'd be surprised if you were tweaking your ARP tables at all!) - but in my case I get consistent drop outs after a few hours without it, and no problems at all with it.

(one of the similarities between my set up and your is that both ethernet and wifi are enabled on the powerwall - even though I'm not using ethernet, I suspect the multiple active networks is causing the ARP problem).

@cwagz
Copy link
Contributor

cwagz commented May 29, 2024

| Both Ethernet and WiFi are assigned fixed addresses

@jgleigh - Just to be clear we are on the same page, I'm talking about ARP - not static IP addresses. It's part of the protocol layer that links IP address to MAC address. This fix shouldn't be required - for almost everything ARP should just work without user intervention (to the point I'd be surprised if you were tweaking your ARP tables at all!) - but in my case I get consistent drop outs after a few hours without it, and no problems at all with it.

(one of the similarities between my set up and your is that both ethernet and wifi are enabled on the powerwall - even though I'm not using ethernet, I suspect the multiple active networks is causing the ARP problem).

I am going to give this a shot. Maybe something weird in the latest pfsense version that made this crop up.

@BuongiornoTexas
Copy link
Contributor

| I am going to give this a shot

I've got some notes on how I set it up somewhere - let me know if you would like me to share them. (for both windows and one of the linux implementations).

@jasonacox
Copy link
Owner

let me know if you would like me to share

Please do, @BuongiornoTexas ! I suspect you aren't the only one with that issue.

@jgleigh
Copy link
Author

jgleigh commented May 30, 2024

@BuongiornoTexas I'm assigning fixed IP addresses to the WiFi and Ethernet modules through DHCP reservations.

@BuongiornoTexas
Copy link
Contributor

I'm assigning fixed IP addresses to the WiFi and Ethernet modules through DHCP reservations.

@jgleigh - DHCP reservations are effectively static IP (I'm also using these).

The ARP issue is something else again. I'll post how to do checks on that tomorrow (sadly, with the way the firmware is now, not any guarantee this will help, but worth eliminating as a possible issue at the very least). Which platform are you on?

@jasonacox - I'll post what I've done in discussions and cross link here - it'll be easier to find for future reference and can be linked in the docs if more people than me find it useful (although I'm getting used to being an edge case ;-)).

@jgleigh
Copy link
Author

jgleigh commented May 30, 2024

@BuongiornoTexas Dashboard is running on macOS and using UniFi UDM Pro for my router.

@BuongiornoTexas
Copy link
Contributor

Dashboard is running on macOS

Dashboard machine is the key - I'll see if I can find docs on the macos approach as well then.

@cwagz
Copy link
Contributor

cwagz commented Jun 1, 2024

I already had static arp assigned in pfsense for everything on my network. Both Ethernet and WiFi are connected from the TEG and they both are in the arp table as permanent. I will have the dashboard back up soon but am still unsure what is going on with my TEG networking. I could move the WIFI to an isolated vlan with internet access only and see if that helps. Maybe the teg does not do well having both connected to the same network with the newer firmware.

@BuongiornoTexas
Copy link
Contributor

I already had static arp assigned in pfsense for everything on my network.

@cwagz Well, I hope that eliminates it as a possible cause for the problem for you. But I do note that my ARP issue happens in spite of using a static DHCP allocation with MAC address on the router - the ARP problem happens on the client machines trying to access the powerwall, and this seems to be independent of what the router knows (if my weak understanding is correct, each machine maintains its own ARP table?).

Still digging out the docs and unfortunately distracted by other IRL things. Hopefully get them up in a day or two.

@BuongiornoTexas
Copy link
Contributor

I've put together a discussion thread outlining the static arp thing at #480.

@jgleigh - Unfortunately, I haven't had time to do more than a cursory check on setup for macos - joys of other things going on. I will hopefully update this in a couple of days or less.

@BuongiornoTexas
Copy link
Contributor

@jgleigh After doing a bit more digging, I'm pretty sure the macos solution will be using some variant on the arp command. However, I have no idea how to get it set up permanently.

So before digging into that, it's probably worth doing some trials to see if your problem is arp related. The first thing I'd do is try the following from a machine that is losing access to the powerwall:

arp -a <powerwall ip address>

If your machine can currently reach the powerwall, it should show something like "reachable" or "dynamic" in the status line. If the powerwall is not responding and it is the arp issue, then you will see a "stale" status. If the latter is happening, then we may the have the arp problem and can try to figure a solution from there. (Otherwise, it's probably one of the other manifold issues that seem to afflict the powerwall wifi networking, and all we can do is keep looking ...).

@jgleigh jgleigh closed this as completed Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants