Why extra queries to fusionsolar for date in future

[rvk01]

In query() in huawei-solar/api/fusionsolar.py you call self.query_single_unit_2_days.
But if the date is for today/now (because you are up to date with pvoutput) the self.query_single_unit() is still called for timestamp+24*3600 in self.query_single_unit_2_days(), which is tomorrow (a date in the future).

Is there a special reason for this?

This is done for every signal so about 7 extra calls which return empty days.
Wouldn't it be better to check in query_single_unit() (or somewhere else) that querying a date in the future isn't done?

Something like:

def query_single_unit(self, device, timestamp, signals):
      if timestamp > datetime.now().timestamp(): return {}  # no date in the future
      data = self.call_api(

[szczeles]

That's very good point, thanks! IIRC, the 2-days query was to fix some backfill issue, so querying the future is not needed in in-sync operations at all. Let me test your fix over a 2 days to make sure it works as expected :-)

[szczeles]

Hm, it stopped to work after midnight and now I think I recall that date param for rest/pvms/web/device/v1/device-history-data API call uses some strange timezone, that's why I needed to call 2 days there. And all old data are dropped based on last PV entry, so having too much information doesn't harm the system here.

It would be for sure beneficial to get rid of the extra call, but not knowing the logic in the backend it could just introduce delays in the data delivery... What do you think, @rvk01 ?

[rvk01]

It would be for sure beneficial to get rid of the extra call, but not knowing the logic in the backend it could just introduce delays in the data delivery... What do you think, @rvk01 ?

I've noticed that this morning too :) I will investigate this and get back.

[rvk01]

It would be for sure beneficial to get rid of the extra call, but not knowing the logic in the backend it could just introduce delays in the data delivery... What do you think, @rvk01 ?

Ok, I investigated this. (assuming current date/time 18-02-2023 13:00)
The problem is that when pvoutput reports the last data is from 17-02-2023 20:00 the current date can be 18-02-2023 13:00. Then the first day is retrieved as complete day of 17-02-2023 (whole day). But it is done with epoch of 17-02-2023 20:00. Not epoch 17-02-2023 00:00. So when the next day is about to be retrieved, the timestamp is 18-02-2023 20:00. And that's in the future. But you still want that day because it's already 13:00 (and you want the morning readings too).

What we really want is all data until today/tonight at midnight. So until 19-02-2023 00:00.
This could be done in 2 ways. With the query_single_unit_2_days you could trim to the asking date and always pass 00:00.
The other way is to limit the call in query_single_unit not to the current time, but to the next day at midnight 00:00.

Something like:

def query_single_unit(self, device, timestamp, signals):
    # get current data and add one to get next day at midnight 00:00
    nd = datetime.now().replace(microsecond=0, second=0, minute=0, hour=0).timestamp() + 24 * 3600
    if timestamp > nd:
        return {} # timestamp is beyond the current date

The only downside of this is that because there is no reports to pvoutput after sundown, it will still try to retrieve the previous complete day from 00:00 to sunrise AND the current day, resulting in those extra calls in that timeslot. But it's already much better than calling for the next day in the timeslot sunrise to 23:59, which it did (because pvoutput was up to date).

(This code is running now so I will keep an eye on it :) )

[szczeles]

Fantastic investigation! If the code works as expected, feel free to push a PR, I'm happy to merge :-)

[rvk01]

For now it seems to be working correctly. From the logs (below) it will pull 7 requests (from one day) until 00:00.
After that, until sunrise at 2023-02-20 05:55:16, it pulls two times 7 requests (for yesterday and today).
Then from 05:55:16 until 00:00 again, it pulls 7 requests (for that same day) again.

This weekend there was maintenance on fusionsolar site and my version crashed out with first some Backing off call_api(...) Connection refused. Then after 10 minutes, it got a "Login successful" after which it crashed out with "TypeError: getattr(): attribute name must be string". Not a big deal but I thought I mention it.

For pushing a PR, I'm still not very familiar with GIT. I've always used github in combination with SVN myself.
I'll need to learn :)

But I also found that, for me, the 7-14 requests are not all needed. I only use it in combination with pvoutput, and for that only day_cap and active_power are needed (30016 and 30014). Now they are pulled in with different requests, per unit (including lots I don't need), but I can request 30016 and 30014 together in one call (even though the units differ, kwH and Wh), cutting down the request from 7 to 14 to just 1 or 2.

Furthermore, I was also thinking about querying the device-list before every loop. In it, there is inverter shutdown and startup. If shutdown is > startup, you know the inverter hasn't had any data after that. So there would be no need to query for the data (cutting down the requests to just one). I first thought about pulling device-real-kpi to get the startup and shutdown, but that only works when the inverter is online. After shutdown, it only reports that the device isn't connected. But the device-list keeps both times.

Another thought would be to just use sunrise and sunset times and sleeping (not querying) during sunset+1hour to sunrise-1hour. Although in that case it might be needed to still do a call to idle on fusionsolar. But then the calls can be cut down to absolute minimum.

Those where just some of my ideas... :)

2023-02-19 23:51:51,610 No data to update
2023-02-19 23:56:51,710 Calling api for day 1676839200.0
2023-02-19 23:56:51,986 Calling api for day 1676839200.0
2023-02-19 23:56:52,180 Calling api for day 1676839200.0
2023-02-19 23:56:52,350 Calling api for day 1676839200.0
2023-02-19 23:56:52,549 Calling api for day 1676839200.0
2023-02-19 23:56:52,739 Calling api for day 1676839200.0
2023-02-19 23:56:52,955 Calling api for day 1676839200.0
2023-02-19 23:56:53,167 No data to update
2023-02-20 00:01:53,199 Calling api for day 1676839200.0
2023-02-20 00:01:53,458 Calling api for day 1676925600.0
2023-02-20 00:01:53,658 Calling api for day 1676839200.0
2023-02-20 00:01:53,840 Calling api for day 1676925600.0
2023-02-20 00:01:54,027 Calling api for day 1676839200.0
2023-02-20 00:01:54,196 Calling api for day 1676925600.0
2023-02-20 00:01:54,363 Calling api for day 1676839200.0
2023-02-20 00:01:54,555 Calling api for day 1676925600.0
2023-02-20 00:01:54,727 Calling api for day 1676839200.0
2023-02-20 00:01:54,910 Calling api for day 1676925600.0
2023-02-20 00:01:55,096 Calling api for day 1676839200.0
2023-02-20 00:01:55,268 Calling api for day 1676925600.0
2023-02-20 00:01:55,463 Calling api for day 1676839200.0
2023-02-20 00:01:55,671 Calling api for day 1676925600.0
2023-02-20 00:01:55,859 No data to update

2023-02-20 05:45:12,621 No data to update
2023-02-20 05:50:12,712 Calling api for day 1676839200.0
2023-02-20 05:50:12,922 Calling api for day 1676925600.0
2023-02-20 05:50:13,159 Calling api for day 1676839200.0
2023-02-20 05:50:13,391 Calling api for day 1676925600.0
2023-02-20 05:50:13,628 Calling api for day 1676839200.0
2023-02-20 05:50:13,803 Calling api for day 1676925600.0
2023-02-20 05:50:14,007 Calling api for day 1676839200.0
2023-02-20 05:50:14,183 Calling api for day 1676925600.0
2023-02-20 05:50:14,351 Calling api for day 1676839200.0
2023-02-20 05:50:14,524 Calling api for day 1676925600.0
2023-02-20 05:50:14,707 Calling api for day 1676839200.0
2023-02-20 05:50:15,035 Calling api for day 1676925600.0
2023-02-20 05:50:15,235 Calling api for day 1676839200.0
2023-02-20 05:50:15,449 Calling api for day 1676925600.0
2023-02-20 05:50:15,619 Pushing 1 records to PVOutput
2023-02-20 05:55:16,247 Calling api for day 1676868300.0
2023-02-20 05:55:16,471 Calling api for day 1676868300.0
2023-02-20 05:55:16,692 Calling api for day 1676868300.0
2023-02-20 05:55:16,902 Calling api for day 1676868300.0
2023-02-20 05:55:17,082 Calling api for day 1676868300.0
2023-02-20 05:55:17,286 Calling api for day 1676868300.0
2023-02-20 05:55:17,467 Calling api for day 1676868300.0
2023-02-20 05:55:17,670 Pushing 1 records to PVOutput
2023-02-20 06:00:18,358 Calling api for day 1676868600.0
2023-02-20 06:00:18,561 Calling api for day 1676868600.0
2023-02-20 06:00:18,748 Calling api for day 1676868600.0
2023-02-20 06:00:18,919 Calling api for day 1676868600.0
2023-02-20 06:00:19,126 Calling api for day 1676868600.0
2023-02-20 06:00:19,305 Calling api for day 1676868600.0
2023-02-20 06:00:19,495 Calling api for day 1676868600.0