Data @ 11pm-12am is lost every night

[smithbill17]

Each night at the end of the day, the accumulating value of kWh consumed is reset to zero and begins increasing again for the next day from after midnight. This value (for electricity) is sensor.electric_consumption_today

However, if you use a HistoryGraph (see attached image) to graph sensor.electric_consumption_today it's apparent that there is no data from 11pm until 12.30am every night (the consumption data is zero for at least 1.5hrs). This means that there is at least 1.5hrs of consumption not being accounted for every night.

Expected behaviour would be for consumption data to continue until 12.00am when it would reset to 0kWh and begin accumulating again from 12.01am.

HomeAssistant v2021.10.6
Hildebrand Glow (DCC) integration v0.3.3

[ColinRobbins]

I think this may be related to a time zone calculation error I have spotted.
I am about to make a PR this might fix it.

[ColinRobbins]

I think there is possibly a second factor here as well.

The data from DCC is delayed up to 30 minutes.
The sensor attempts to read the data every 5 minute or so.
So, let’s say at 11.35 DCC updates, then again at 12.05.
The sensor will get the 11.35 change at say, 11.40.
That will be the last reading entered for the day. So 25 minutes of data lost.

Or a second case.
DCC updates at 11.27 and 11.57.
The sensor reads at 11.56 and 00.01.
35 minutes data lost.

Not an easy fix, as HA works on “now” reading, so you can’t push history into HA to close out “yesterday”.
Possibly something along the lines of, after midnight, the sensor could take “yesterday” readings for the first 30 minutes, until we are sure the full yesterdays data is in.
The on the next run, take the first “today” reading. No data will then be “lost”, with the downside it introduces a slight lag in getting reading for the new day (e.g., worst case, no reading until 00.30.

Let’s see what @HandyHat thinks, but maybe worth a question on HomeAssistant wider forum to see how other integrations deal with it.

[ColinRobbins]

Another possibility could be to change the sensor behaviour (or a new sensor) to not reset to zero at midnight, by reading the cumulative data instead. Feature Request #25

[smithbill17]

I think I understood what you were saying about the sensor using post-midnight data until a full set of 24hrs was accumulated, but I'm no technical expert, so no real idea how that could work.

But I am wondering where the data disappears to? I mean, presumably the DCC has a full 24hrs of data, and sends it every 30mins on request? So if data is 'lost' from the sensor, then it must be being over-written with the next 30mins of data??

Or is HA request data from the DCC by timestamp - so if it doesn't request data for 11.30pm-12.30am, then the DCC never sends it?

Isn't DCC data timestamped? Can't the data values be stored in the sensor based on DCC timestamp?

And if the issue is the "upto 30min" data delay from the DCC, why does it always seem to be zero rated for 1hr from 12am every night without any variation to it (at least I think that's the time period it shows 0kWh).

Also, why bother resetting the sensor to zero at midnight anyway? After all, my SMETS2 SmartMeter doesn't reset it's meter value at midnight, so why doesn't the sensor value just reflect the kWh reading from the SmartMeter?

Sorry for all the dumb questions which doesn't help fix the 0kWh value around midnight.

[ColinRobbins]

The zero could be related to this.
After midnight, the sensor asks for the data for “today”. The data from DCC could be delayed 30 minutes. So the reading from DCC would be zero until 00.29. In the sensor reads every 5 minutes, it could 00.34 before the sensor sees the data.

The choice to reset the sensor at midnight is an implementation choice to give the “consumption today”.
So the more I think about it, the solution is to provide a cumulative sensor as well (e.g., usage this year). This would then report the correct data to the Energy monitor (apart from a few minutes at year end).

I think it would be easy to put this in place, as an add on to the PR I’ve just made.

[smithbill17]

Well, I'm no expert, but I do think resetting to zero at midnight everyday isn't the best idea. I mean, 24hrs is just an arbitrary thing to help us humans figure out how much energy we're using from day to day.

I think it would be better to just keep accumulating indefinitely (same as my SMETS2 meter does). Why do you have to reset it annually? Can't the value just 'match' the SMETS2 meter reading which never gets reset (well, maybe it does when it reaches 999999kWh).

[ColinRobbins]

A year’s consumption is the biggest chunk of data the Glow API will return.
See page 9 of the documentation.

[smithbill17]

Looking at p9 of the documentation, can't the electricity be returned at minute level (losing a minute or two at midnight is going to be irrelevant), and gas at the 30min level, the majority of people probably don't burn at lot of gas around midnight, so losing an hour of gas consumption at midnight is probably not going to have much impact on the daily kWh or £ cost of gas consumed.

Whereas most people are probably consuming a background level of electricity throughout the day & night. So perhaps two different sensor data retrieval periods would be appropriate?

Sorry, my lack of understanding how it all works is probably resulting in a lot of dumb questions

[ColinRobbins]

Sadly you can’t get the per-minute data for free via DCC as it will only update the data from your meter once every 30 minutes.
If you want minute by minute, you need the buy the Hildebrand Glow hardware, which connects to your meter and cuts DCC out the loop (and then use this integration to read the data from the hardware.)

[smithbill17]

Okay. Well I suppose the real issue is: because sensor.electric_consumption_today is used to derive cost of electricity used each day, then it must mean the £ calculated is too low every day.

In the attached graph of sensor.electric_consumption_today you can see every 24hrs at midnight, there is 1.5hrs of zero consumption and therefore 1.5hrs worth of £ missing from the daily energy cost. Ideally the graph should start climbing immediately after midnight and therefore reach a higher kWh by midnight the next day, and so the derived £ for the day should be higher.

Obviously, like most people, I'm never not burning electricity every 24hrs. It's constant consumption. So there should never be any flatlined zero consumption showing in the graph (unless I was to switch off my entire electricity supply).

At least, I think that's what the problem is :)

If the inherent 30min/DCC thing means it's simply not possible to ensure the data around midnight isn't lost, then perhaps the best thing is to say it's for 'guidance only' and just close this issue?

[ColinRobbins]

Let’s summarise…
You are seeing a gap of 1.5 hours. I believe this is caused by two issues.

1. a bug in the calculation of daylight saving time. A fix is proposed in a pull request. This may account for 1 hour of the gap.
2. the 30 minutes, which is due the DCC 30 minute delay, and wrapping at the end of a day. I suggest this can be fixed by cumulative annual sensor which can be used by Energy for an accurate calculation.

I’ll work on point 2. When this is added the documentation can be updated.

[smithbill17]

In the Hildebrand Bright app (which I think must get it's data from the DCC too?) it doesn't appear to be missing any data around midnight - see attached image.

However, I think this is something that has changed between their currently 'released' version of the app and the 'beta' version available in the Google PlayStore, because I think I started to notice this same issue when I looked in the app, and then I joined the 'beta' programme and looking this morning it appears to be fine.

[ColinRobbins]

We are agreeing. The data is all there in the Glow API / DCC. It just how you get at it and manage the 30 minute delay. The Home Assistant model assumes real time data, not historic (30 minute old) data. I have a proposed solution in my summary post above, let’s wait and see if that works.

[ColinRobbins]

I now have the cumulative (Usage since Jan 1 in current year) sensors implemented in my test system.
Need to let it run for a few days to make sure the Energy integration works OK with it.
If all goes well, I'll make a PR.

[HandyHat]

Yeah unfortunately it is impossible to backdate data in Home Assistant from what I can tell. Using cumulative on the energy dashboard means that some days will be overreported, as the usage during the last 30 mins that is missing will get attributed to the next day :(

[HandyHat]

I've clarified that the last 30 minutes of the day will be lost on the daily sensors, the bug in the calculation of DST has been fixed and cumulative sensors have been added. Please reopen if you are still losing more than 30 minutes of data!