/config/helpers struggles (a lot) more than before, and won't completely load robustly

[Mariusthvdb]

Checklist

• I have updated to the latest available Home Assistant version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in safe mode to rule out problems with unsupported custom resources.

Describe the issue you are experiencing

where I had already filed an issue before on /config/entities not loading most of the time, after updating to 2024.8.0b0 also the helpers page is now struggling to display all correctly. ( helpers count 815)

notice the lack of icons on the right. And this is no longer loading, it has finished and been like this for some minutes now.

At first I figured this could have been cause by the new feature to sort devices & entities based on the date they have been added or modified and showing those columns.

But then I realized that is not used on the Helpers page...

---

Describe the behavior you expected

A complete and fluent loading of the helpers (and entities) pages. Name, icon and all that belongs to this display.
I wouldnt not mind a small gap (have a large number of entities after all), but currently these pages are completely disfunctional to me.

Steps to reproduce the issue

1. described above

...

What version of Home Assistant Core has the issue?

2024.8.0b0

What was the last working version of Home Assistant Core?

No response

In which browser are you experiencing the issue with?

any

Which operating system are you using to run this browser?

sonoma 14.5

State of relevant entities

No response

Problem-relevant frontend configuration

No response

Javascript errors shown in your browser console/inspector

No response

Additional information

Just this observation: the more functionality is added to the/config pages regarding filtering/showing, the more trouble these pages have being responsive and complete robust

[silamon]

I've went to add thousand helpers as well, in a kind of automated way, and it lags when scrolling. It will show the correct information though.

[Mariusthvdb]

Glad you can confirm.

it has become rather unstable though. I sometimes have to reload the view because nothing shows. Sometimes it load but scrolling down shows none of the icons which then get lazy loaded. Very lazy…

This is new in current beta, was quite swift for helpers up to 2024.7, lagged a bit in 2024.7

[silamon]

I'm not going to say that I can confirm. The scroll lag is quite acceptable.
In the newest version, helpers from yaml are displayed as well.
How many helpers did you had in 2024.7, which assumes to show only helpers defined in UI?

[Mariusthvdb]

I didnt add any helpers myself. How can I count those numbers? I can count total amount of templates but not the number of yaml vs UI templates

And yes, only the helpers created in UI showed there. I guess adding all of those yaml helpers doesnt help... again...

its truly terrible, the number in the filter box is probably right, but look what the text in the list says....
:

forced some reloads to reveal that it's mostly the Yaml template entities, 1 yaml group and 1 yaml filter (took that one out too now) showing the pencil-off

and again the number is correct (minus 2) but nothing showing

the deterioration in performance is directly related to changes in. core HA, probably the feature bloat of the filter selection

see #21001 (comment)

its been a great experience in these pages up to 2024.5
2024.6 (beta) introduced new filter options And memory of those options turning Entities dashboard into a nightmare of unstableness

2024.8 now adds that to the Helpers dashboard

[Mariusthvdb]

separate post to easily refer to:

main difference in the Helpers page in 2024.8 is the addition of the Yaml entities. In my case, only template entities (and 1 group...)
Since these are also shown in the Entities dashboard, could it be these are the culprit of the issue?

Given the fact these templates are constantly being evaluated, as that is what templates do, maybe there is something written in local cache/storage all the time, and blowing up because of that??

is this what Karwosts meant when stating this: #21001 (comment) ?

because if it is, this has certainly changed now in the Helpers dashboard as those Yaml entities were not listed here before.

I'll copy this to the other Issue on entities as it seems at least an option.

[silamon]

There's a button under developer tools to debug your connection. You can enable that (and be sure to disable after) to see what the backend is sending. After enabling, open the developer tools, console, and it will print the messages sent and received. Go to the helper page, then to recheck. That would possibly confirm if the lag is coming from a too busy websocket connection.

[silamon]

I can confirm that any state changing, reforces the logic to rerender the table. Does the lag get acceptable if you would filter for 10-20 helpers?

[Mariusthvdb]

what is

then to recheck
?

I have enabled debug and opened Inspector tab Console. opened Helpers page.
I can see the page load, then wait for a few seconds, and then

websocket died doesnt sound good ;-)

it logs a few more of the Sending and received lines, but the template and its icons are not completely loaded now. can only reload it probably

same story of unfinished display, and also see another error in the log:

[silamon]

Your screenshots indicate that your server has restarted.
With recheck I mean, you should reproduce the reproduction steps you have provided.

[Mariusthvdb]

I can confirm that any state changing, reforces the logic to rerender the table. Does the lag get acceptable if you would filter for 10-20 helpers?

it would be nice to be able to deselect per domain ....

Also, we must not forget this is not anything of my doing (other than having more than a few helpers), the Frontend deteriorated because of new features, and I cant test those/unselect those. other than downgrading to 2024.5 (for the entities) and 2024.7 for the helpers.

It would be interesting to see if excluding the template entities would help. I suppose it would as those are the only entities that are being re-evaluated frequently, and cause the Frontend to recalculate contantly?

[Mariusthvdb]

Your screenshots indicate that your server has restarted.

Not sure I can see what you mean. You mean just now during testing? Because I did not restart the server. It was ofc restarted after updating to b3 this morning, but these screenshots just now are from after that

With recheck I mean, you should reproduce the reproduction steps you have provided.

I see, sure, that is what I did, thanks

[Mariusthvdb]

above Inspector was in Safari, this is in Chrome btw:

[silamon]

It doesn't seem to list all received messages, making this a bit waste of a time. I'm out of ideas on how to debug this further, I think we need to improve the tools before going further.

[Mariusthvdb]

I now understand what you meant with

Your screenshots indicate that your server has restarted.

I can indeed see the toast ,message pop up since recently, this happens when I reload the view on Helpers or entities. After that, on helpers I can see the Helpers come in as described above.
On Entities, this remains a very unreliable experience, even hard to describe succinctly..

Yet, the Toast message was not seen before, and seems to be induced in 2024.8

[silamon]

There are all kind of mystery things going on your instance. There is no toast message to be shown when you open the helpers, what does it mention exactly in the toeast message?

Are you sure it isn't restarting when you load the helpers? If that would be case, we would have a track to follow.

[Mariusthvdb]

I meant this:

it happens when I am on the Entities page, see nothing and click the reload. It the pops up, and after that the page starts to load

[silamon]

Would you be able to find out if your server restarts at this time? Check the backend logging as well for any errors. That message indicates that the websocket connection is lost. (Still many reasons possible to why the websocket lost happens. )

[Mariusthvdb]

well, got to admit I had the Helpers a few times now with some delay, but at least they are back (2024.9.0b8 now I think...)

I have Logviewer open currently, and when the connection lost notification happens, nothing is showing in the log.
Which integration should I set to debug?

update

just ignore that slightly optimistic first sentence in this post.
running release now, and the pages are as bad as before.
if I am not wrong, the full Frontend is starting to feel a bit laggy.

[ildar170975]

I wonder - why yaml template sensors were added to Helpers?
Was it properly discussed?
This addition is very questionable.
I also realised that yaml-based history_stats sensors were added to Helpers as well.

[Mariusthvdb]

thx @silamon !!!
I so hope this will fix our troubles.

I still dont understand the huge difference in response comparing the iOS Companion app and the desktop browser though.

would that be something we have to scrutinize further?

[Mariusthvdb]

this should be re-opened as the Helpers page still displays the same trouble
since it's WTH month, Ill add it there too....

[karwosts]

Reopened on user request.