Cannot disable refresh after watching

[benyamin-codez]

Context

• Add-on Version: 7.2.0+beta.4
• Kodi Version: 20.5
• Kodi GUI Language: English
• Operating System: Linux 5.15.92-1-osmc armv71
• Operating System Language: English

---

Expected Behavior

When "Refresh after watching" is disabled, My Subscriptions will not be refreshed when a video finishes.

---

Current Behavior

My Subscriptions is refreshed after every video finishes despite setting of "Refresh after watching".

---

Additional Information

iirc, the setting "Refresh after watching" was originally implemented by @anxdpanic as a stop-gap measure prior to the introduction of the cache mechanism to slow the rate of Data API calls and hence quota limit hits, and to improve performance. At some point the setting appears to have been relegated to the subsection Mark as watched, and perhaps no longer serves it's original purpose.

Due to recent changes re refresh of My Subscriptions, the list is refreshed after every video plays and every page navigation. With 248 channels and 3 bookmarked (MadeForKids) channels this takes over 20 seconds each time on some of my platforms. Previously, this would take 20s on the initial load and then about 9s for each refresh, per this. This then improved to 10 to 11 seconds for the initial load, per this. We are now back to 20+ seconds per refresh and I must say there is no longer a perception that there is any caching performed whatsoever.

This very significant reduction in performance is ruining the user experience and is very frustrating.

It is my understanding the present refresh solution was implemented due to complaints that the My Subscriptions list was not updating at a sufficiently high enough frequency for some users. I would suggest a more widely acceptable solution might be to have a setting dictate the refresh frequency with choices being the standard cache retention periods + an option to refresh after every video is watched.

In similar vein, if refreshes do not occur after every video is watched, the Refresh context menu item would best be relocated back to the top of the menu rather than somewhere near the middle bottom...

---

[MoojMidge]

Got a log of loading My Subscriptions, playing a video, and seeing the listing reload?

[benyamin-codez]

Ok, so it just so happened I got 9s on initial load and for the refresh after first play. 8^d
This happens occasionally but anecdotally I would say it is the exception.
Relevant log file - Formatted: https://paste.kodi.tv/elodukuzay.kodi
Relevant log file - RAW: https://paste.kodi.tv/raw/elodukuzay

So afterwards I did a manual refresh and another play, and got 33s and 40s respectively.
Relevant log file - Formatted: https://paste.kodi.tv/quqefuwixu.kodi
Relevant log file - RAW: https://paste.kodi.tv/raw/quqefuwixu

I also did another run, per the first and got 9s initial load and a 28s refresh after first play.
This is probably the most common scenario.
Relevant log file - Formatted: https://paste.kodi.tv/elomasoxot.kodi
Relevant log file - RAW: https://paste.kodi.tv/raw/elomasoxot

Having said all of that, I would still prefer NOT to refresh after every play (but preferably when the cache timer expires)...! 8^D

[MoojMidge]

iirc, the setting "Refresh after watching" was originally implemented by @anxdpanic as a stop-gap measure prior to the introduction of the cache mechanism to slow the rate of Data API calls and hence quota limit hits, and to improve performance. At some point the setting appears to have been relegated to the subsection Mark as watched, and perhaps no longer serves it's original purpose.

This option was always just a refresh of the Kodi container content, and I believe it was originally intended to update a video listing so that the Kodi watched status was properly displayed. It serves no purpose in recent versions of Kodi and this addon, since:

1. Kodi updates the container content anyway after playback stops. This can't be disabled.
2. The inconsistencies in playback status being displayed in the UI have been fixed.

In similar vein, if refreshes do not occur after every video is watched, the Refresh context menu item would best be relocated back to the top of the menu rather than somewhere near the middle bottom...

People complained about the play related context menu items being too far down in the list... there have also been recent changes in the Kodi context menu that may make it necessary to have these items at the top of the list. The compromise was to have refresh close to the bottom of the list so it can be selected quickly by scrolling up (rather that down) in the context menu.

We are now back to 20+ seconds per refresh and I must say there is no longer a perception that there is any caching performed whatsoever.

Your logs indicate that things are working largely as expected. The reason that the caching doesn't appear to work is because new content is available that is not in your cache.

That being said, it is not necessary to check all your subscriptions all the time. The current beta changed to fetching subscriptions sorted by last active, and this can be extended further.

Try the following:

1. Increase the cache size - Settings > Advanced > History > Cache size (MB). Aside from storage limitations there is no real downside to increasing the size.
2. Try https://github.com/MoojMidge/plugin.video.youtube/archive/refs/heads/master.zip

[benyamin-codez]

Thanks @MoojMidge

That's all very helpful.

The reason that the caching doesn't appear to work is because new content is available that is not in your cache.

I guess the retort to this is if I'm likely to have new content in my feed, then I'm unlikely to hit the cache.
Moreover, if I have to check my data source to see if there's anything new, it sort of defeats the purpose of having the cache.
I'm happy to accept my cache is stale for a while if it improves my experience (which it does).

I can't check it right now, but I have probably forgotten to increase my cache size since last setup wizard run.
Ok, maybe a few runs back... 8^d

Also the changes in 2faf94b look like they will certainly help. I'll try to run #1073 asap and let you know.

[benyamin-codez]

@MoojMidge

I'll try to run #1073 asap and let you know.

That made a demonstrable difference.
Refresh still runs far too often for my liking, and the time to run is somewhat sporadic, but it is better.

Formatted log: https://paste.kodi.tv/hegidoqeto.kodi
RAW log: https://paste.kodi.tv/raw/hegidoqeto

[MoojMidge]

Refresh still runs far too often for my liking, and the time to run is somewhat sporadic, but it is better.

The refresh happens all the time. It is done by Kodi itself and cannot be disabled.

However you are making things worse for yourself because you are manually triggering a refresh (which bypasses the addon cache) and then when Kodi does its own refresh it will then also bypass the cache.

Normally when a plugin url does not change then the plugin invocation will just silently use the internal Kodi disc cache. Except when Kodi does its own forced refresh.

To manually trigger a refresh the addon therefore needs to change the plugin url and does this by adding a query parameter to the plugin url, which is the parsed and used to identify that the addon cache should be bypassed.

However now everytime you play a video from the same listing, Kodi will force refresh the listing (can't be disabled), and because you triggered a manual refresh previously which added the refresh query parameter to the plugin url, every single one of the forced refreshes will also bypass the addon cache.

Try #1073 again, it has been updated to include some improvements to handling the various locks used in the addon, which the profiler indicates is causing slowdowns on your device, and also workaround the Kodi forced refresh bypassing the addon cache if a manual refresh was triggered beforehand.

[benyamin-codez]

Try #1073 again...

Take 3... 8^d 8^D

[MoojMidge]

And again. The changes to disable the refresh parameter has broader consequences that needed to be properly handled.

Don't really see any changes on a faster system, but it should result in some noticeable improvements for slower systems I think.

Already saw duration drop from 20s to 15s and 9s to 6s in your logs, depending on whether the feeds needed to be fetched and parsed, let's see if the lock contention changes offer any further improvements.

[benyamin-codez]

Thanks @MoojMidge

All good. Giving that a try now...

[MoojMidge]

Updated again. Profiler seems to indicate that disk I/O used for the data cache may be a bottleneck.

[benyamin-codez]

Updated again.

Thanks. I'll give it a try shortly.
The previous cut was pretty good too.
Came to get a log and got distracted. 8^d
So I'll grab one using your latest instead...

Profiler seems to indicate that disk I/O used for the data cache may be a bottleneck.

Yep, SD card.

[benyamin-codez]

@MoojMidge

That's almost bearable... 8^D

Formatted log: https://paste.kodi.tv/olatikacib.kodi
RAW log: https://paste.kodi.tv/raw/olatikacib

[MoojMidge]

Lol you are tough to please.

Not sure there is more to be optimised, but that will do for now.

[benyamin-codez]

@MoojMidge

Many thanks for the optimisations...!

I'm adjusting... 8^d
It's quite the change from managing refreshes manually if less than the cache TTL to have it refresh after every play.

[MoojMidge]

It's quite the change from managing refreshes manually if less than the cache TTL to have it refresh after every play.

Don't quite get this. Kodi has been doing this since version 16 or 17 I think. Are you saying that something has actually changed recently?

Yep, SD card.

Try changing this from WAL to PERSIST

plugin.video.youtube/resources/lib/youtube_plugin/kodion/sql_store/storage.py

Line 263 in 9ad15bf

'PRAGMA journal_mode = WAL;',

[benyamin-codez]

@MoojMidge

Thanks for further works..!

It's quite the change from managing refreshes manually...

Don't quite get this. Kodi has been doing this since version 16 or 17 I think.

Yeah, when you said this earlier, it had me thinking that perhaps I just wasn't perceiving it before...

Are you saying that something has actually changed recently?

Yes, it has definitely changed.
IIRC, it was shortly after you introduced the progress whirligig, but it might have been contemporaneous with that change.
Previously, there would only be an observable refresh after playback if the TTL had expired. This seemed to be 30 minutes.
Perhaps this was previously more obvious because the (blocking) Kodi spinner always displayed whenever the threadpool refreshed the feed.

Yep, SD card.

Try changing this from WAL to PERSIST

I see the recent commit (MoojMidge@2906797), so I'll give your master another go...

[MoojMidge]

Yes, it has definitely changed.

Can you get a debug log when using https://github.com/anxdpanic/plugin.video.youtube/releases/tag/v7.1.0.1?

Recent version are consistently faster than this older version for me.

[benyamin-codez]

@MoojMidge

Yes, it has definitely changed.

Can you get a debug log when using v7.1.0.1?

Formatted log: https://paste.kodi.tv/cediqoxupe.kodi
RAW log: https://paste.kodi.tv/raw/cediqoxupe

Notable differences in experience with v7.1.0.1:

1. After stopping a video, NO refresh occurs
2. The issue in Audio dropping on initial play #1087 is NOT present
3. The refresh after playing occurs under the Kodi spinner before the list is shown
4. The lack of progress whirligig gives an impression of readiness

Differences [3] and [4] are illusory, for which I wouldn't suggest any change. Difference [1] is really the one that I struggle with, especially after mistakenly playing the wrong video, or when needing to replay a video due to issues like that at [2]. The refresh after stopping in more recent versions adds to the frustration of needing to replay the video. I think I probably stop many videos early too. Previously, I might have skipped to the end to pre-empt an expected refresh.

Regarding speed, it wasn't clear to me which version was faster, as each version has a somewhat different experience.
Your latest cut is indeed very quick, but it feels a bit like comparing apples and oranges.

[MoojMidge]

Thanks for the logs.

1. After stopping a video, NO refresh occurs

That is not what your log shows. Everytime the video stops, the previous video listing is reloaded. This has been standard Kodi behavior for a very long time.

It takes around 3 seconds to refresh the listing according to your logs. However one key difference between v7.1.0.1 is that the listing wasn't filled if you were filtering, so in v7.1.0.1 you are only displaying around 25 items but in recent versions you are displaying the full 50.

The last log you provided showed that this takes around 5 seconds in newer versions, and I would expect the latest commits to make that slightly faster again.

2. The issue in Audio dropping on initial play #1087 is NOT present

As mentioned in your other issue there is nothing in your log that would indicate why you are not hearing the audio. The logs are pretty much identical in that regards.

The recent version will add some latency to the streaming, but can't see why that would stop the audio (and not the video) completely.

One thing you could check is the Player process info and confirm that what it says about the audio for a working and non-working play through.

3. The refresh after playing occurs under the Kodi spinner before the list is shown

Now this might explain what you are perceiving to be a slowdown.

The newer versions have a progress dialog, and I suspect what you are seeing is Kodi queueing these GUI dialogs up so that the busy spinner that occurs after video playback completes prior to the next dialog is shown, which delays the refresh of the listing.

To confirm you can comment out these two lines:

plugin.video.youtube/resources/lib/youtube_plugin/youtube/helper/yt_specials.py

Line 345 in 9ad15bf

progress_dialog=progress_dialog,

plugin.video.youtube/resources/lib/youtube_plugin/youtube/helper/yt_specials.py

Line 360 in 9ad15bf

progress_dialog=progress_dialog,

[benyamin-codez]

@MoojMidge

Thanks for having a look.

The newer versions have a progress dialog, and I suspect what you are seeing is Kodi queueing these GUI dialogs up so that the busy spinner that occurs after video playback completes prior to the next dialog is shown, which delays the refresh of the listing.

To confirm you can comment out these two lines...

This resolved the issue for me. Neither the Kodi spinner nor progress dialog now show after stopping a video.

I'm guessing users might be split as to preference on this one. Perhaps a new setting can help accommodate both?

The refresh is now certainly quick. Thank you for the tuning.

[MoojMidge]

Perhaps a new setting can help accommodate both?

Not going to happen unfortunately.

If the latest commits (https://github.com/MoojMidge/plugin.video.youtube/archive/refs/heads/master.zip) don't resolve the problem, you will either need to live with it or comment out those lines yourself.

[benyamin-codez]

Perhaps a new setting can help accommodate both?

Not going to happen unfortunately.

If the latest commits don't resolve the problem, you will either need to live with it or comment out those lines yourself.

Good thing your latest cut appears to offer a reasonable compromise...! 8^D

In practice, in seems to not refresh (returning immediately to the list) if the watched threshold hasn't yet been reached.

It's looking pretty polished. Thanks again..!

[benyamin-codez]

In practice, [it] seems to not refresh (returning immediately to the list) if the watched threshold hasn't yet been reached.

That's not really what's happening, of course. It's a bit of a mixed bag.

It seems a little less stable... Trying the latest master...

[benyamin-codez]

@MoojMidge

It seems a little less stable... Trying the latest master...

Very rarely getting a change of viewtype (from WideList) when stopping a video early, where the watched status icon becomes a thumbnail. This hasn't shown up in some time, being a previously fixed bug.

Sometimes the watched status is volatile (disappears after playing other videos) or does not update. This happens quite a lot but every time I run debug logging it doesn't happen... Maybe a race somewhere?

Anyway, iinm these seem to have been (re)introduced with commits addressing this issue....

EDIT: Tagged Mooj in case this was overlooked...