Negotiate cache_expiration_time #26

Alkarex · 2024-09-15T13:13:21Z

Negotiate the cache expiration time based on the HTTP response headers.
Return the cache duration time in number of seconds since the Unix Epoch, accounting for:

Cache-Control: max-age minus Age, bounded by $cache_duration_min and $cache_duration_max
Cache-Control: must-revalidate will set $cache_duration to $cache_duration_min
Cache-Control: no-cache will return time() + $cache_duration_min
Cache-Control: no-store will return time() + $cache_duration_min - 3 (to distinguish from no-cache if needed)
Expires like Cache-Control: max-age but only if it is absent

Parameters:

$cache_duration Desired cache duration in seconds, potentially overridden by HTTP response headers
$cache_duration_min Minimal cache duration (in seconds), overriding HTTP response headers Cache-Control and Expires
$cache_duration_max Maximal cache duration (in seconds), overriding HTTP response headers Cache-Control and Expires

- `Cache-Control: max-age` minus `Age`, extendable by `$simplepie_cache_duration` - `Cache-Control: must-revalidate` will prevent `$simplepie_cache_duration` from extending past the `max-age` - `Cache-Control: no-cache` will return the current time - `Cache-Control: no-store` will return `0` - `Expires` but only if `Cache-Control: max-age` is absent

Alkarex · 2024-09-15T19:53:51Z

@Frenzie , @marienfressinaud, @Art4 and others: discussion welcome.

At the moment SimplePie is not HTTP compliant when it comes to HTTP requests and caching. This PR addresses some of it.

But I have some doubts. On my local tests, I see plenty of feeds responding with Cache-Control: no-cache, no-store, max-age=0, must-revalidate or a variation of that. no-store being the strictest.

For instance, our own GitHub releases feed returns cache-control: max-age=0, private, must-revalidate (it does support conditional requests though, luckily).

So on FreshRSS instances with many users, there is a good likelihood of having several users subscribing to the same feed(s) (for instance this GitHub release feed), and preventing the cache from being used by the following users during a refresh cron job would be a waste of resources - but potentially good for servers which care about download statistics.

We could consider updating our heuristics based on e.g. Cache-Control: private. There are fewer feeds with this directive but still a number in my own test set, such as https://www.journalduhacker.net/rss (returns Cache-Control: max-age=0, private, must-revalidate) or https://www.blogger.com/feeds/8698702854482141883/posts/default (returns cache-control: private, max-age=0, must-revalidate, no-transform)

Thoughts?

Frenzie · 2024-09-15T20:41:26Z

At least on GH I would suspect it's closer to a mistake, intended to make sure the comments are up to date on a page like this one.

But you can certainly think of scenarios where it would be intentional. Perhaps something like a server minimum cache time constant at 5 or 10 minutes by default?

Alkarex · 2024-09-15T20:45:00Z

Perhaps something like a server minimum cache time constant at 5 or 10 minutes by default

I have introduced $cache_duration_min: Minimal cache duration (in seconds), overriding HTTP response headers Cache-Control: max-age and Expires, ~~but without effect on Cache-Control: no-store and Cache-Control: no-cache and Cache-Control: must-revalidate~~

Its default is 60s but can be set higher. But not obeying Cache-Control: no-store or Cache-Control: no-cache or Cache-Control: must-revalidate would not be HTTP compliant.

Discussion in FreshRSS/simplepie#26

Alkarex · 2024-09-16T07:07:28Z

Changed logic, to account for the many servers disabling cache all-together. Simpler, and probably more pragmatic

Frenzie · 2024-09-16T07:39:35Z

You don't think 24h sounds a bit short as a default maximum if the site is explicitly asking for it? :-)

Alkarex · 2024-09-16T08:18:52Z

You don't think 24h sounds a bit short as a default maximum if the site is explicitly asking for it? :-)

Maybe, but I am balancing that with users, who will believe that the refresh is broken...

P.S.: And I can also see some feeds returning high values, which are most likely broken.
For instance 30d for https://gts-net.dk/feed/ during the conditional response (I need to debug that one a bit more).

Frenzie · 2024-09-16T08:43:14Z

Right, I'm not saying to just listen to whatever the feed might say, just that a few days still seems reasonable. But the point about users makes sense.

Alkarex · 2024-09-18T11:12:23Z

For some unknown reason, I observe some strange feeds (e.g. https://gts-net.dk/feed/ ), which are returning a much higher max-age (30d) for a 304 response than for the initial 200 response (1h).

$ curl -v -A 'FreshRSS/1.25.0-dev (Linux; https://freshrss.org)' -H "Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8" 'https://gts-net.dk/feed/'
> GET /feed/ HTTP/2
> Host: gts-net.dk
> User-Agent: FreshRSS/1.25.0-dev (Linux; https://freshrss.org)
> Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8
>
< HTTP/2 200
< date: Wed, 18 Sep 2024 10:38:12 GMT
< content-type: application/rss+xml; charset=UTF-8
< server: Apache
< link: <https://gts-net.dk/wp-json/>; rel="https://api.w.org/", <https://gts-net.dk/>; rel="canonical"
< x-tec-api-version: v1
< x-tec-api-root: https://gts-net.dk/wp-json/tribe/events/v1/
< x-tec-api-origin: https://gts-net.dk
< last-modified: Wed, 18 Sep 2024 10:24:19 GMT
< cache-control: max-age=3600
< expires: Wed, 18 Sep 2024 11:38:12 GMT
< vary: Accept-Encoding
< x-content-type-options: nosniff
< simplycom-server: Apache
< simplycom-server: nginx
< alt-svc: h3=":443"; ma=86400, h3-29=":443"
<
<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
        xmlns:content="http://purl.org/rss/1.0/modules/content/"
        xmlns:wfw="http://wellformedweb.org/CommentAPI/"
        xmlns:dc="http://purl.org/dc/elements/1.1/"
        xmlns:atom="http://www.w3.org/2005/Atom"
        xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
        xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
        >
<!-- ... -->
</rss>

$ curl -v -A 'FreshRSS/1.25.0-dev (Linux; https://freshrss.org)' -H "Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8" -H 'If-Modified-Since: Wed, 18 Sep 2024 10:24:19 GMT' 'https://gts-net.dk/feed/'
> GET /feed/ HTTP/2
> Host: gts-net.dk
> User-Agent: FreshRSS/1.25.0-dev (Linux; https://freshrss.org)
> Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8
> If-Modified-Since: Wed, 18 Sep 2024 10:24:19 GMT
>
< HTTP/2 304
< date: Wed, 18 Sep 2024 10:41:23 GMT
< server: Apache
< last-modified: Wed, 18 Sep 2024 10:24:19 GMT
< cache-control: max-age=2592000
< expires: Fri, 18 Oct 2024 10:41:23 GMT
< x-content-type-options: nosniff
< simplycom-server: Apache
< vary: Accept-Encoding
< simplycom-server: nginx
<

Frenzie · 2024-09-18T12:51:55Z

Oof. :-/

Seems to only make sense for CDNs

Alkarex · 2024-09-20T13:51:26Z

Another feed with same problem: https://www.smappee.com/blog/feed/

< HTTP/2 200
< server: nginx
< date: Fri, 20 Sep 2024 13:46:47 GMT
< content-type: application/rss+xml; charset=UTF-8
< last-modified: Thu, 19 Sep 2024 14:26:27 GMT
< cache-control: max-age=3600
< expires: Fri, 20 Sep 2024 14:46:45 GMT

< HTTP/2 304
< server: nginx
< date: Fri, 20 Sep 2024 13:47:32 GMT
< last-modified: Thu, 19 Sep 2024 14:26:27 GMT
< cache-control: max-age=2592000
< expires: Sun, 20 Oct 2024 13:47:31 GMT

Alkarex · 2024-09-20T14:05:26Z

Well, when nginx is in the equation, it is often the culprit...
An explanation h5bp/server-configs-nginx#230

Problem:

Expiration time is currently determined by content-type.

Content-type is not set for 304 responses.

This means 304 responses gets the default expiration time, which could be very different from the original.

…r 304 responses

Alkarex · 2024-09-20T16:06:12Z

Workaround implemented: Allow servers to return a shorter cache duration for 304 responses, but not longer than the value in the 200 response

Alkarex · 2024-09-20T21:19:33Z

Seems to work quite fine

* SimplePie support for HTTP cache policies Discussion in FreshRSS/simplepie#26 * Bump SimplePie commit * Typos * Typos * Simpler logic * Explicitly disable cache for non-GET flows * Bump SimplePie commit * Bump SimplePie commit * Bump SimplePie commit * Bump SimplePie commit

Alkarex mentioned this pull request Sep 15, 2024

Allow SimplePie updates with composer FreshRSS/FreshRSS#4374

Merged

5 tasks

Alkarex added 2 commits September 15, 2024 19:39

Refined logic

0c7c435

More doc

af830e1

Alkarex mentioned this pull request Sep 15, 2024

[Bug] FreshRSS is sending a referrer when there is none FreshRSS/FreshRSS#6811

Closed

Alkarex added a commit to Alkarex/FreshRSS that referenced this pull request Sep 15, 2024

SimplePie support for HTTP cache policies

958805c

Discussion in FreshRSS/simplepie#26

Alkarex mentioned this pull request Sep 15, 2024

SimplePie support for HTTP cache policies FreshRSS/FreshRSS#6812

Merged

Add support for s-maxage

5793627

Alkarex closed this Sep 15, 2024

Alkarex deleted the negociate_cache_expiration_time branch September 15, 2024 22:25

Alkarex restored the negociate_cache_expiration_time branch September 15, 2024 22:31

Alkarex reopened this Sep 15, 2024

Typos

f8e3f92

Alkarex changed the title ~~Negociate_cache_expiration_time~~ Negotiate cache_expiration_time Sep 15, 2024

Pragmatism?

b72dd25

Typos

9cf1883

Alkarex added 4 commits September 18, 2024 21:44

Remove support of s-maxage

d70f420

Seems to only make sense for CDNs

Move order of headers

0f405df

Remove superfluous comments

545f7cd

Remove wrong comment

b18c7d3

Alkarex added 2 commits September 20, 2024 16:32

Workaround for buggy servers returning wrong cache-control headers fo…

ecee8d4

…r 304 responses

fix logic for 304 workaround

2a60217

Alkarex added 2 commits September 20, 2024 18:08

Fix whitespace

3c35bf5

Minor simplification

88b53f7

Alkarex merged commit 7090eed into freshrss Sep 20, 2024
18 checks passed

Alkarex deleted the negociate_cache_expiration_time branch September 20, 2024 21:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Negotiate cache_expiration_time #26

Negotiate cache_expiration_time #26

Alkarex commented Sep 15, 2024 •

edited

Loading

Alkarex commented Sep 15, 2024 •

edited

Loading

Frenzie commented Sep 15, 2024

Alkarex commented Sep 15, 2024 •

edited

Loading

Alkarex commented Sep 16, 2024

Frenzie commented Sep 16, 2024

Alkarex commented Sep 16, 2024 •

edited

Loading

Frenzie commented Sep 16, 2024

Alkarex commented Sep 18, 2024 •

edited

Loading

Frenzie commented Sep 18, 2024

Alkarex commented Sep 20, 2024 •

edited

Loading

Alkarex commented Sep 20, 2024 •

edited

Loading

Alkarex commented Sep 20, 2024

Alkarex commented Sep 20, 2024

Negotiate cache_expiration_time #26

Negotiate cache_expiration_time #26

Conversation

Alkarex commented Sep 15, 2024 • edited Loading

Alkarex commented Sep 15, 2024 • edited Loading

Frenzie commented Sep 15, 2024

Alkarex commented Sep 15, 2024 • edited Loading

Alkarex commented Sep 16, 2024

Frenzie commented Sep 16, 2024

Alkarex commented Sep 16, 2024 • edited Loading

Frenzie commented Sep 16, 2024

Alkarex commented Sep 18, 2024 • edited Loading

Frenzie commented Sep 18, 2024

Alkarex commented Sep 20, 2024 • edited Loading

Alkarex commented Sep 20, 2024 • edited Loading

Alkarex commented Sep 20, 2024

Alkarex commented Sep 20, 2024

Alkarex commented Sep 15, 2024 •

edited

Loading

Alkarex commented Sep 15, 2024 •

edited

Loading

Alkarex commented Sep 15, 2024 •

edited

Loading

Alkarex commented Sep 16, 2024 •

edited

Loading

Alkarex commented Sep 18, 2024 •

edited

Loading

Alkarex commented Sep 20, 2024 •

edited

Loading

Alkarex commented Sep 20, 2024 •

edited

Loading