Purge Cloudflare cache on update

[abstractionfactory]

Currently, the Cloudflare cache is 4 hours. This is useful for most endpoints, but specifically for search it may not be. We should clear the Cloudflare caches as we upload files and when we update the search index.

[diofeher]

Hi @abstractionfactory !

"as we upload files and when we update the search index."

Do you think a valid approach would be clearing the Cloudflare cache after "Update search Index" step is done?

[abstractionfactory]

@diofeher the main issue is that we don't want to dump the entire cache, but only the files we actually changed, otherwise we are dumping the cache once every 15 minutes, which would run up the costs significantly. This needs to be implemented in Go.

[diofeher]

Oh, I see the problem now!

So let's say we can split this issue in two different parts:

1 - See which specific resources were changed and save that information;

2 - Calling Cloudflare API to purge these specific resources. We can use something like https://developers.cloudflare.com/cache/how-to/purge-cache/purge_by_prefix/ or another endpoint depending on how we identify the resources on step 1.

So I think the biggest issue here is the point 1. I'm still reading the code but it seems that we could use the pg_indexer

registry-ui/search/pg-indexer/main.go

Line 94 in f33c8c2

batchItems, err := readItems(scanner, batchSize)

to know which ones would be the new items to be added/updated/deleted, then we can call CloudFlare API per item or per a batch of items to save network and processing power, preferably.

What do you think?

[abstractionfactory]

No, the pg-indexer is the wrong place to do this as it doesn't update all endpoints. This needs likely needs to be hooked into the index storage as an overlay with additional configurable prefixes to purge.

[diofeher]

When you say "We should clear the Cloudflare caches as we upload files and when we update the search index." :

1 - Update cache from search endpoints

https://search.opentofu.org/module/00inboxtest/bigquery/google/latest URLs like that should be updated if files related to it are changed, right?

2 - By upload files, do you mean which files? Since markdown files are not updated. So, if we change a json file in the provider, should we purge the cache for this provider?

@abstractionfactory

[abstractionfactory]

@diofeher I believe the cache clearing should affect the api.opentofu.org endpoint as the markdown and JSON files are located here. So, whatever we upload in the Cloudflare layer we should also purge from the cache.

The upload is done in the storage layer. An extra overlay could be implemented that takes care of the cache clearing.