Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caching for remote files during extraction #2

Open
awagner-mainz opened this issue Sep 2, 2016 · 2 comments
Open

Caching for remote files during extraction #2

awagner-mainz opened this issue Sep 2, 2016 · 2 comments

Comments

@awagner-mainz
Copy link
Member

When resources are fetched from remote servers, download them once and use the cached version instead of re-downloading them every time.

(Code is operational at my place (TM), move it to git branch and test it!)

Think about:

  • invalidate/clean cache
@metacontext
Copy link
Member

metacontext commented May 10, 2020

To keep cache pruning simple I think it would suffice to add a slim cleanup function that gets called last on return of the main query. All resources fetched for a specific extraction could be saved to a subfolder in /temp named with a temporary identifier. The cleanup function could then delete the whole cache folder with a xmldb:remove during return.

@metacontext metacontext changed the title [feature request] cache remote files Caching for remote files during extraction May 10, 2020
@metacontext
Copy link
Member

Just to keep all thoughts we had about this feature so far here is a related snippet from my TODO.txt:

Only load external resources once => retrieve and cache them for all following statements
maybe by checking the configuration first for all resource attributes, load them and temp. put them in DB with URI <=> hashname map
file: cache/extraction_1235612rauafd.xml

<cachedResources>
<resource key="http://my.gnd.resource/12345" value="temp/13717dhgqf.xml" />
</cachedResources>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants