Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parquet for caching responses #7

Open
mattpodolak opened this issue Feb 17, 2021 · 1 comment
Open

parquet for caching responses #7

mattpodolak opened this issue Feb 17, 2021 · 1 comment
Assignees
Labels
feature request A feature request for future enhancements

Comments

@mattpodolak
Copy link
Owner

mattpodolak commented Feb 17, 2021

Use parquet instead of pickle for caching responses.

Need to assess if this is a reasonable improvement:

  • Benchmark parquet cache speed for storing and retrieval against pickle
  • Compare resulting cache sizes
@mattpodolak mattpodolak added the feature request A feature request for future enhancements label Feb 17, 2021
@mattpodolak mattpodolak self-assigned this Feb 19, 2021
@mattpodolak
Copy link
Owner Author

This could be expanded by increasing the generality of the caching mechanism to identify the type of file used in the cache and load accordingly.

Saving responses to cache should be performed with a default type (pickle/parquet/feather/jay) - whichever is fastest, with optional config to select a caching function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request A feature request for future enhancements
Projects
None yet
Development

No branches or pull requests

1 participant