Pluggable compression for cache values. #42
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
skatkov
changed the title
dim
requested changes
Sep 22, 2023
dim
reviewed
Sep 22, 2023
lib/generators/cache/database/templates/add_cache_compression_column.rb
Outdated
Show resolved
Hide resolved
For storing cache entries compression can be very beneficial, as most renders from Rails are actually strings with whitespace and all. This allows for pluggable compression methods, with 'plain' and 'gzip' being available by default. If desired, additional compression methods (such as Brotli) can be added later on. The entries will be decompressed with the appropriate decompression algorithm since the compression method gets recorded in the cache entries table, so a switch to compressed entries is both reversible and deployable gradually - without having to delete existing cache data.
The compression method can be specified by adding the `compression:` kwarg when instantiating the Store:
Store.new(compression: 'gzip')
skatkov
force-pushed
the
variable-compression
branch
from
626416d to
e514bfd
Compare
dim
requested changes
Sep 25, 2023
| @@ -131,7 +138,7 @@ def write_multi_entries(hash, **_options) | |||
| def entry_attributes(entry) | |||
| expires_at = Time.zone.at(entry.expires_at) if entry.expires_at | |||
|
|
|||
| { value: Marshal.dump(entry.value), version: entry.version.presence, expires_at: expires_at } | |||
| compression_attributes(entry.value).merge(version: entry.version.presence, expires_at: expires_at) | |||
There was a problem hiding this comment.
pls use update, not merge here too. this will allocate a new object and there's no need
| def decompress(record) | ||
| return Marshal.load(record.value) if record.compression.nil? | ||
|
|
||
| COMPRESSION_HANDLERS.fetch(record.compression).decompress(record.value) |
There was a problem hiding this comment.
you still need to Marshal.load the result
There was a problem hiding this comment.
sorry, one last bit, could you please add a test to demonstrate that this is working/was not working before? something where you are e.g. storing a (large enough) Hash in the cache.
added 2 commits
September 25, 2023 16:50
|
Thanks! Looks like there are some CI failures though. |
skatkov
force-pushed
the
variable-compression
branch
from
ff27ab3 to
69c9c93
Compare
skatkov
force-pushed
the
variable-compression
branch
from
69c9c93 to
6c37835
Compare
|
@dim resolved, sorry about that -) |
dim
reviewed
Sep 26, 2023
| @@ -21,6 +24,10 @@ def self.supports_cache_versioning? | |||
| # option options [Class] :model model class. Default: ActiveSupport::Cache::DatabaseStore::Model | |||
There was a problem hiding this comment.
Sorry, one thing I forgot to check. What I meant by "documentation" is adding a line here too, to document the new option. I would appreciate if could you do a quick fix PR?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We deployed database-cache-database gem to one of our endpoints and for ~3000 records we're averaging ~1.3 GB memory that cache table takes. It seems a bit excessive.
Usually, most cache stores in Rails deploy compression to minimize storage requirements (with on difference of local storage). With standard gzip we're able to cut down 85% of data stored with very little overhead on write/read operation.
Some other compression algorithms can do it slightly better and slightly faster (like brotli), but adding another dependency to project might not be everyone's preference. This PR offers pluggable compression solution (only gzip support at this point) with 'plain' compression as default. This gives backward-compatibility for existing installations.
This PR depends on #41