Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include solr query string in cache key #396

Merged
merged 3 commits into from
Jan 21, 2016
Merged

Conversation

defect
Copy link
Contributor

@defect defect commented Jan 19, 2016

When doing queries across many collins instances (using multi-collins) we cache the response from the remote instances for 30 seconds. The problem is that we do not include the CQL query in the key which can result in two different queries returning the same thing.

This PR changes the behavior by adding the solr query string to the cache key, which should eliminate the problem. I'm "unprotecting" traverseQueryString() but I think that should be fine as the method doesn't modify or remove anything, it just traverses the solr expression and builds the query string.

@tumblr/systems rfr

@defect defect added the bug label Jan 19, 2016
@@ -54,7 +54,7 @@ case class AssetFinder(
updatedAfter.map(t => "updatedAfter" -> Formatter.dateFormat(t)) ::
updatedBefore.map(t => "updatedBefore" -> Formatter.dateFormat(t)) ::
state.map(s => "state" -> s.name) ::
query.map { q => "query" -> "UHOH!!!!" } :: //FIXME: need toCQL traversal

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is magical.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this might be my favorite bug

@byxorna
Copy link
Contributor

byxorna commented Jan 20, 2016

💅 game on fleek @defect!

@@ -3,4 +3,5 @@ multicollins {
instanceAssetType = DATA_CENTER
locationAttribute = LOCATION
thisInstance = NONE
queryTimeout = 30
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this value queryTimeout but the config class is referencing queryCacheTimeout?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because i'm sloppy :) Fix incoming.

@defect defect force-pushed the felix-remote_query_cache branch from 10bfffd to 6ff36d8 Compare January 20, 2016 03:30
@defect defect force-pushed the felix-remote_query_cache branch from 6ff36d8 to 74b2e80 Compare January 20, 2016 13:46
@william-richard
Copy link
Contributor

LGTM - @defect please also be sure to make a PR to update the multicollins parameters docs. https://github.com/tumblr/collins/blob/gh-pages/_includes/configuration/multicollins.html#L23-51

@defect
Copy link
Contributor Author

defect commented Jan 21, 2016

@Primer42 #399 opened

@roymarantz
Copy link
Contributor

Do documentation changes go in a different PR? How about updated tests?
The code LGTM, but this is not my favorite language :-)

@defect
Copy link
Contributor Author

defect commented Jan 21, 2016

Yes, documentation goes in a different PR (#399) since those are pulled in to another branch (gh-pages). I also added a test to make sure the cql query gets included.

@schallert
Copy link
Contributor

LGTM. What a find :shipit:

defect added a commit that referenced this pull request Jan 21, 2016
Include solr query string in cache key
@defect defect merged commit b59293c into master Jan 21, 2016
@roymarantz
Copy link
Contributor

👍

@defect defect mentioned this pull request Feb 1, 2016
@william-richard william-richard deleted the felix-remote_query_cache branch April 26, 2016 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants