Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fetch non-cached reports in parallel #1554

Merged
merged 3 commits into from
Jun 8, 2016
Merged

Fetch non-cached reports in parallel #1554

merged 3 commits into from
Jun 8, 2016

Conversation

jml
Copy link
Contributor

@jml jml commented Jun 7, 2016

Currently, if a bunch of requested reports aren't in the cache, we fetch them one at a time inline with the request. This patch updates the code to fetch reports in parallel instead.

While making the change, I noticed that we handled S3 errors differently from gzip & decoder errors. I've updated the code to handle all errors in the same way (i.e. no special logging, just return).

Currently, getNonCached returns only the first error it finds. It should probably aggregate all of the errors somehow so we aren't silently ignoring important conditions. I don't know how best to do that.

Along the way, I extracted the logic for fetching a single report (to make the actual parallelism logic more clear).


This change is Reviewable

reports = append(reports, rep)
c.cache.Set(reportKey, rep)
reports = append(reports, *r.report)
c.cache.Set(r.key, *r.report)

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

@tomwilkie
Copy link
Contributor

Did you see any performance improvement running this on your laptop?

@jml
Copy link
Contributor Author

jml commented Jun 7, 2016

Haven't actually run it on my laptop, so will at least do that before landing.

I'll also at least attempt to write unit tests (but might end up deciding they're too hard).

@jml
Copy link
Contributor Author

jml commented Jun 8, 2016

Ran this locally & verified that it doesn't break. No metrics. Won't bother with unit tests now.

@tomwilkie OK to merge?

@jml
Copy link
Contributor Author

jml commented Jun 8, 2016

@tomwilkie asked for metrics:

Before

20160608-143026_3196x1717_scrot

After

20160608-144305_3196x1717_scrot

Running on my laptop locally.

@tomwilkie
Copy link
Contributor

LGTM

@jml jml merged commit 38af5e0 into master Jun 8, 2016
@jml jml deleted the parallelism branch June 8, 2016 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants