Feature blocky benchmark #541
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hardbyte
requested changes
Apr 16, 2020
There was a problem hiding this comment.
The benchmarking code looks pretty good to me. I'd like to see some indication of the number and size of of blocks used in each experiment. Ideally we could configure the experiments to run with different sizes but I realize that would be tricky.
Please update docs/benchmarking.rst before merging, in particular it would be good to see more about how the blocks were created.
A feature request - we could show some form of progress in the benchmark containers output. If we printed the result token the user could attach a rest_client to watch progress if they really wanted.
The performance it reveals is another story 🤢
Running on my desktop I see uploading 100k clknblock takes ~45s versus ~6s for binary encodings!
During upload I see I log the size of each block (opps) and that most(?) block has just 1 element.
For the 100k x 100k experiment it creates 112551 chunks. Creating the chunks appears to take almost 3 minutes. I scaled up to using 10 workers to give it a chance of finishing. On my machine one chunk is taking as much as 50ms - although I saw some ~10ms. My cpu cores are all <20% active during this process :-/
benchmarking/benchmark.py
Outdated
| and `clk_{user}_{size_data}.json` where $user is a letter starting from `a` indexing the data owner, and `size_data` | ||
| is a integer representing the number of data rows in the dataset (e.g. 10000). Note that the csv usually has a header. | ||
| the 3 party linkage), and then a number a file following the format `PII_{user}_{size_data}.csv`, | ||
| `clk_{user}_{size_data}_v2.bin`, `clk_{user}_{size_data}.json` and `clknblocks_{user}_{size_data}.json` where $user |
There was a problem hiding this comment.
Suggested change
| `clk_{user}_{size_data}_v2.bin`, `clk_{user}_{size_data}.json` and `clknblocks_{user}_{size_data}.json` where $user | |
| `clk_{user}_{size_data}_v2.bin`, `clk_{user}_{size_data}.json` and `clknblocks_{user}_{size_data}.json` where `user` |
benchmarking/benchmark.py
Outdated
| is a integer representing the number of data rows in the dataset (e.g. 10000). Note that the csv usually has a header. | ||
| the 3 party linkage), and then a number a file following the format `PII_{user}_{size_data}.csv`, | ||
| `clk_{user}_{size_data}_v2.bin`, `clk_{user}_{size_data}.json` and `clknblocks_{user}_{size_data}.json` where $user | ||
| is a letter starting from `a` indexing the data owner, and `size_data` is a integer representing the number of data |
There was a problem hiding this comment.
Suggested change
| is a letter starting from `a` indexing the data owner, and `size_data` is a integer representing the number of data | |
| is a letter starting from `a` indexing the data providers, and `size_data` is an integer representing the number of |
|
I left it running last night with a 12 hour timeout, 6 workers and worker settings: Benchmark LogsThe gist is one 100k x 100k run failed with a timeout, and the other took |
hardbyte
approved these changes
Apr 16, 2020
|
Re: binary encodings. we should definitely look into allowing to upload binary clks and block info in separate files for big jobs. |
This was referenced Apr 30, 2020
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This extends the benchmark script to be able to run experiments which use blocking.
An experiment definition for blocking looks like this:
The corresponding 'clknblocks' files are uploaded to S3.
For now I haven't changed the
default-experiements.jsonfile, as the blocked experiments take a very long time and will most likely trigger a timeout.Once we addressed that issue in the entity service, we can replace
default-experiements.jsonwithdefault-experiements-wawo-blocking.json.