Split PBF Parser into a single Read and a single Parse thread to increase parsing speed. #20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ease parsing speed.
| @@ -32,6 +32,84 @@ or see http://www.gnu.org/licenses/agpl.txt. | |||
| #include "ExtractorStructs.h" | |||
There was a problem hiding this comment.
Since concurrent_queue is not just a small struct, it is justified to put into its own class file
There was a problem hiding this comment.
Will split up into its own file.
|
Great addition, I put a few comments into the pull request |
|
|
||
| bool empty() const | ||
| { | ||
| boost::mutex::scoped_lock lock(queue_mutex); |
There was a problem hiding this comment.
Is a lock on the empty function really necessary. This is a concurrent read and it does not ensure that the queue does not change after the lock goes out of scope.
DennisOSRM
pushed a commit
that referenced
this pull request
Nov 3, 2011
Split PBF Parser into a single Read and a single Parse thread to increase parsing speed.
|
Thanks |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I noticed extracting was inefficient in the original threading model, and I wasn't making any significant gains by increasing the number of threads to more than 1 (the default). I split the PBF Parser code into two separate threads instead - one thread reads all data from the PBF, and another thread parses all the data. The Read thread places all data into a thread-safe queue. I added a maximum queue size limit to avoid issues with reading all of the data into RAM (data is read quicker than it is parsed). I noticed an improvement of around 30% over original parsing method. Threading is done with Boost threads. I removed the OMP pragmas as they were no longer necessary.
Note - Read/Parse methods are NOT thread safe any longer. Only one thread may Read and Parse at the same time. I experimented with multiple Parse threads, but the improvement was negligible due to greedy locking of the STXXLContainers object. I'm guessing a better locking mechanism may improve the performance, though I did not have time to test it out.