-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add datafacade allocator backed by mmap #4881
Conversation
As a first prototype I tried to see what the general overhead is for using No renumberingWith renumberingYou can see that without memory pressure both perform about the same. The hope here is that renumbering will improve the locality. |
We ran some experiments using the bicycle profile on North-America. To test the behavior under memory pressure, we first determined the minimal memory usage as 5.8G (first time serving queries completed without an OOM). Shows are the results for random queries which are generally the worst-case. The difference between 20% and 25% is quite large since we start to trash pages at a high rate due to the random access. For a |
93718b5
to
6dd5143
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! I have two remarks to fix appveyor builds.
include/util/mmap_file.hpp
Outdated
{ | ||
// Create a new file with the given size in bytes | ||
boost::iostreams::mapped_file_params params; | ||
params.path = file.c_str(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
file.string()
would be better here as params.path
is std::string
.
include/util/mmap_file.hpp
Outdated
// Create a new file with the given size in bytes | ||
boost::iostreams::mapped_file_params params; | ||
params.path = file.c_str(); | ||
params.mode = std::ios::in | std::ios::out; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mode
is deprecated, better to use params.flags = boost::iostreams::mapped_file::readwrite;
ab7971a
to
d67e400
Compare
@oxidase can you check if this works for you? 🙇♂️ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@TheMarex works 👍, lgtm
Issue
This is a first step towards #1947. We don't mmap the OSRM dataset directly but create a new file that basically stores the memory content as it is now. This is less effective both from a disk storage and startup time perspective, but should give us a good idea how the query performance using mmap without any optimizations.
Tasklist
Requirements / Relations
#1947