-
Notifications
You must be signed in to change notification settings - Fork 570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable concurrent raw trace processing by factoring out core conversion functionality #3129
Labels
Comments
derekbruening
pushed a commit
that referenced
this issue
Aug 20, 2018
Trace conversion is CPU bound. To enable alternative, parallel processing of raw files, we refactor raw2trace. Here we extract the module parsing and mapping functionality into a new class module_mapper_t. The new class contains its own versions of the following raw2trace_t APIs: handle_custom_data(), do_module_parsing(), do_module_parsing_and_mapping(), and find_mapped_trace_address(). The raw2trace versions remain but are deprecated. Uses in samples and tests are updated here to use module_mapper_t. Issue: #3129
mtrofin
added a commit
to mtrofin/dynamorio
that referenced
this issue
Aug 20, 2018
Summarize instr_t to decouple trace conversion from DR internals (e.g. instru_t APIs) Issue: DynamoRIO#3129
mtrofin
added a commit
to mtrofin/dynamorio
that referenced
this issue
Aug 22, 2018
This allows us to move raw2trace_t implementation to the header, into an upcoming reusable template, without incurring unwanted dependencies. Issue: DynamoRIO#3129
derekbruening
pushed a commit
that referenced
this issue
Aug 22, 2018
This allows us to move the raw2trace_t implementation to the header, into an upcoming reusable template, without incurring unwanted dependencies. Issue: #3129
mtrofin
added a commit
to mtrofin/dynamorio
that referenced
this issue
Aug 22, 2018
Added thread start checking/validation APIs, and a related, general-purpose API for fetching out the timestamp out of a raw trace. Issue: DynamoRIO#3129
derekbruening
pushed a commit
that referenced
this issue
Aug 22, 2018
Adds thread start checking/validation APIs to raw2trace, and a related, general-purpose API for fetching the timestamp out of a raw trace to drmemtrace. Adds testing to the raw2trace_io test. Issue: #3129
mtrofin
added a commit
to mtrofin/dynamorio
that referenced
this issue
Aug 23, 2018
Refactored raw2trace in prepartion for peeling off reusable functionality into a template. Issue: DynamoRIO#3129
mtrofin
added a commit
that referenced
this issue
Aug 24, 2018
This move the code we plan to offer as part of a reusable template to raw2trace.h. The code isn't modified, except: - the "CHECK" macro used in raw2trace.cpp was copied to the header and renamed "DR_CHECK", to avoid collisions. The code moved to the header uses DR_CHECK instead of CHECK. This macro is undefined as soon as it's not used anymore. - FAULT_INTERRUPTED_BB and INVALID_THREAD_ID became static const members of raw2trace_t instead of macros, to avoid adding macros to the header file. Issue: #3129
mtrofin
added a commit
that referenced
this issue
Aug 24, 2018
This move the code we plan to offer as part of a reusable template to raw2trace.h. The code isn't modified, except: - the "CHECK" macro used in raw2trace.cpp was copied to the header and renamed "DR_CHECK", to avoid collisions. The code moved to the header uses DR_CHECK instead of CHECK. This macro is undefined as soon as it's not used anymore. - FAULT_INTERRUPTED_BB and INVALID_THREAD_ID became static const members of raw2trace_t instead of macros, to avoid adding macros to the header file. Issue: #3129
mtrofin
added a commit
that referenced
this issue
Aug 27, 2018
Introducing trace_converter_t, a template factoring out of raw2trace_t the functionality necessary for building custom trace converters. Issue #3129
mtrofin
added a commit
that referenced
this issue
Aug 29, 2018
…callback We avoid holding on to global state past module_mapper_t construction. However, we need to remember the custom free_cb callback, if provided, as it will be needed when ~module_mapper_t is called. Issue #3129
mtrofin
added a commit
that referenced
this issue
Aug 30, 2018
The operator is implicitly deleted because a member (cached_user_free) is a const pointer. Issue: #3129
mtrofin
added a commit
that referenced
this issue
Sep 3, 2018
The move ctors for trace_converter_t and module_mapper_t should be public. Issue: #3129
mtrofin
added a commit
that referenced
this issue
Sep 4, 2018
The move ctors for trace_converter_t and module_mapper_t should be public. Issue: #3129
mtrofin
added a commit
that referenced
this issue
Sep 11, 2018
The code handling interrupted basic blocks assumed serial processing of thread buffers. If the thread buffers are processed in parallel, it might be the last offline_entry_t in a buffer was a memref. We can't look past it to see if the BB was interrupted. Issue: #3129
mtrofin
added a commit
that referenced
this issue
Sep 12, 2018
The code handling interrupted basic blocks assumed serial processing of thread buffers. If the thread buffers are processed in parallel, it might be the last offline_entry_t in a buffer was a memref. We can't look past it to see if the BB was interrupted. The CL leverages the currently-made assumption that a faulting memref will not have its signal/exception marker split into the next buffer, so if this memref is the last one there can be no interruption. Issue: #3129
mtrofin
added a commit
that referenced
this issue
Sep 12, 2018
module_mapper_t should have move semantics, but simply relying on default ctors/assignment operands isn't enough: we also need to clear the originating object, otherwise the destructor would free resources twice; and the assignment operator would have to figure out what to do with the original resources. We don't need an overly-complicated design, and there's very little value to be gained from module_mapper_t being stack-allocatable. This CL hides constructors and instead offers a static API that returns a unique pointer. Issue: #3129
mtrofin
added a commit
that referenced
this issue
Sep 13, 2018
…3163) module_mapper_t should have move semantics, but simply relying on default ctors/assignment operands isn't enough: we also need to clear the originating object, otherwise the destructor would free resources twice; and the assignment operator would have to figure out what to do with the original resources. We don't need an overly-complicated design, and there's very little value to be gained from module_mapper_t being stack-allocatable. This CL hides constructors and instead offers a static API that returns a unique pointer. Issue: #3129
The trace file split #3230 will enable concurrent post-processing. |
derekbruening
added a commit
that referenced
this issue
Dec 7, 2018
Adds concurrent post-processing of raw thread files by default. Adds a new option -jobs for selecting the worker thread pool size. Uses a simple static round-robin work assignment of traced threads to worker threads. Splits the decode cache into a separate cache per worker, to keep it lockless. Adds a per-traced-thread last_decode_pc mini-cache to speed up rep string loop processing. Issue: #3230 Fixes #3129
derekbruening
added a commit
that referenced
this issue
Dec 7, 2018
Adds concurrent post-processing of raw thread files by default. Adds a new option -jobs for selecting the worker thread pool size. Uses a simple static round-robin work assignment of traced threads to worker threads. Splits the decode cache into a separate cache per worker, to keep it lockless. Adds a per-traced-thread last_decode_pc mini-cache to speed up rep string loop processing. Looks for and requires libpthread (and libm and libdl) even for BUILD_TESTS=OFF. Issue: #3230 Fixes #3129
This was referenced Jan 28, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Trace conversion (raw2trace) is CPU bound. This issue tracks factoring out the core conversion logic, such that extensions may be created, employing specialized parallelization techniques, to speed up trace processing.
The text was updated successfully, but these errors were encountered: