Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable concurrent raw trace processing by factoring out core conversion functionality #3129

Closed
mtrofin opened this issue Aug 17, 2018 · 1 comment
Assignees

Comments

@mtrofin
Copy link
Contributor

mtrofin commented Aug 17, 2018

Trace conversion (raw2trace) is CPU bound. This issue tracks factoring out the core conversion logic, such that extensions may be created, employing specialized parallelization techniques, to speed up trace processing.

derekbruening pushed a commit that referenced this issue Aug 20, 2018
Trace conversion is CPU bound. To enable alternative, parallel
processing of raw files, we refactor raw2trace.  Here we extract
the module parsing and mapping functionality into a new class
module_mapper_t.  The new class contains its own versions of the
following raw2trace_t APIs: handle_custom_data(), do_module_parsing(),
do_module_parsing_and_mapping(), and find_mapped_trace_address().
The raw2trace versions remain but are deprecated.  Uses in samples
and tests are updated here to use module_mapper_t.

Issue: #3129
mtrofin added a commit to mtrofin/dynamorio that referenced this issue Aug 20, 2018
Summarize instr_t to decouple trace conversion from DR internals (e.g.
instru_t APIs)

Issue: DynamoRIO#3129
derekbruening pushed a commit that referenced this issue Aug 21, 2018
…3130)

Summarize instr_t with a custom data structure that contains just the direct information
needed for raw2trace, rather than querying the instr_t structure, to decouple trace conversion
from DR internals (e.g., instru_t APIs) and improve performance.

Issue: #3129
mtrofin added a commit to mtrofin/dynamorio that referenced this issue Aug 22, 2018
This allows us to move raw2trace_t implementation to the header,
into an upcoming reusable template, without incurring unwanted
dependencies.

Issue: DynamoRIO#3129
derekbruening pushed a commit that referenced this issue Aug 22, 2018
This allows us to move the raw2trace_t implementation to the header,
into an upcoming reusable template, without incurring unwanted
dependencies.

Issue: #3129
mtrofin added a commit to mtrofin/dynamorio that referenced this issue Aug 22, 2018
Added thread start checking/validation APIs, and a related, general-purpose API for fetching out the timestamp out of a raw trace.

Issue: DynamoRIO#3129
derekbruening pushed a commit that referenced this issue Aug 22, 2018
Adds thread start checking/validation APIs to raw2trace, and a related, general-purpose API for
fetching the timestamp out of a raw trace to drmemtrace.
Adds testing to the raw2trace_io test.

Issue: #3129
mtrofin added a commit to mtrofin/dynamorio that referenced this issue Aug 23, 2018
Refactored raw2trace in prepartion for peeling off reusable
functionality into a template.

Issue: DynamoRIO#3129
derekbruening pushed a commit that referenced this issue Aug 24, 2018
…3145)

Refactored raw2trace in preparation for peeling off reusable
functionality into a template.  Adds indirection of logging and
file i/o.

Issue: #3129
mtrofin added a commit that referenced this issue Aug 24, 2018
This move the code we plan to offer as part of a reusable template
to raw2trace.h. The code isn't modified, except:
- the "CHECK" macro used in raw2trace.cpp was copied to the header
and renamed "DR_CHECK", to avoid collisions. The code moved to the
header uses DR_CHECK instead of CHECK. This macro is undefined
as soon as it's not used anymore.
- FAULT_INTERRUPTED_BB and INVALID_THREAD_ID became static const
members of raw2trace_t instead of macros, to avoid adding macros to
the header file.

Issue: #3129
mtrofin added a commit that referenced this issue Aug 24, 2018
This move the code we plan to offer as part of a reusable template to raw2trace.h. The code isn't modified, except:

- the "CHECK" macro used in raw2trace.cpp was copied to the header and renamed "DR_CHECK", to avoid collisions. The code moved to the header uses DR_CHECK instead of CHECK. This macro is undefined as soon as it's not used anymore.

- FAULT_INTERRUPTED_BB and INVALID_THREAD_ID became static const members of raw2trace_t instead of macros, to avoid adding macros to the header file.

Issue: #3129
mtrofin added a commit that referenced this issue Aug 27, 2018
Introducing trace_converter_t, a template factoring out of raw2trace_t
the functionality necessary for building custom trace converters.

Issue #3129
mtrofin added a commit that referenced this issue Aug 28, 2018
…3149)

Introducing trace_converter_t, a template factoring out of raw2trace_t the functionality necessary for building custom trace converters.

Issue #3129
mtrofin added a commit that referenced this issue Aug 29, 2018
…callback

We avoid holding on to global state past module_mapper_t
construction. However, we need to remember the custom free_cb
callback, if provided, as it will be needed when ~module_mapper_t is called.

Issue #3129
mtrofin added a commit that referenced this issue Aug 29, 2018
…callback (#3155)

We avoid holding on to global state past module_mapper_t construction. However, we need to remember the custom free_cb callback, if provided, as it will be needed when ~module_mapper_t is called.

Issue: #3129
mtrofin added a commit that referenced this issue Aug 30, 2018
The operator is implicitly deleted because a member (cached_user_free)
is a const pointer.

Issue: #3129
mtrofin added a commit that referenced this issue Aug 30, 2018
#3156)

The operator is implicitly deleted because a member (cached_user_free) is a const pointer.

Issue: #3129
mtrofin added a commit that referenced this issue Sep 3, 2018
The move ctors for trace_converter_t and module_mapper_t should be public.

Issue: #3129
mtrofin added a commit that referenced this issue Sep 4, 2018
The move ctors for trace_converter_t and module_mapper_t should be public.

Issue: #3129
mtrofin added a commit that referenced this issue Sep 11, 2018
The code handling interrupted basic blocks assumed serial processing
of thread buffers. If the thread buffers are processed in parallel, it
might be the last offline_entry_t in a buffer was a memref. We can't
look past it to see if the BB was interrupted.

Issue: #3129
mtrofin added a commit that referenced this issue Sep 12, 2018
The code handling interrupted basic blocks assumed serial processing of thread buffers. If the thread buffers are processed in parallel, it might be the last offline_entry_t in a buffer was a memref. We can't look past it to see if the BB was interrupted.

The CL leverages the currently-made assumption that a faulting memref will not have its signal/exception marker split into the next buffer, so if this memref is the last one there can be no interruption.

Issue: #3129
mtrofin added a commit that referenced this issue Sep 12, 2018
module_mapper_t should have move semantics, but simply relying on
default ctors/assignment operands isn't enough: we also need to clear
the originating object, otherwise the destructor would free resources
twice; and the assignment operator would have to figure out what to
do with the original resources.

We don't need an overly-complicated design, and there's very little
value to be gained from module_mapper_t being stack-allocatable. This
CL hides constructors and instead offers a static API that returns a
unique pointer.

Issue: #3129
mtrofin added a commit that referenced this issue Sep 13, 2018
…3163)

module_mapper_t should have move semantics, but simply relying on default ctors/assignment operands isn't enough: we also need to clear the originating object, otherwise the destructor would free resources twice; and the assignment operator would have to figure out what to do with the original resources.

We don't need an overly-complicated design, and there's very little value to be gained from module_mapper_t being stack-allocatable. This CL hides constructors and instead offers a static API that returns a unique pointer.

Issue: #3129
@derekbruening derekbruening self-assigned this Nov 16, 2018
@derekbruening
Copy link
Contributor

The trace file split #3230 will enable concurrent post-processing.

derekbruening added a commit that referenced this issue Dec 7, 2018
Adds concurrent post-processing of raw thread files by default.
Adds a new option -jobs for selecting the worker thread pool size.
Uses a simple static round-robin work assignment of traced threads to
worker threads.

Splits the decode cache into a separate cache per worker, to keep it
lockless.

Adds a per-traced-thread last_decode_pc mini-cache to speed up rep
string loop processing.

Issue: #3230
Fixes #3129
derekbruening added a commit that referenced this issue Dec 7, 2018
Adds concurrent post-processing of raw thread files by default.
Adds a new option -jobs for selecting the worker thread pool size.
Uses a simple static round-robin work assignment of traced threads to
worker threads.

Splits the decode cache into a separate cache per worker, to keep it
lockless.

Adds a per-traced-thread last_decode_pc mini-cache to speed up rep
string loop processing.

Looks for and requires libpthread (and libm and libdl) even for BUILD_TESTS=OFF.

Issue: #3230
Fixes #3129
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants