Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add branch target address to drmemtrace traces for use without instruction encodings #5490

Closed
derekbruening opened this issue May 11, 2022 · 0 comments · Fixed by #6219
Closed

Add branch target address to drmemtrace traces for use without instruction encodings #5490

derekbruening opened this issue May 11, 2022 · 0 comments · Fixed by #6219
Labels
Component-DrMemtrace Google-Affecting Type-Feature

Comments

@derekbruening
Copy link
Contributor

When a drmemtrace and associated application binaries are available to supply
the instruction encodings, the targets of non-taken branches are obtained
from the encodings (and targets of taken branches from the subsequent PC -- which
has complications across intervening kernel transfers, solved with special kernel
transfer fields, and across thread interleavings, solved by tweaking those in raw2trace for
offline at least). However, the binaries not always available (e.g., the Google workload traces).

Adding the target of every conditional branch could be done in raw2trace.
The main cost would be a size increase. If the average block is 5 instrs + 2 addrs then
this would be a 14% increase scaled down a little since not all branches
are conditional, but still a non-trivial increase of likely >10% by this
un-measured back-of-envelope estimate anyway.

Presumably we would implement this in post-processing, waiting for the next
block, rather than having online instrumentation deduce the target before
the branch occurs.
This was referenced Jul 17, 2023
Closed
Merged
derekbruening added a commit that referenced this issue Jul 21, 2023
@derekbruening
i#6213,i#5490: Add drmemtrace branch targets up front
5aecd44 
The drmemtrace record type TRACE_TYPE_INSTR_CONDITIONAL_JUMP is now
deprecated in offline traces where it is replaced by
TRACE_TYPE_INSTR_TAKEN_JUMP and TRACE_TYPE_INSTR_UNTAKEN_JUMP.  The
version number is bumped to accomplish this.

Indirect branches in drmemtrace traces now contain a marker holding
the actual target (TRACE_MARKER_TYPE_BRANCH_TARGET), which immediately
precedes the branch record.

These changes are implemented inside raw2trace and in PT ir2trace.
raw2trace tests are updated and new tests added.

Special cases with new logic and tests include:
+ Rseq side exits must specially set whether taken or untaken
+ Rseq abort rollbacks to a branch are tested as this is the case
  where it was not easy to identify whether a branch was taken
  in the past (it required particular inferences).
+ Branches prior to signals.
+ Trace-final branches: we just delete these.

Adds view support:

```
     2212815     1648444:     1249326 ifetch       6 byte(s) @ 0x00007f3406720707 48 3d 01 f0 ff ff    cmp    %rax, $0xfffff001
     2212816     1648445:     1249326 ifetch       2 byte(s) @ 0x00007f340672070d 73 01                jnb    $0x00007f3406720710 (untaken)
     2212817     1648445:     1249326 <marker: indirect branch target 0x7f34066a8b37>
     2212818     1648446:     1249326 ifetch       1 byte(s) @ 0x00007f340672070f c3                   ret
     2212819     1648446:     1249326 read         8 byte(s) @ 0x00007ffd91e24fa8 by PC 0x00007f340672070f
     2212820     1648447:     1249326 ifetch       5 byte(s) @ 0x00007f34066a8b37 4c 8b 54 24 48       mov    0x48(%rsp), %r10
```

Adds several new invariant checks and augments the existing PC
continuity checks.  This required a little refactoring to check
branches before signals which is part of #5912.  Unit tests for each
cases are added.

Updates the documentation and the changelist.

Issue: #5490, #6213, #5912
Fixes #6213
Fixes #5490
@derekbruening derekbruening mentioned this issue Jul 21, 2023
Merged
derekbruening added a commit that referenced this issue Jul 25, 2023
@derekbruening
i#6213,i#5490: Add drmemtrace branch targets up front (#6219)
2220a83 
The drmemtrace record type TRACE_TYPE_INSTR_CONDITIONAL_JUMP is now
deprecated in offline traces where it is replaced by
TRACE_TYPE_INSTR_TAKEN_JUMP and TRACE_TYPE_INSTR_UNTAKEN_JUMP. The
version number is bumped to accomplish this.

Indirect branches in drmemtrace traces now contain a marker holding the
actual target (TRACE_MARKER_TYPE_BRANCH_TARGET), which immediately
precedes the branch record.

These changes are implemented inside raw2trace and in PT ir2trace.
raw2trace tests are updated and new tests added.

Special cases with new logic and tests include:
+ Rseq side exits must specially set whether taken or untaken
+ Rseq abort rollbacks to a branch are tested as this is the case where
it was not easy to identify whether a branch was taken in the past (it
required particular inferences).
+ Branches prior to signals.
+ Trace-final and window-final branches: we just delete these.

Adds view support:

```
     2212815     1648444:     1249326 ifetch       6 byte(s) @ 0x00007f3406720707 48 3d 01 f0 ff ff    cmp    %rax, $0xfffff001
     2212816     1648445:     1249326 ifetch       2 byte(s) @ 0x00007f340672070d 73 01                jnb    $0x00007f3406720710 (untaken)
     2212817     1648445:     1249326 <marker: indirect branch target 0x7f34066a8b37>
     2212818     1648446:     1249326 ifetch       1 byte(s) @ 0x00007f340672070f c3                   ret
     2212819     1648446:     1249326 read         8 byte(s) @ 0x00007ffd91e24fa8 by PC 0x00007f340672070f
     2212820     1648447:     1249326 ifetch       5 byte(s) @ 0x00007f34066a8b37 4c 8b 54 24 48       mov    0x48(%rsp), %r10
```

Adds several new invariant checks and augments the existing PC
continuity checks. This required a little refactoring to check branches
before signals which is part of #5912. Unit tests for each case are
added.  Adds a couple of comments on issues that #5912 should
address.

Updates the documentation to remove the documented guarantee
that branches are delayed.

Updates the changelist.

Issue: #5490, #6213, #5912
Fixes #6213
Fixes #5490
derekbruening added a commit to DynamoRIO/drmemtrace_samples that referenced this issue Jul 26, 2023
@derekbruening
Update traces for new branch decoration
83f1e70 
Updates the x86_64 and aarch64 traces and README files with new traces
that contain the new taken/untaken and indirect branch target
decoration.

Issue: DynamoRIO/dynamorio#5490
Issue: DynamoRIO/dynamorio#6213
derekbruening added a commit that referenced this issue Jul 26, 2023
@derekbruening
i#6213,i#5490: Update drmemtrace docs on new branch decoration
00f399f 
Updates the drmemtrace record format docs to include the new branch
decoration.

Issue: #6213, #5490
@derekbruening derekbruening mentioned this issue Jul 26, 2023
Merged
derekbruening added a commit that referenced this issue Jul 26, 2023
@derekbruening
i#6213,i#5490: Update drmemtrace docs on new branch decoration (#6229)
e759e40 
Updates the drmemtrace record format docs to include the new branch
decoration.

Issue: #6213, #5490
ivankyluk pushed a commit to ivankyluk/dynamorio that referenced this issue Jul 28, 2023
@derekbruening @ivankyluk
i#6213,i#5490: Add drmemtrace branch targets up front (DynamoRIO#6219)
55c46bc 
The drmemtrace record type TRACE_TYPE_INSTR_CONDITIONAL_JUMP is now
deprecated in offline traces where it is replaced by
TRACE_TYPE_INSTR_TAKEN_JUMP and TRACE_TYPE_INSTR_UNTAKEN_JUMP. The
version number is bumped to accomplish this.

Indirect branches in drmemtrace traces now contain a marker holding the
actual target (TRACE_MARKER_TYPE_BRANCH_TARGET), which immediately
precedes the branch record.

These changes are implemented inside raw2trace and in PT ir2trace.
raw2trace tests are updated and new tests added.

Special cases with new logic and tests include:
+ Rseq side exits must specially set whether taken or untaken
+ Rseq abort rollbacks to a branch are tested as this is the case where
it was not easy to identify whether a branch was taken in the past (it
required particular inferences).
+ Branches prior to signals.
+ Trace-final and window-final branches: we just delete these.

Adds view support:

```
     2212815     1648444:     1249326 ifetch       6 byte(s) @ 0x00007f3406720707 48 3d 01 f0 ff ff    cmp    %rax, $0xfffff001
     2212816     1648445:     1249326 ifetch       2 byte(s) @ 0x00007f340672070d 73 01                jnb    $0x00007f3406720710 (untaken)
     2212817     1648445:     1249326 <marker: indirect branch target 0x7f34066a8b37>
     2212818     1648446:     1249326 ifetch       1 byte(s) @ 0x00007f340672070f c3                   ret
     2212819     1648446:     1249326 read         8 byte(s) @ 0x00007ffd91e24fa8 by PC 0x00007f340672070f
     2212820     1648447:     1249326 ifetch       5 byte(s) @ 0x00007f34066a8b37 4c 8b 54 24 48       mov    0x48(%rsp), %r10
```

Adds several new invariant checks and augments the existing PC
continuity checks. This required a little refactoring to check branches
before signals which is part of DynamoRIO#5912. Unit tests for each case are
added.  Adds a couple of comments on issues that DynamoRIO#5912 should
address.

Updates the documentation to remove the documented guarantee
that branches are delayed.

Updates the changelist.

Issue: DynamoRIO#5490, DynamoRIO#6213, DynamoRIO#5912
Fixes DynamoRIO#6213
Fixes DynamoRIO#5490
ivankyluk pushed a commit to ivankyluk/dynamorio that referenced this issue Jul 28, 2023
@derekbruening @ivankyluk
i#6213,i#5490: Update drmemtrace docs on new branch decoration (Dynam…
dc03db1 
…oRIO#6229)

Updates the drmemtrace record format docs to include the new branch
decoration.

Issue: DynamoRIO#6213, DynamoRIO#5490
ivankyluk pushed a commit to ivankyluk/dynamorio that referenced this issue Jul 29, 2023
@derekbruening @ivankyluk
i#6213,i#5490: Add drmemtrace branch targets up front (DynamoRIO#6219)
d37eee7 
The drmemtrace record type TRACE_TYPE_INSTR_CONDITIONAL_JUMP is now
deprecated in offline traces where it is replaced by
TRACE_TYPE_INSTR_TAKEN_JUMP and TRACE_TYPE_INSTR_UNTAKEN_JUMP. The
version number is bumped to accomplish this.

Indirect branches in drmemtrace traces now contain a marker holding the
actual target (TRACE_MARKER_TYPE_BRANCH_TARGET), which immediately
precedes the branch record.

These changes are implemented inside raw2trace and in PT ir2trace.
raw2trace tests are updated and new tests added.

Special cases with new logic and tests include:
+ Rseq side exits must specially set whether taken or untaken
+ Rseq abort rollbacks to a branch are tested as this is the case where
it was not easy to identify whether a branch was taken in the past (it
required particular inferences).
+ Branches prior to signals.
+ Trace-final and window-final branches: we just delete these.

Adds view support:

```
     2212815     1648444:     1249326 ifetch       6 byte(s) @ 0x00007f3406720707 48 3d 01 f0 ff ff    cmp    %rax, $0xfffff001
     2212816     1648445:     1249326 ifetch       2 byte(s) @ 0x00007f340672070d 73 01                jnb    $0x00007f3406720710 (untaken)
     2212817     1648445:     1249326 <marker: indirect branch target 0x7f34066a8b37>
     2212818     1648446:     1249326 ifetch       1 byte(s) @ 0x00007f340672070f c3                   ret
     2212819     1648446:     1249326 read         8 byte(s) @ 0x00007ffd91e24fa8 by PC 0x00007f340672070f
     2212820     1648447:     1249326 ifetch       5 byte(s) @ 0x00007f34066a8b37 4c 8b 54 24 48       mov    0x48(%rsp), %r10
```

Adds several new invariant checks and augments the existing PC
continuity checks. This required a little refactoring to check branches
before signals which is part of DynamoRIO#5912. Unit tests for each case are
added.  Adds a couple of comments on issues that DynamoRIO#5912 should
address.

Updates the documentation to remove the documented guarantee
that branches are delayed.

Updates the changelist.

Issue: DynamoRIO#5490, DynamoRIO#6213, DynamoRIO#5912
Fixes DynamoRIO#6213
Fixes DynamoRIO#5490
derekbruening added a commit that referenced this issue Aug 3, 2023
@derekbruening
i#5490 ind br tgt: Stop skip and switch at target marker
7ed74dc 
Updates reader_t::skip_instructions() and the scheduler's context
switches and replay to stop at an indirect branch target marker
instead of at the subsequent instruction record itself.

Adds an indirect branch to allasm_x86 and uses it in a new
skip_unit_tests test.

Adds a new raw2trace test that an indirect branch marker is after a
chunk boundary.

Adds scheduler tests that a context switch during live and replay
modes goes to the marker.

Updates the invariant checker to avoid failure on the offline.skip*
tests where the filetype record is not seen: now the checker uses the
stream to get the filetype, except for the checker unit tests (added a
XXX comment on possibly giving them a valid stream).

Issue: #5490
@derekbruening derekbruening mentioned this issue Aug 3, 2023
Merged
derekbruening added a commit that referenced this issue Aug 4, 2023
@derekbruening
i#5490 ind br tgt: Stop skip and switch at target marker (#6241)
bd11f79 
Updates reader_t::skip_instructions() and the scheduler's context
switches and replay to stop at an indirect branch target marker instead
of at the subsequent instruction record itself.

Adds an indirect branch to allasm_x86 and uses it in a new
skip_unit_tests test.

Adds a new raw2trace test that an indirect branch marker is after a
chunk boundary.

Adds scheduler tests that a context switch during live and replay modes
goes to the marker.

Updates the invariant checker to avoid failure on the offline.skip*
tests where the filetype record is not seen: now the checker uses the
stream to get the filetype, except for the checker unit tests (added a
XXX comment on possibly giving them a valid stream).

Issue: #5490
derekbruening added a commit that referenced this issue Aug 4, 2023
@derekbruening
i#5490 ind br tgt: Only set flag when recording
cd0fd1c 
Follow-up to PR #6241: only set the switching_pre_instruction flag
when recording.  Otherwise an assert added by the PR fires.

Adds a no-record test which reproduces the assert and passes with the
fix.

Issue: #5490
This was referenced Aug 4, 2023
Closed
Merged
derekbruening added a commit that referenced this issue Aug 5, 2023
@derekbruening
Revert "i#5490 ind br tgt: Stop skip and switch at target marker (#6241…
267ac80 
…)" (#6248)

This reverts commit bd11f79. It fails
to apply a -1 to other outputs waiting for an input to reach its start
instruction count when the input is at the pre-instr marker. The
solution is non-trivial so we revert for now.

Issue: #5490
derekbruening added a commit that referenced this issue Aug 8, 2023
@derekbruening
i#5490 ib targets: Add memref ind branch target field
14dbc1d 
Adds a new "indirect_branch_target" field to memref_t.instr.
Converts the new TRACE_MARKER_TYPE_BRANCH_TARGET marker to
an internal record used by reader_t to fill in the new field.

Updates the invariant checker and test.
Updates the legacy test's marker counts.
Updates the documentation.

Issue: #5490
@derekbruening derekbruening mentioned this issue Aug 8, 2023
Merged
derekbruening added a commit to DynamoRIO/drmemtrace_samples that referenced this issue Aug 8, 2023
@derekbruening
Update for new indirect branch target field
b7b48eb 
Updates the x86_64 and aarch64 trace view and count output in the
README files for the new indirect branch target field.

Issue: DynamoRIO/dynamorio#5490
derekbruening added a commit that referenced this issue Aug 9, 2023
@derekbruening
i#5490 ib targets: Add memref ind branch target field (#6250)
402aede 
Adds a new "indirect_branch_target" field to memref_t.instr. Converts
the new TRACE_MARKER_TYPE_BRANCH_TARGET marker to an internal record
used by reader_t to fill in the new field.

Updates the invariant checker and test.
Updates the legacy test's marker counts.
Updates the documentation.

Issue: #5490
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component-DrMemtrace Google-Affecting Type-Feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

i#6213,i#5490: Add drmemtrace branch targets up front DynamoRIO/dynamorio
1 participant
@derekbruening