Inject system call kernel trace templates in raw2trace #6495
Comments
abhinav92003
added a commit
that referenced
this issue
Dec 6, 2023
Adds support to raw2trace to read system call trace templates from the given file and inject the system call trace templates at the corresponding system call number marker during raw2trace conversion of user-space traces. Adds a new offline file type OFFLINE_FILE_TYPE_KERNEL_SYSCALL_TRACE_TEMPLATES used by such system call trace template files. Adds the burst_syscall_inject.cpp test that writes a sample system call template file and verifies that it can be processed by the analyzer framework; further, it collects a user-space trace for an app function that makes some system calls, then performs raw2trace conversion with system call template injection using the above-mentioned file. It then verifies that the trace templates are injected properly. The sample system call template file written by burst_syscall_inject.cpp also serves as documentation for the expected format of OFFLINE_FILE_TYPE_KERNEL_SYSCALL_TRACE_TEMPLATES files. More aspects of system call trace injection is left to future work: supporting multiple trace templates per system call (e.g. differentiated by system call arguments or return value), or other adjustment to the template at injection time. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Dec 9, 2023
Adds support to raw2trace to read system call kernel trace templates from the given file and inject them at the corresponding system call number marker during raw2trace conversion of user-space traces. Adds a new offline file type OFFLINE_FILE_TYPE_KERNEL_SYSCALL_TRACE_TEMPLATES used by such system call trace template files. Adds the burst_syscall_inject.cpp test that writes a sample system call template file and verifies that it can be processed by the analyzer framework; further, it collects a user-space trace for an app function that makes some system calls, then performs raw2trace conversion with system call template injection using the above-mentioned file. It then verifies that the trace templates are injected properly in the resulting trace. The test also runs basic_counts on the resulting trace to ensure it can be processed by the analyzer framework. Extracts raw2trace logic to write the essential header entries (that is, the entries that are expected by the analyzer framework in the trace) into a separate public API that can be used by other libraries (such as the new burst_syscall_inject test or any other library writing the syscall trace template file) to write valid header entries. The sample system call template file written by burst_syscall_inject.cpp also serves as documentation for the expected format of OFFLINE_FILE_TYPE_KERNEL_SYSCALL_TRACE_TEMPLATES files. More aspects of system call kernel trace injection is left to future work: supporting multiple trace templates per system call (e.g. differentiated by system call arguments or return value), or other adjustment to the template at injection time. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Dec 12, 2023
Fixes print format specifier in a raw2trace verbose level 2 log. This causes a segmentation fault with -verbose 2. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Dec 12, 2023
Fixes print format specifier in a raw2trace verbose level 2 log. This causes a segmentation fault with -verbose 2. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Dec 17, 2023
Adds is_record_kernel() to the memtrace_stream_t API to return whether the stream is currently in the kernel part of the trace. Adds implementation for reader_t and record_reader_t. Adds a new file type OFFLINE_FILE_TYPE_KERNEL_SYSCALL_INSTR_ONLY for kernel instr-only traces collected using Intel-PT. Adjusts various logic (tracer output.cpp, basic_counts.cpp, raw2trace.cpp) to use this type in addition to the existing OFFLINE_FILE_TYPE_KERNEL_SYSCALL. Adds various new invariant checks for traces that also have kernel part for system calls: file type, syscall num in marker, start-end match, prior syscall num marker, prior instr is syscall. Relaxes the read/write memref count check for instr-only kernel PT traces. Also relaxes the PC discontinuity check for first kernel instruction of system call trace. Fixes a branch target invariant error by adding the TRACE_MARKER_TYPE_BRANCH_TARGET marker just prior to indirect branch instrs in ir2trace conversion of Intel-PT traces Runs the invariant checker in the burst_syscall_inject.cpp test which injects a dummy syscall template trace. Runs the invariant checker for Intel-PT traces in a new test. There are some PC discontinuities in the PT trace which is left to future work. For now we ignore these in the test using -test_mode_name. Fixes a bug in raw2trace conversion of Intel-PT traces to add instr encodings only one time per instr. Adds a new raw2trace stat for injected system call trace count. Issue: #6495, #5505
abhinav92003
added a commit
that referenced
this issue
Dec 19, 2023
Adds is_record_kernel() to the memtrace_stream_t API to return whether the stream is currently in a part of the trace corresponding to the kernel execution. This could be either the kernel execution at system calls or context switches. Adds implementation for reader_t, record_reader_t and scheduler_t. Adds a new file type OFFLINE_FILE_TYPE_KERNEL_SYSCALL_INSTR_ONLY for kernel instr-only traces collected using Intel-PT. Adjusts various logic (tracer output.cpp, basic_counts.cpp, raw2trace.cpp) to use this type in addition to the existing OFFLINE_FILE_TYPE_KERNEL_SYSCALL. Adds various new invariant checks for traces that also have kernel part for system calls: file type, syscall num in trace start/end markers, trace start-end marker match, prior syscall num marker, prior syscall instr. Relaxes the read/write memref count check for instr-only kernel PT traces. Also relaxes the PC discontinuity check for the first kernel instruction of the system call traces. Fixes a branch target invariant error by adding the TRACE_MARKER_TYPE_BRANCH_TARGET marker just prior to indirect branch instrs in ir2trace conversion of Intel-PT traces The kernel-to-user transition at the end of system calls is not seamless today because there are noise instructions at the end of PT syscall traces. Added a TODO to add a TRACE_MARKER_TYPE_BRANCH_TARGET marker at the final syscall instr which should be a CTI to the next user-space instr, and also a PC discontinuity invariant check for the kernel-to-user transition using the marker value, when the noise instr problem is fixed. Runs the invariant checker in the burst_syscall_inject.cpp test which injects dummy syscall trace templates. Runs the invariant checker for Intel-PT traces in a new test. There are some PC discontinuities in the PT trace which is left to future work. For now we ignore these in the test using -test_mode_name. Fixes a bug in raw2trace conversion of Intel-PT traces to add instr encodings only one time per instr. Adds a new raw2trace stat for injected system call trace count. Tested the Intel-PT tests locally: ``` $ ctest -VV -R 'SUDO' ... The following tests passed: code_api|client.drpttracer_SUDO-test code_api|tool.drcachesim.phys_SUDO code_api|tool.drcachesim.phys-threads_SUDO code_api|tool.drcacheoff.phys_SUDO code_api|tool.drcacheoff.kernel.simple_SUDO code_api|tool.drcacheoff.kernel.opcode-mix_SUDO code_api|tool.drcacheoff.kernel.syscall-mix_SUDO code_api|tool.drcacheoff.kernel.invariant-checker_SUDO 100% tests passed, 0 tests failed out of 8 ``` Issue: #6495, #5505
abhinav92003
added a commit
that referenced
this issue
Mar 23, 2024
Handles various invariant errors seen in system call trace templates collected on x86 QEMU. Modifies syscall trace template file format to use the TRACE_MARKER_TYPE_SYSCALL_TRACE_START and TRACE_MARKER_TYPE_SYSCALL_TRACE_END markers to show start and end respectively of each syscall trace template, instead of separating them using a TRACE_MARKER_TYPE_SYSCALL marker. This makes it easier to write invariant checks that also work for the syscall trace template file (in addition to an actual trace file injected with trace templates). Handles cases where there are a different number of read/write records than expected; after iret, variants of xrstor, variants of xsaves, and prefetch instrs. Relaxes the PC discontinuity check after hlt, sysret, and within two instrs of sti (which enables interrupts, so there may be an interrupt shortly after, as seen in some QEMU syscall trace templates). Makes other misc changes to make sure the syscall trace template file passes the invariant checker: add thread exit (since we already have a thread start), relaxation of various invariant checks. Adds and implements the instr_is_xrstor API that identifies variants of the xrstor opcode, and adds supervisor versions of xsave to instr_is_xsave. Adds unit tests for these new scenarios. Added a TODO to handle other arch equivalent versions of these scenarios. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Mar 28, 2024
Handles various invariant errors seen in system call trace templates collected on x86 QEMU. Modifies syscall trace template file format to use the TRACE_MARKER_TYPE_SYSCALL_TRACE_START and TRACE_MARKER_TYPE_SYSCALL_TRACE_END markers to show start and end respectively of each syscall trace template, instead of separating them using a TRACE_MARKER_TYPE_SYSCALL marker. This makes it easier to write invariant checks that also work for the syscall trace template file (in addition to an actual trace file injected with trace templates). Also adds cache line size and page size markers to the template, similar to the context switch sequence template file. Handles cases where there are a different number of read/write records than expected by the decoder; after iret, variants of xrstor, variants of xsaves, and prefetch instrs. Relaxes the PC discontinuity check after hlt, and within two instrs of sti (which enables interrupts, so there may be an interrupt shortly after, as empirically seen in some QEMU syscall trace templates). Makes other misc changes to make sure the syscall trace template file passes the invariant checker: add thread exit (since we already have a thread start), relaxation of various invariant checks. Adds and implements the instr_is_xrstor API that identifies variants of the xrstor opcode, and adds supervisor versions of xsave to instr_is_xsave. Adds unit tests for these new scenarios. Added a TODO to handle other arch equivalent versions of these scenarios. Adds a new flag `-abort_on_invariant_error` which is true by default, to allow the user to instruct the invariant checker to continue past invariant errors (using `-no_abort_on_invariant_error`). This is helpful since there are still a few instances of some invariant errors in the syscall trace template that are harder to generalize and fix/ignore. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Apr 3, 2024
Ensures that we write out the buffered entries at each no_fetch repstr instance. This is to avoid overflowing the write buffer, since some syscall trace templates have repstr with high iteration count. Adds a unit test to the existing burst_syscall_inject test that verifies injection of a syscall template with a large repstr iteration count. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Apr 3, 2024
) Writes out the buffered entries at each instr_no_fetch instance. This is to avoid overflowing the raw2trace write buffer, since some syscall trace templates have repstr with a very high iteration count. Adds separate tracking in the basic counts tool for user and kernel nofetch instrs. Adds a unit test to the existing burst_syscall_inject test that verifies injection of a syscall template with a high repstr iteration count. Also removes some log spam in the burst_syscall_inject test. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Apr 10, 2024
Adapts more invariant checks to the kernel system call trace templates collected on QEMU. Skips tracking the retaddr_stack_ for calls and rets inside the system call trace part, since this is used only to verify the function marker return address which is user-space only. Allows reads following xsaveopt which apparently also reads the header of the xsave area. Issue: #6495
abhinav92003
added a commit
that referenced
this issue
Apr 11, 2024
Adapts more invariant checks to the kernel system call trace templates collected on QEMU. Skips tracking the retaddr_stack_ for calls and rets inside the system call trace part, since this is used only to verify the function marker return address which is user-space only. Allows reads following xsave which apparently also reads the header of the xsave area (https://www.felixcloutier.com/x86/xsave). Relaxes the check that verifies that the instr preceding the syscall trace is a system call, for the case when there are no encodings in the trace. This is particularly useful for some unit test setup. Issue: #6495
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Today drmemtrace supports tracing kernel execution during system calls using
-enable_kernel_tracing, but this feature is available only on x86 machines with the Intel-PT feature. Another possible way to get user+kernel traces is to collect templates for system call kernel execution using QEMU/Gem5 etc and then inject them using raw2trace at the system call marker locations in the user-space trace.Such traces would not have the actual kernel execution done during system calls, but they may still be useful to model certain aspects of the application's execution. E.g. a cache simulator may yield better results for user-space MPKI because the kernel interference is modelled to some extent. More experimentation should reveal how useful this method can be.
To start, we can collect and inject only one trace template per system call, but later we may collect multiple trace templates for each system call, differentiated perhaps by its arguments or the return value and inject the appropriate one in the trace during raw2trace. We may also adjust some aspects of the trace template while injecting it; e.g. use a different base address for the kernel code or data accesses.
The text was updated successfully, but these errors were encountered: