switch to dstack at DR initial time

[derekbruening]

From [email protected] on March 07, 2013 12:11:56

Currently, DR use app stack at initialization time, which may have some problems on x64, especially after r1979 , "Use a stack allocated buffer for private_instr_encode"

The problem is I create some instrlist at dr_init including instruction "mov instr => [mem]", DR converts it to "mov immed => [mem]" and then private_instr_encode try to encode it. Because DR uses app stack, the immed would be something value like 0x7fff43d0e980, then a CLIENT_ASSERT triggered in encode_immed:1428
1427 if (size == OPSZ_4) { 1428 CLIENT_ASSERT((val > 0) && (val < INT_MAX), 1429 "encode error: immediate has invalid size"); 1430 }

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=1105

[derekbruening]

From [email protected] on March 07, 2013 09:18:05

the reachability assert should perhaps be a separate issue.

perhaps r1970 should be reverted now to eliminate that issue ASAP as it will break anyone using instr ptrs on x64 linux or windows 8

being on the app stack also complicates priv lib TEB StackLimit which I had to work around as part of issue #1102

[derekbruening]

From [email protected] on March 07, 2013 09:54:01

How about deferring the encode_immed() assert until encode time? We can propagate _IF_DEBUG(assert_reachable) into encode_immed(), I think.

[derekbruening]

From [email protected] on March 07, 2013 10:11:30

The comment #2 solution isn't perfect because the user could be doing something like encoding to a buffer, ignoring reachability, copying, and fixing up ctis manually. If they do that, they miss out on the truncation assertions. I might be able to live with that though, since it's a semi-advanced use case. A novice user will try to use encode_to_copy() with a final_pc first.

[derekbruening]

From [email protected] on April 26, 2013 08:47:08

r1970 was reverted in r1972

[derekbruening]

From [email protected] on April 26, 2013 09:47:21

probably we'd fix issue #1152 and issue #626 and issue #1105 simultaneously

I just hit another place where being on app stack is painful: https://code.google.com/p/drmemory/issues/detail?id=1196

Owner: ---

[derekbruening]

From [email protected] on June 05, 2013 20:41:28

Once this is fixed we should make sure to swap TEB.StackLimit (xref https://code.google.com/p/drmemory/issues/detail?id=1262 )