Add interoperability with W^X policies

[derekbruening]

This is a feature request to add support in DR for working in the presence of W^X policies. These PaX-like policies typically disallow execution from any page that came from mmap +wx or ever had mprotect +x applied to it, which rules out writing and later marking read-only. The proposal here is to use a dual-mapping strategy: having two virtual pages backed by the same physical page, with one +rw for writing and the other +rx for execution.

This will likely be limited to 64-bit Linux but could be extended to Windows in the future if desired.

With #1132 in place, we can make a new reservation the same size as vmcode and have a constant offset between each pair of pages. We would then keep all variables tracking addresses as scalars, and when we want to write, we would simply add the offset.

It may be best to first extend #774 and #1132 to make the size of vmcode 2G by default, to avoid any issues with out-of-reservation allocations.

[derekbruening]

I am observing an interesting performance issue: I have to use mmap to incrementally commit instead of mprotect, since most W^X policies do not like mprotect. With mmap, I see a huge (4-5 second!) delay at init time apparently due to the kernel IMA (Integrity Measurement Architecture) going and computing sha256 digests of every page in our anonymous mmap. Doing this for an anonymous mmap seems bizarre. It happens for both memfd_create and /dev/shm as the backing file.

We can see the delay scaling by vm size:

$ for i in 8M 32M 64M 128M 256M 512M 1G; do /usr/bin/time bin64/drrun -vm_size $i -satisfy_w_xor_x -- suite/tests/bin/simple_app 2>&1 | grep elapsed; done
0.01user 0.07system 0:00.10elapsed 83%CPU (0avgtext+0avgdata 4304maxresident)k
0.01user 0.15system 0:00.16elapsed 100%CPU (0avgtext+0avgdata 4316maxresident)k
0.00user 0.30system 0:00.31elapsed 100%CPU (0avgtext+0avgdata 4268maxresident)k
0.01user 0.59system 0:00.60elapsed 99%CPU (0avgtext+0avgdata 4224maxresident)k
0.01user 1.20system 0:01.23elapsed 98%CPU (0avgtext+0avgdata 4212maxresident)k
0.01user 2.36system 0:02.39elapsed 99%CPU (0avgtext+0avgdata 4312maxresident)k
0.01user 4.72system 0:04.75elapsed 99%CPU (0avgtext+0avgdata 4336maxresident)k

Note how it's all kernel time. perf shows things like:

-   98.33%     0.00%  simple_app  [kernel.kallsyms]  [k] ima_file_mmap
     ima_file_mmap
     process_measurement
     ima_collect_measurement
   + ima_calc_file_hash
+   98.33%     0.00%  simple_app  [kernel.kallsyms]  [k] process_measurement
+   98.33%     0.00%  simple_app  [kernel.kallsyms]  [k] ima_collect_measurement
+   98.33%     0.01%  simple_app  [kernel.kallsyms]  [k] ima_calc_file_hash
+   93.76%     0.06%  simple_app  [kernel.kallsyms]  [k] crypto_shash_update
+   93.70%     0.06%  simple_app  [kernel.kallsyms]  [k] crypto_sha256_update
+   93.64%     0.22%  simple_app  [kernel.kallsyms]  [k] sha256_generic_block_fn
+   93.51%    89.91%  simple_app  [kernel.kallsyms]  [k] sha256_transform

On kernels without IMA enabled the delay is not there.

[derekbruening]

A simple reproducer shows that the hash is only performed on the first mmap +x of any size, and covers the full file size. Subsequent mmaps do not incur extra overhead. So this is limited to DR init and will not show up later during app execution.

$ gcc -o ima_overhead ima_overhead.c  && /usr/bin/time ./ima_overhead 
                 Created memfd:        5 usec elapsed
                Truncated size:       27 usec elapsed
                    Mapped +rw:       34 usec elapsed
                    Mapped +rx:  9661297 usec elapsed
            Mapped another +rx:  9661339 usec elapsed
                       Success:  9777377 usec elapsed
0.00user 9.77system 0:09.77elapsed 99%CPU (0avgtext+0avgdata 1296maxresident)k

[derekbruening]

I messed the number up in my commits so manually adding references here:

• 26ea967 i#3566 w^x: Work around IMA overhead (i#3566 w^x: Work around IMA overhead #3612)
• bc0aa77 i#3566 w^x: Add -satisfy_w_xor_x for Linux (i#3566 w^x: Add -satisfy_w_xor_x for Linux #3607)

[derekbruening]

I failed to think about how to handle fork. We have to use MAP_SHARED to mirror the views, so we will not get copy-on-write (even if we did we wouldn't get mirroring in a copy). I think the child has to make a new memfd file and copy all the old content into new views? For a large vmcode that could take a while. Fortunately most forks are done early in a process and rarely late in a large (esp multi-threaded) app?

[derekbruening]

We also have trouble working with dr_nonheap_alloc() with +wx memory. Since I don't think we can turn this on by default yet I'm going to just issue a usage error in that case.

[derekbruening]

The fork copy solution of course has a race. I'm not sure there's a better solution. For now I may live with the race as the alternative is having the parent wait.

Summarizing where we are:

    /* XXX i#3566: Support for W^X has some current limitations:
     * + It is not implemented for Windows or Mac.
     * + Fork is not perfectly supported: there is overhead and a race.
     * + Pcaches are not supported.
     * + -native_exec_list is not supported.
     * + dr_nonheap_alloc(rwx) is not supported.
     *   Clients using other non-vmcode sources of +wx memory will also not comply.
     */

[derekbruening]

I see two solutions to the fork race:

1. Parent waits for child to finish copying.
2. Parent makes a temp copy and continues. When child is done copying from
   the temp it sets some flag so the parent can lazily free the temp copy.

I'm not sure which is better.
The copy size: for -no_reachable_heap (which maybe we should disallow with
-satisfy_w_xor_x) it's just vmcode but that can be 200MB+ for very large apps.
As mentioned above, hopefully such apps do not fork often.

[derekbruening]

DR_ALLOC_CACHE_REACHABLE is in the same boat as dr_nonheap_alloc: we'd have to expose the mapping function to support it. It should also have an assert for now.

[derekbruening]

Just clarifying one reason this is left open: how to enable clients to operate properly wrt W^X.
See the above comments on DR_ALLOC_CACHE_REACHABLE, etc. Exposing the mapping function sounds confusing to clients (and it should have a different name for data: s/executable/reachable/).

Better would be to shrink the vmcode mapping and have a 2nd one there for
reachable heap so nobody has to do any mapping, but that has sizing issues.
Xref the client lib being in the same space and the default 1G vs 2G issues.