ForkserverExecutor #111

tokatoka · 2021-05-19T12:37:54Z

#82

Only the ForkserverExecutor, still WIP
TODO:

use Pipe made by Dominik.
We need to check if the execution has crashed
We might want to allocate the shared memory (calling StdShMemProvider::new()) outside the Executor?, because the Observer needs that address.
Add an example that uses ForkserverExecutor

…erver

tokatoka · 2021-05-19T12:41:23Z

ehm, the commit log went messy when I rebased forkserver onto the main branch...

domenukk · 2021-05-19T13:22:22Z

libafl/src/executors/forkserver.rs

+            .setlimit(memlimit)
+            .setsid()
+            .setstdin(out_filefd, use_stdin)
+            .setpipe(st_pipe.clone(), ctl_pipe.clone())


I think we don't need Arc here, read and write ends are independent of each other

but if we don't wrap it in Arc then Pipe's drop() would be prematurely called after setpipe, and following communication through the ctl_pipe and st_pipe won't work.
(and we can't pass normal reference & to Pipe either)

ah, ok.
I guess it would simpler to pass 4 RawFds (st_pipe.read_end, st_pipe.write_end, ctl_pipe.read_end, ctl_pipe.write_end) to setpipe, then no Arc is needed

Wouldn't it on drop the pipe in the parent after the child got spawned, in which case it is fine, though? Or does it drop before exec, too?
Maybe we need to ManuallyDrop, then?
In any case we shouldn't need Arc since we don't have more than 1 access to each side of the pipe I think (worst case, Rc?)

it drops after the forkserver is spawned (so after the match Command::new(target) line) for the parent,

unsafe { libc::close(ctl_pipe.read_end); libc::close(st_pipe.write_end); }

these 2 are safe to close for the parent, but if the Pipe::drop() is called here, the remaining two ends (ctl_pipe.write_end, st_pipe.read_end) would also be closed, and we don't want that.
what we want instead is calling Pipe::drop() when the forkserver dies.
I'll look into ManuallyDrop.
I tried to use Rc, but pre_exec in std::os::unix::process::CommandExt wants std::marker::Send, so it didn't work.

We can change the Pipe interface to anything that makes sense.
I feel like we should move out the two ends for the forkserver, so they get dropped in the parent, and keep the respective other ends around in the fs parent, right?

domenukk · 2021-05-19T13:22:46Z

libafl/src/executors/forkserver.rs

+
+    fn setpipe(&mut self, st_pipe: Arc<Pipe>, ctl_pipe: Arc<Pipe>) -> &mut Self {
+        let func = move || {
+            let ret = unsafe { libc::dup2(ctl_pipe.read_end, FORKSRV_FD) };


We have a wrapper for dup2 in bolts::os.

domenukk · 2021-05-19T13:24:04Z

libafl/src/executors/forkserver.rs

+    use crate::inputs::NopInput;
+    #[test]
+    fn test_forkserver() {
+        let command = "/home/toka/work/aflsimple/test";


Would be cool to have a testcase, but this one can't work, sadly :)

Yes 😄, I think I'll add a simple example binary file and a c source code file along with a example fuzzer.

I think we should add a libafl-tests project that can have more complex testcases, such as a short fuzz session, etc.
Committing binaries will break on various platforms, and in various colors. :)

… but trying to figure out other solutions

tokatoka · 2021-05-23T11:24:14Z

ah.. I'm sorry, I did not notice that the test code hung up.. (and it has run for around 80 minutes..)

tokatoka · 2021-05-25T06:02:36Z

Hi, thanks for the help andrea!

@tokatoka is ~3k exec/sec the speed that you was expecting?

Yes I was expecting around 3k exec/sec.
but with this commit e772742, the fuzzer forkserver_simple still does not run as fast as AFL/afl-fuzz when it is run with cargo run on my computer,
(still around 200 exec/sec after several minutes)
am I missing some compiler options to build forkserver_test.o?

andreafioraldi · 2021-05-25T06:28:41Z

Hi, thanks for the help andrea!

@tokatoka is ~3k exec/sec the speed that you was expecting?

Yes I was expecting around 3k exec/sec.
but with this commit e772742, the fuzzer forkserver_simple still does not run as fast as AFL/afl-fuzz when it is run with cargo run on my computer,
(still around 200 exec/sec after several minutes)
am I missing some compiler options to build forkserver_test.o?

--release

tokatoka · 2021-05-25T06:31:24Z

--release

😲
ah, I'm so stupid...
OK!, now it is working as I am expecting.

tokatoka · 2021-05-25T10:17:25Z

I'd like to add a feature like forkserver_test for libafl_tests, and then we can modify build.rs so that we fetch AFL++, compile AFL++ and compile forkserver_test.c only when forkserver_test feature is enabled.
but I also want to make the test in forkserver.rs dependent on libafl_tests's forkserver_test feature (not libafl's feature!), (otherwise we don't have the binary file, forkserver_test.o).
What would be a good workaround for this?
(I think yes, we can just check libafl_tests/src/forkserver_test.o is existent for the forkserver.rs's test to work, but that looks somewhat ugly)

domenukk · 2021-05-25T10:20:48Z

Just add minimial tests to forkserver.rs and do the "real" testing in libafl_tests. No need to test it two times.

* add Forkserver, Pipe Outfile struct * add forkserver executor struct, and shmem init * close pipes in the destructor of Forkserver * fill pre_exec to write out the inputs * fix * read_st, write_ctl * more handshakes * wrap Pipe in Arc, fill post_exec * add Forkserver, Pipe Outfile struct * add forkserver executor struct, and shmem init * close pipes in the destructor of Forkserver * fill pre_exec to write out the inputs * fix * read_st, write_ctl * more handshakes * wrap Pipe in Arc, fill post_exec * fix for the lastest HasExecHooks trait * use Dominik's pipe, remove Arc and temporarily pass RawFd to setstdin but trying to figure out other solutions * add libafl_tests, put a very simple vulnerable program * fix * added forkserver_simple (mostly copy-pasted from babyfuzzer) * fix test * handle crash in post_exec * add README.md * check exec time to see why it's so slow * remove double invokation of is_interesting for the obejctive * make forkserver_simple AFL-like and improve speed * some debugging help * do not evaluate feedback if solution * speedup the things * working input placement via stdin in Forkserver * don't call panic! but return errors, rewrite some comments * use AFLplusplus/afl-cc instead of AFL * use .cur_input like AFL * bring the test for forkserver back * add better README.md message * failing the initial handshake should return an error * delete some commented-out code * format * format * ForkserverExecutor needs std and is unix-only for now * clippy * OutFile error handling * fmt * clippy * don't build libafl_tests on windows * fix * keep test in forkserver.rs simple * add forkserver_test feature for libafl_tests * format * some doc Co-authored-by: Andrea Fioraldi <[email protected]>

tokatoka added 18 commits May 9, 2021 22:10

add Forkserver, Pipe Outfile struct

7d86e1a

add forkserver executor struct, and shmem init

4242bc2

close pipes in the destructor of Forkserver

529f6c7

fill pre_exec to write out the inputs

5370afe

fix

1c6c5c1

read_st, write_ctl

46ca808

more handshakes

3142fe9

wrap Pipe in Arc, fill post_exec

52b1223

add Forkserver, Pipe Outfile struct

f9de53b

add forkserver executor struct, and shmem init

a0f6053

close pipes in the destructor of Forkserver

8c6986f

fill pre_exec to write out the inputs

c80afe7

fix

a563c15

read_st, write_ctl

871e3a5

more handshakes

1f32d9e

wrap Pipe in Arc, fill post_exec

fcbc8fe

Merge branch 'forkserver' of github.com:AFLplusplus/LibAFL into forks…

14fe041

…erver

fix for the lastest HasExecHooks trait

786398f

domenukk reviewed May 19, 2021

View reviewed changes

tokatoka added 2 commits May 21, 2021 17:24

Merge branch 'main' into forkserver

eac76c6

use Dominik's pipe, remove Arc and temporarily pass RawFd to setstdin…

8fb668c

… but trying to figure out other solutions

andreafioraldi mentioned this pull request May 22, 2021

Proposed projects #119

Open

10 tasks

tokatoka added 3 commits May 23, 2021 17:47

add libafl_tests, put a very simple vulnerable program

205a0c8

fix

c1e3eb3

added forkserver_simple (mostly copy-pasted from babyfuzzer)

1d7ecc1

fix test

d8deb80

use AFLplusplus/afl-cc instead of AFL

c132eb5

use .cur_input like AFL

2aeae6d

tokatoka added 13 commits May 25, 2021 16:40

bring the test for forkserver back

7dc74db

add better README.md message

9e09b3f

failing the initial handshake should return an error

ad51f67

delete some commented-out code

aba50b3

format

bc96363

format

d4df1f6

ForkserverExecutor needs std and is unix-only for now

6a4582b

clippy

ab5c1c6

OutFile error handling

538b917

fmt

dcff4ac

clippy

882adfe

don't build libafl_tests on windows

8d0e64b

fix

1d284f4

tokatoka marked this pull request as ready for review May 25, 2021 10:05

tokatoka added 3 commits May 25, 2021 19:46

keep test in forkserver.rs simple

0c68a5f

add forkserver_test feature for libafl_tests

3970801

format

1f9a69a

tokatoka changed the title ~~WIP: ForkserverExecutor~~ ForkserverExecutor May 25, 2021

some doc

29fcbf1

andreafioraldi merged commit d4410c0 into main May 25, 2021

tokatoka mentioned this pull request May 28, 2021

Timeout forkserver #136

Merged

tokatoka deleted the forkserver branch June 2, 2021 07:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ForkserverExecutor #111

ForkserverExecutor #111

tokatoka commented May 19, 2021 •

edited

Loading

tokatoka commented May 19, 2021

domenukk May 19, 2021

tokatoka May 19, 2021 •

edited

Loading

tokatoka May 19, 2021 •

edited

Loading

domenukk May 19, 2021

tokatoka May 19, 2021

domenukk May 19, 2021

domenukk May 19, 2021

domenukk May 19, 2021

tokatoka May 19, 2021 •

edited

Loading

domenukk May 19, 2021

tokatoka commented May 23, 2021 •

edited

Loading

tokatoka commented May 25, 2021 •

edited

Loading

andreafioraldi commented May 25, 2021

tokatoka commented May 25, 2021

tokatoka commented May 25, 2021 •

edited

Loading

domenukk commented May 25, 2021

ForkserverExecutor #111

ForkserverExecutor #111

Conversation

tokatoka commented May 19, 2021 • edited Loading

tokatoka commented May 19, 2021

domenukk May 19, 2021

Choose a reason for hiding this comment

tokatoka May 19, 2021 • edited Loading

Choose a reason for hiding this comment

tokatoka May 19, 2021 • edited Loading

Choose a reason for hiding this comment

domenukk May 19, 2021

Choose a reason for hiding this comment

tokatoka May 19, 2021

Choose a reason for hiding this comment

domenukk May 19, 2021

Choose a reason for hiding this comment

domenukk May 19, 2021

Choose a reason for hiding this comment

domenukk May 19, 2021

Choose a reason for hiding this comment

tokatoka May 19, 2021 • edited Loading

Choose a reason for hiding this comment

domenukk May 19, 2021

Choose a reason for hiding this comment

tokatoka commented May 23, 2021 • edited Loading

tokatoka commented May 25, 2021 • edited Loading

andreafioraldi commented May 25, 2021

tokatoka commented May 25, 2021

tokatoka commented May 25, 2021 • edited Loading

domenukk commented May 25, 2021

tokatoka commented May 19, 2021 •

edited

Loading

tokatoka May 19, 2021 •

edited

Loading

tokatoka May 19, 2021 •

edited

Loading

tokatoka May 19, 2021 •

edited

Loading

tokatoka commented May 23, 2021 •

edited

Loading

tokatoka commented May 25, 2021 •

edited

Loading

tokatoka commented May 25, 2021 •

edited

Loading