[WIP] Frontends synchronization, resuming and statistics

[GrosQuildu]

This PR:

• fixes angora and eclipser compilation mode
• rename compiler (like "dotnet") to runner (not to mixup with compilators like clang)
• changes statistic handling (--fuzzer_out for fuzzer output, by default populate_stats and print_stats is called that ideally should print statistic in uniform fashion)
• fixes run method (deepstate-* runs and manages one fuzzer process in (possibly infinite) loop: communicate/sleep for sync_cycle seconds, then do stats printing and syncing, repeat)
• Working out directory structure for each fuzzer:

For fuzzer syncing and resuming it's important to find out how each of them handle input and output directories. Once known, we may make it uniform as much as possible.

Now each fuzzer instance uses output_test_dir as workspace. It should be either empty directory or the same dir as used by previous fuzzing run. So for resuming, we can run deepstate-afl -o out, stop fuzzer and then again deepstate-afl -o out. Note that we may resume only from exactly the same frontend.

Inside workspace frontends automatically create required dirs as described below. This way it should be easier to handle different fuzzers (as opposed to one, shared directory) since some of the fuzzers implements auto-syncing and some not (require restart). Also, this way we know exactly where to push or from where pull testcases.

Frontends may use this variables:

• self.push_dir
• self.pull_dir
• self.crash_dir

afl

• output stats

  • GUI if normal output (curses + clear)
  • stdout incremental stats if out redirected to file
  • fuzzer_stats file

• parallel

  • needs to run in sync mode (-S)
  • copy output_dir/the_afl/queue/* to each instance
  • push testcases into output_dir/some_tool/queue/ (files must be sequentially numbered id:nnnnnn)
  • maybe copy fuzzer_stats also

• proposed dir structure

  • output_dir (-o option)
    • sync_dir
      • queue (push testcases here)
    • the_fuzzer (-S option)
      • crashes
      • hangs
      • queue (pull testcases from here)

• resume

  • --input to - (-i-)
  • output to output_dir
  • afl automatically copy inputs to output_dir/the_afl/queue
  • creates multiple output_dir/the_afl/crashes* directories
    • must handle that: merge in post_exec

libfuzzer

• output stats

  • stdout incremental stats
  • -print_final_stats
  • writes testcases to first directory

• parallel

  • auto testcases reload (-reload)
  • only the first directory is reloaded
  • arbitrary filenames

• proposed dir structure

  • output_dir
    • sync_dir
      • queue (first directory, push pull here, new testcases generated here)
    • the_fuzzer
      • crashes (-artifact_prefix)

• resume

  • just use sync_dir/queue
  • libfuzzer automatically copy inputs to sync_dir/queue

honggfuzz

• output stats
  GUI + stdout incremental stats
  HONGGFUZZ.REPORT.TXT, not too much info
  --verbose: simpler incremental stdout
  --logfile: write output to this file (disables --verbose)

• parallel

  • not implemented ([Question] Run-time sync/reseed support? google/honggfuzz#125)
  • push new testcases into input dir and restart hoggfuzz

• dir structure

  • output_dir
    • sync_dir
      • queue (--output, push pull here, new testcases generated here)
    • the_fuzzer
      • crashes (--crashdir)

• resume

  • push testcases into --input directory

angora

• output stats
  GUI stdout

• parallel

  • AFL sync implemented
  • --sync_afl plus -o output pointing to afl's output
  • new seeds must have id > angora's max id

• dir structure (with --sync_afl)

  • output_dir (-o option)
    • sync_dir (create it for each instance)
      • queue (push testcases here)
    • angora
      • crashes
      • hangs
      • queue (pull testcases from here)

• resume (with --sync_afl)

  • --input to - (-i-)
  • output to output_dir
  • creates multiple output_dir/angora* directories
    • must handle that: move output_dir/angora/queue/* into sync_dir/queue/

eclipser

• output stats

  • GUI incremental stdout (not much info)
  • files: .coverage

• parallel

  • probably push new cases into input dir and restart eclipser (Confirmation on sharing state between multiple instances of Eclipser SoftSec-KAIST/Eclipser#12)
  • it saves files as json(base64)

• dir structure

  • output_dir
    • sync_dir
      • queue (pull push testcases here)
    • the_fuzzer (--outputdir option)
      • crash (json encoded crashes)
      • crashes (decoded crashes)
      • testcase (json encoded testcases)

• resume

  • just run with the same output dir