Update to LibAFL 0.13.2 #5

domenukk · 2024-09-07T00:11:58Z

This is a quick and dirty update to LibAFL 0.13.2.
I have not tested this at all - some behaviour might have changed. Use at your own risk.
However, it does build, so it may be a good starting point.

I left one TODO in there in fuzzer.rs:

        fuzzer.process_execution(
            &mut state,
            &mut mgr,
            &input,
            &ExecuteInputResult::Corpus, // << Not sure what the result should be here.
            &collective_observer,
        )?;

ThomasTNO · 2024-09-07T09:03:50Z

Thanks for your efforts so far @domenukk, much appreciated!
In the coming days, I will try to find some time to continue the work.

ThomasTNO · 2024-09-07T09:58:32Z

Interestingly, despite being allowed to edit this pull request GitHub does not allow me to push.

Not from local machine

ERROR: Authentication error: Authentication required: You must have push access to verify locks
error: failed to push some refs to '[email protected]:domenukk/WuppieFuzz.git'

and not even from codespaces..

I'll dive into that

ThomasTNO · 2024-09-07T10:24:29Z

Interestingly, despite being allowed to edit this pull request GitHub does not allow me to push.

Not from local machine
ERROR: Authentication error: Authentication required: You must have push access to verify locks
error: failed to push some refs to '[email protected]:domenukk/WuppieFuzz.git'
and not even from codespaces..

I'll dive into that

Might have been related to non-existing submodules as we use relative urls for those. I fixed the urls to be correct in forks as well in #6

This reverts commit 8c39a2f.

ThomasTNO · 2024-09-07T10:32:24Z

Note to other git push --no-verify does the trick for now.

ThomasTNO · 2024-09-07T11:10:46Z

Two findings so far:

Corpus entries are not force added yet (this relates to your TODO)
During testing the fuzzing loop crashes

Error: Error in the fuzzing loop

Caused by:
    Invalid corpus: This testcase doesnot trigger trigger any edges. Check your instrumentation!

The typo will be fixed in AFLplusplus/LibAFL#2515.
TODO: check what goes wrong here.

Those two issues are probably related, TODO: check harness.

EDIT: 1. seems resolved now. For 2. we somehow reproducibly have no coverage at all for precisely the 4th input that is processed (tested on two different targets).

… good. Modify logging to reflect this behaviour

ThomasTNO · 2024-09-08T14:04:15Z

Played around a bit, including modifying the DummyCoverageClient to have a non-zero coverage map of length 1 or MAP_SIZE. Both without any success. Somehow the map is cleared, even when explicitly filling the map at the end of the harness. This happens during the calibration stage.

As a side note, it would be better to get rid of this DummyCoverageClient and fallback to EndpointCoverageClient when there is no instrumentation of the target (black box mode).

Still to check if the fuzzing also fails when we use e.g. the LCOV client. If that does work, I would suggest to refactor and ensure that we use the EndpointCoverageClient as a fallback in blackbox mode.

domenukk · 2024-09-08T15:51:11Z

Somehow the map is cleared, even when explicitly filling the map at the end of the harness. This happens during the calibration stage.

Is this in (new) LibAFL code?

ThomasTNO · 2024-09-08T16:29:55Z

Is this in (new) LibAFL code?

It seems like it happens in the LibAFL code. I am not sure if it is 'new' as the check on which it fails is introduced later. It could well be the case that this is also not working properly in the current version of WuppieFuzz that is based on LibAFL 0.11.2. That would require some additional testing.

In the meantime I tried a java target using code coverage coverage, same behaviour.

For testing purposes, I created the following example:

petstore.zip

Steps to reproduce:

docker compose up
cargo run fuzz --log-level debug --timeout 60 --coverage-format jacoco <path to petstore_openapi.yaml>

Edit: I just noticed that at least in the logging the coverage also hangs on 'unknown' in v1.0.0.

@grebnetiew, something to explore

ThomasTNO · 2024-09-08T17:19:31Z

Ok, maybe this does not happen in the LibAFL code.

Let's find out why the fetch_coverage function does not fill the map..

ThomasTNO · 2024-09-08T17:46:20Z

Ok, maybe this does not happen in the LibAFL code.

Let's find out why the fetch_coverage function does not fill the map..

This is resolved in #9 and also in this branch. So we're back at an issue in the LibAFL part, I guess.

domenukk · 2024-09-08T18:18:20Z

What's the buggy behavior? LibAFL emptying out observers before the feedback has been fully processed?

ThomasTNO · 2024-09-08T19:19:33Z

What's the buggy behavior? LibAFL emptying out observers before the feedback has been fully processed?

That seems to be the case indeed. I expect this goes wrong in the perform() function of the CalibrationStage.

To my surprise

        let map_first = observers[&self.map_observer_handle].as_ref();

is already a map of pure zeroes.

ThomasTNO · 2024-09-09T07:47:33Z

Ok, some progress.. This behaviour is not new. On the main branch I spot similar behaviour, where bitmap_size = 0. It does not crash there, because the bitmap_size is not checked to be >=1 in LibAFL 0.11.2..

Still I have no clue as to why the map is empty in this part

        // If weighted scheduler or powerscheduler is used, update it

@domenukk, any thoughts?

domenukk · 2024-09-09T12:05:17Z

I mean, the only time LibAFL resets the observer map is in pre_exec. You could add a log there?
Other option is that you have two copies of the map for some reason, and you point to the wrong one?

domenukk · 2024-09-09T12:12:47Z

WuppieFuzz/src/fuzzer.rs

Line 264 in 7b47203

collective_observer.clone(),

Working theory: This will clone the inner OwnedSlice to be a new slice (see https://github.com/AFLplusplus/LibAFL/blob/25624d8eecdd3eae85c94a43669add3d3bae0269/libafl_bolts/src/ownedref.rs#L760 )

domenukk · 2024-09-09T12:34:12Z

@ThomasTNO can you try if ab53217 helps at all?

(empty coverage maps are disallowed)

grebnetiew

I think it looks very good! Very happy about the new stopping mechanism. I made some small changes:

refactored some of the new unwraps since they could be statically guaranteed
add a multimap observer to let the scheduler see both endpoint coverage and line coverage, and turn the Dummy client back to doing nothing (but now with a size-1 map)

I have't tested this yet on the usual targets.

grebnetiew · 2024-09-12T15:07:30Z

Oh actually, though it cargo checks, it doesn't build!

error[E0080]: evaluation of `libafl::schedulers::MinimizerScheduler::<CS, F, M, O>::new::TrackingEnabledCheck::<libafl::observers::MultiMapObserver<'_, u8, false>>::TRACKING_ENABLED` failed
   --> /home/erieke/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.13.2/src/schedulers/minimizer.rs:385:9
    |
385 |         require_index_tracking!("MinimizerScheduler", O);
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the evaluated program panicked at '
    |
    = note: index tracking is required by MinimizerScheduler
    = note: see the documentation of CanTrack for details
    |
    = hint: call `.track_indices()` on the map observer passed to MinimizerScheduler at the point where it is defined
    |
    | ', /home/erieke/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.13.2/src/schedulers/minimizer.rs:385:9
    |
    = note: this error originates in the macro `$crate::panic::panic_2021` which comes from the expansion of the macro `require_index_tracking` (in Nightly builds, run with -Z macro-backtrace for more info)

That's a new one for me :) Not sure I just broke it, either. (edit: I did! fortunately the hint was accurate)

tokatoka · 2024-09-12T15:19:17Z

- add a multimap observer to let the scheduler see both endpoint coverage and line coverage, and turn the Dummy client back to doing nothing (but now with a size-1 map)

that error comes from this. can you call track_indices() on your combined observer?

ThomasTNO · 2024-09-12T15:41:53Z

TODO

check licenses and whether they require a mention (zlib and bsd2?)
test whether fuzzer runs properly
add changelog entry add a multimap observer to let the scheduler see both endpoint coverage and line coverage, and turn the Dummy client back to doing nothing (but now with a size-1 map)

ThomasTNO · 2024-09-12T21:55:35Z

@grebnetiew, just a quick test run:
With code coverage instrumentation (jacoco)

Error: Error in the fuzzing loop

Caused by:
    Key: `MapObserver not found` - not found

Black box mode

Error: Error in the fuzzing loop

Caused by:
    Invalid corpus: This testcase doesnot trigger trigger any edges. Check your instrumentation!

The latter is fixed in bd676f7

ThomasTNO · 2024-09-12T23:11:59Z

src/fuzzer.rs


    // A fuzzer with feedbacks and a corpus scheduler
    let mut fuzzer = StdFuzzer::new(scheduler, collective_feedback, objective);

-    let collective_observer = tuple_list!(endpoint_observer, coverage_observer, time_observer);
+    let collective_observer = tuple_list!(


I am not sure what you exactly tried to achieve. But The cause of the missing key lies here, I believe.

The combined_map_observer is not present in this list, which is later used to get the all_maps entry from.

In case this was a question for me, I didn't change this :)

Nope, this is a question for @grebnetiew :)

I tried to leave this the same, apart from a rename.

As far as I understand (but I might not :)), the MultiMapObserver internally points to the same map as the other observer, and I would expect all_maps to eventually get the same maps from the tuple-list as from the multimap.

I'll check my assumptions shortly ;)

I have attempted to fix the problem by (inlining the creation of the multimap observer because naming its type is prohibitively time consuming, and) using the multimap observer also in the tuple list. Does this fix the missing key problem you speak of @ThomasTNO ?

This is the behaviour I observe

thread 'main' panicked at /home/thomas/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.13.2/src/stages/calibrate.rs:156:34: Could not find entry matching Handle { name: "code_coverage", type: "libafl::observers::map::ExplicitTracking<libafl::observers::map::StdMapObserver<u8, false>, true, true>" } stack backtrace: 0: rust_begin_unwind at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:652:5 1: core::panicking::panic_fmt at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/panicking.rs:72:14 2: <libafl_bolts::tuples::RefIndexable<RM,M> as core::ops::index::Index<&libafl_bolts::tuples::Handle<T>>>::index at /home/thomas/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl_bolts-0.13.2/src/tuples.rs:638:13 3: <libafl::stages::calibrate::CalibrationStage<C,E,O,OT> as libafl::stages::Stage<E,EM,Z>>::perform at /home/thomas/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.13.2/src/stages/calibrate.rs:156:34 4: libafl::stages::Stage::perform_restartable at /home/thomas/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.13.2/src/stages/mod.rs:127:13 5: <(Head,Tail) as libafl::stages::StagesTuple<E,EM,<Head as libafl::state::UsesState>::State,Z>>::perform_all at /home/thomas/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.13.2/src/stages/mod.rs:214:17 6: <libafl::fuzzer::StdFuzzer<CS,F,OF> as libafl::fuzzer::Fuzzer<E,EM,ST>>::fuzz_one at /home/thomas/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.13.2/src/fuzzer/mod.rs:804:9 7: wuppiefuzz::fuzzer::fuzz at ./src/fuzzer.rs:334:9 8: wuppiefuzz::main at ./src/main.rs:87:34 9: core::ops::function::FnOnce::call_once at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/ops/function.rs:250:5

We are now missing a different map ;)

Possibly we are misinterpretating the working of the MultiMapOberserver..

a94cb25 seems to make it all operational.

Currently JaCoCo coverage seems to work fine. Blackbox mode is operational. LCOV coverage seems to fail (tested on Python). I suspect an existing issue though. Possibly the coverage agent is too fast / slow compared to the coverage gathering resulting in an empty map.

LCOV coverage seems to fail (tested on Python)

This was a lie, I accidentally used a pre-release, outdated version of WuppieFuzz-Python. With the latest version it does work. Will post a reproducible setup soon.

ThomasTNO · 2024-09-16T18:18:16Z

LCOV testing setup:

swagger_server.zip

docker compose up
cargo run fuzz --coverage-format lcov <path to openapi.yaml>

ThomasTNO · 2024-09-17T08:19:58Z

Thanks for the help @domenukk, v1.1.0 has just been released including the update to LibAFL 0.13.2. Let's stay up to date from now on :)

domenukk · 2024-09-17T12:57:11Z

Happy to help, keep up the good work! :)

Initial Update to LibAFL 0.13.2

f180337

domenukk changed the title ~~Initial Update to LibAFL 0.13.2~~ Initial Update to LibAFL 0.13.2 (please test & fix :) ) Sep 7, 2024

Missing word in docs

6661d9b

ThomasTNO added 3 commits September 7, 2024 12:26

Merge branch 'main' into libafl0.13.2

399e810

No relative urls

8c39a2f

Revert "No relative urls"

ac37a5e

This reverts commit 8c39a2f.

ThomasTNO mentioned this pull request Sep 7, 2024

Fix typo in error message AFLplusplus/LibAFL#2515

Merged

ThomasTNO added 3 commits September 7, 2024 20:04

Corpus seems loaded before we start the initial corpus loop, which is…

302f6ef

… good. Modify logging to reflect this behaviour

Clippy

c4e3f63

Use the new request stop functionality?

3992713

I believe this should be a not..

53a10ad

Name the observers correctly to aid in debugging

7b47203

domenukk added 2 commits September 9, 2024 14:23

Remove duplicate NopState

6a3e8e7

Maybe fix empty observer?

ab53217

grebnetiew added 3 commits September 12, 2024 15:07

Adds a multimap observer combining endpoint and line coverage

ade7b1a

Turn dummy client back to doing nothing, but with nonempty map

abdaa73

(empty coverage maps are disallowed)

Refactor some unwraps so it's clear it can't panic

99ff8bf

ThomasTNO mentioned this pull request Sep 12, 2024

Implement custom executor #12

Open

grebnetiew reviewed Sep 12, 2024

View reviewed changes

ThomasTNO and others added 2 commits September 12, 2024 17:11

Merge branch 'main' into libafl0.13.2

410e91e

Track indices on the new multimap observer

188287a

Add some changelog entries

0f2bb9c

ThomasTNO mentioned this pull request Sep 12, 2024

Update dependencies #4

Merged

ThomasTNO added 4 commits September 13, 2024 00:26

consistent naming of coverage clients

e797c89

HOping to clarify and align coverage naming

cf78fb8

formatting

814dda4

Fixes empty map in blackbox mode

bd676f7

ThomasTNO reviewed Sep 12, 2024

View reviewed changes

Third-party licenses added

540b075

domenukk changed the title ~~Initial Update to LibAFL 0.13.2 (please test & fix :) )~~ Update to LibAFL 0.13.2 Sep 16, 2024

grebnetiew and others added 3 commits September 16, 2024 14:15

Inelegantly used multimapobserver also in tuple list

ce7e049

Add missing keys

a94cb25

Add changelog entry

0c1465b

ThomasTNO mentioned this pull request Sep 16, 2024

Implement Go code coverage #11

Open

ThomasTNO mentioned this pull request Sep 16, 2024

Collect example applications for testing purposes #13

Open

ThomasTNO merged commit bb85a80 into TNO-S3:main Sep 17, 2024
5 checks passed

domenukk mentioned this pull request Sep 17, 2024

Update LibAFL to latest #3

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to LibAFL 0.13.2 #5

Update to LibAFL 0.13.2 #5

domenukk commented Sep 7, 2024 •

edited

Loading

ThomasTNO commented Sep 7, 2024

ThomasTNO commented Sep 7, 2024

ThomasTNO commented Sep 7, 2024

ThomasTNO commented Sep 7, 2024

ThomasTNO commented Sep 7, 2024 •

edited

Loading

ThomasTNO commented Sep 8, 2024

domenukk commented Sep 8, 2024

ThomasTNO commented Sep 8, 2024 •

edited

Loading

ThomasTNO commented Sep 8, 2024

ThomasTNO commented Sep 8, 2024

domenukk commented Sep 8, 2024 •

edited

Loading

ThomasTNO commented Sep 8, 2024 •

edited

Loading

ThomasTNO commented Sep 9, 2024 •

edited

Loading

domenukk commented Sep 9, 2024

domenukk commented Sep 9, 2024

domenukk commented Sep 9, 2024

grebnetiew left a comment

grebnetiew commented Sep 12, 2024 •

edited

Loading

tokatoka commented Sep 12, 2024

ThomasTNO commented Sep 12, 2024 •

edited

Loading

ThomasTNO commented Sep 12, 2024 •

edited

Loading

ThomasTNO Sep 12, 2024

domenukk Sep 16, 2024

ThomasTNO Sep 16, 2024

grebnetiew Sep 16, 2024

grebnetiew Sep 16, 2024

ThomasTNO Sep 16, 2024

ThomasTNO Sep 16, 2024

ThomasTNO Sep 16, 2024

ThomasTNO Sep 16, 2024

ThomasTNO commented Sep 16, 2024

ThomasTNO commented Sep 17, 2024

domenukk commented Sep 17, 2024

Update to LibAFL 0.13.2 #5

Update to LibAFL 0.13.2 #5

Conversation

domenukk commented Sep 7, 2024 • edited Loading

ThomasTNO commented Sep 7, 2024

ThomasTNO commented Sep 7, 2024

ThomasTNO commented Sep 7, 2024

ThomasTNO commented Sep 7, 2024

ThomasTNO commented Sep 7, 2024 • edited Loading

ThomasTNO commented Sep 8, 2024

domenukk commented Sep 8, 2024

ThomasTNO commented Sep 8, 2024 • edited Loading

ThomasTNO commented Sep 8, 2024

ThomasTNO commented Sep 8, 2024

domenukk commented Sep 8, 2024 • edited Loading

ThomasTNO commented Sep 8, 2024 • edited Loading

ThomasTNO commented Sep 9, 2024 • edited Loading

domenukk commented Sep 9, 2024

domenukk commented Sep 9, 2024

domenukk commented Sep 9, 2024

grebnetiew left a comment

Choose a reason for hiding this comment

grebnetiew commented Sep 12, 2024 • edited Loading

tokatoka commented Sep 12, 2024

ThomasTNO commented Sep 12, 2024 • edited Loading

ThomasTNO commented Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThomasTNO commented Sep 16, 2024

ThomasTNO commented Sep 17, 2024

domenukk commented Sep 17, 2024

domenukk commented Sep 7, 2024 •

edited

Loading

ThomasTNO commented Sep 7, 2024 •

edited

Loading

ThomasTNO commented Sep 8, 2024 •

edited

Loading

domenukk commented Sep 8, 2024 •

edited

Loading

ThomasTNO commented Sep 8, 2024 •

edited

Loading

ThomasTNO commented Sep 9, 2024 •

edited

Loading

grebnetiew commented Sep 12, 2024 •

edited

Loading

ThomasTNO commented Sep 12, 2024 •

edited

Loading

ThomasTNO commented Sep 12, 2024 •

edited

Loading