Fix growing spheres

[kmariuszk]

No description provided.

[kmariuszk]

@pat-alt @VincentPikand @RaunoArike do you guys have any better idea on how to implement update!() for growing spheres?

[kmariuszk]

I thought about dispatching on generate_perturbations but I see some problems with the compatibility of the current update method and the one we need for growing spheres. Namely, in a single iteration of growing spheres we create multiple counterfactual candidates but the current implementation of update supports generators creating only one counterfactual at a time.

Also, even if we find a solution to that problem, the original problem of switching between different 'phases' of growing spheres is still there.

[pat-alt]

Hi @kmariuszk, sorry I'm only now getting to this, been a very hectic week. Could a set up like in DiCE be applicable where multiple counterfactuals are produced at once?

[kmariuszk]

@pat-alt @VincentPikand @RaunoArike would really appreciate if you could give this PR a look and tell me what you think about the direction of changes within growing spheres. I'm very skeptical about adding this new fields to the generator itself to keep the state of it but couldn't find any better solution yet. Thanks!

[izagorac]

Hi @kmariuszk, what is the current status of this PR? Could I possibly help you with anything? I'm currently doing my thesis with @pat-alt and I want to use the GrowingSpheresGenerator for my research

[kmariuszk]

Hi @izagorac, I'll get back to you in a couple of hours

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 25.98%. Comparing base (1504e0b) to head (0ff567e).
Report is 481 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #396       +/-   ##
===========================================
- Coverage   77.57%   25.98%   -51.59%     
===========================================
  Files          74       73        -1     
  Lines        1565     1474       -91     
===========================================
- Hits         1214      383      -831     
- Misses        351     1091      +740     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

	ce.search[:iteration_count] += 1
	return ce.search[:iteration_count] += 1

[github-actions]

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

	counterfactual = find_counterfactual(
	model, target, counterfactual_data, counterfactual_candidates
	ce,
	counterfactual_candidates
	)
	counterfactual = find_counterfactual(ce, counterfactual_candidates)

[kmariuszk]

Hey @izagorac, @pat-alt, I cleaned up the PR and here's a quick summary:

✅ Growing spheres generation has been updated, so now it is actually using the proper convergence logic (i.e., it actually uses the decision_threshold provided by the user.
❌ Growing spheres is still not properly dispatched on generate_perturbations() (currently, it is 'dispatched' on update()). My main concern which blocked me from implementing that is addressed in the next point.
❌ While generating counterfactuals using growing spheres the path cannot be easily updated using current logic. Currently, the update happens in the following line:

CounterfactualExplanations.jl/src/counterfactuals/search.jl

Line 22 in 1504e0b

ce.search[:path] = [ce.search[:path]..., ce.s′]

the problematic part about that is that GrowingSpheresGenerator is generating multiple points at the same time. I noticed @pat-alt suggested to use similar methodology as in DiCE but unfortunately, I didn't look into it. There might be a possible fix for that.
❔ I'm not sure if currently GrowingSpheresGenerator is capable of generating multiple counterfactuals at once. To be fair, I'm not even sure if the growing spheres algorithm is ever able to generate more counterfactuals than just one.

Currently, I don't have anymore time to look into the mentioned issues. I should have some time I could spend on it on Saturday, @pat-alt do you have time then? We could have a pair-programming session which could speed-up the whole process. Otherwise, @izagorac you can try to fix the mentioned issues by yourself.

[izagorac]

Thank you for the elaborate summary @kmariuszk !

First of all, you don't have to worry about fixing this issue ASAP because I still have loads to work on for my thesis and if these issues take a while to fix that is no problem.
Second, I see you mentioned the amount of counterfactuals that the generator can generate at once. Do you refer to the number of counterfactuals that it will generate per factual, which can be specified with the num_counterfactuals parameter in the generate_counterfactual function, or the amount of factuals that counterfactuals can be generated for (hopefully this sentence still makes sense). If you mean the first one, then I don't see any issue for my work. I will namely only be generating one counterfactual per factual.

Anyways, thanks for your time and hopefully we can solve these issues together!

[pat-alt]

Thanks @kmariuszk for looking into this. I'm at AAAI in Canada right now and we have a pretty packed schedule, even on the weekend. I could probably skip a session on Saturday, but then I'm still in a very different time zone 😅 perhaps easier when I'm back in early March.

In the meantime, might it be worth already merging what we have here? I'm puzzled by the errors on 1.7 and 1.8, but that should be fixable. @izagorac would that fix your current issue? If not, could you post the error your getting here?

[izagorac]

@pat-alt @kmariuszk I'm not sure whether the updates fix my current issues but I can always use the updated generator if it gets merged. Nonetheless, I'll post the exact error I get here:

From looking at the error message, it seems to be a very generic error. However, if I run the exact same code but with the FeatureTweakGenerator, I have no problems at all. Hopefully, this stacktrace gives you some additional information @kmariuszk.

[pat-alt]

@kmariuszk it's been a minute 😅 how are you doing? Do you think you'll ever revisit this? Otherwise I'll probably remove GrowingSpheres from the public API and docs

[kmariuszk]

Hey @pat-alt, I'm afraid I won't have time to revisit this, so feel free to remove GrowingSpheres.