From 1814a12d005ccc2c0ed32787e03390af32157d69 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Wed, 23 Oct 2024 11:02:04 +0000 Subject: [PATCH] build based on 5ff1e81 --- dev/.documenter-siteinfo.json | 2 +- dev/CHANGELOG/index.html | 2 +- dev/assets/resources/index.html | 2 +- dev/contribute/index.html | 2 +- dev/contribute/performance/index.html | 2 +- dev/explanation/architecture/index.html | 2 +- dev/explanation/categorical/index.html | 2 +- dev/explanation/evaluation/faithfulness/index.html | 2 +- dev/explanation/evaluation/overview/index.html | 2 +- dev/explanation/generators/clap_roar/index.html | 2 +- dev/explanation/generators/clue/index.html | 2 +- dev/explanation/generators/dice/index.html | 2 +- dev/explanation/generators/feature_tweak/index.html | 2 +- dev/explanation/generators/generic/index.html | 2 +- dev/explanation/generators/gravitational/index.html | 2 +- dev/explanation/generators/greedy/index.html | 2 +- dev/explanation/generators/growing_spheres/index.html | 2 +- dev/explanation/generators/mint/index.html | 2 +- dev/explanation/generators/overview/index.html | 2 +- dev/explanation/generators/probe/index.html | 2 +- dev/explanation/generators/revise/index.html | 2 +- dev/explanation/generators/tcrex/index.html | 2 +- dev/explanation/index.html | 2 +- dev/explanation/optimisers/jsma/index.html | 2 +- dev/explanation/optimisers/overview/index.html | 2 +- dev/extensions/index.html | 2 +- dev/extensions/laplace_redux/index.html | 2 +- dev/extensions/neurotree/index.html | 2 +- dev/how_to_guides/custom_generators/index.html | 2 +- dev/how_to_guides/custom_models/index.html | 2 +- dev/how_to_guides/index.html | 2 +- dev/index.html | 2 +- dev/reference/index.html | 2 +- dev/release-notes/index.html | 2 +- dev/tutorials/benchmarking/index.html | 2 +- dev/tutorials/convergence/index.html | 2 +- dev/tutorials/data_catalogue/index.html | 2 +- dev/tutorials/data_preprocessing/index.html | 2 +- dev/tutorials/evaluation/index.html | 2 +- dev/tutorials/generators/index.html | 2 +- dev/tutorials/index.html | 2 +- dev/tutorials/model_catalogue/index.html | 2 +- dev/tutorials/models/index.html | 2 +- dev/tutorials/parallelization/index.html | 2 +- dev/tutorials/simple_example/index.html | 2 +- dev/tutorials/whistle_stop/index.html | 2 +- 46 files changed, 46 insertions(+), 46 deletions(-) diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index fd7501098..e4e56a062 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.11.1","generation_timestamp":"2024-10-23T10:59:36","documenter_version":"1.7.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.11.1","generation_timestamp":"2024-10-23T11:01:41","documenter_version":"1.7.0"}} \ No newline at end of file diff --git a/dev/CHANGELOG/index.html b/dev/CHANGELOG/index.html index 7ab219bbe..e62313bcb 100644 --- a/dev/CHANGELOG/index.html +++ b/dev/CHANGELOG/index.html @@ -1,2 +1,2 @@ -Changelog · CounterfactualExplanations.jl
GitHub

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note: We try to adhere to these practices as of version [v1.1.1].

Version [1.3.4] - 2024-10-22

Changed

  • Fixed a bug in the find_potential_neighbours method.

Version [1.3.3] - 2024-09-30

Changed

  • Fixed a remaining bug in NeuroTreeExt extensions. [#475]

Version [1.3.2] - 2024-09-24

Added

  • Added support for using a random forest as a surrogate model for the T-CREx generator. [#483]

Changed

  • Improved the T-CREx documentation further by bringing example even closer to the example in the paper. [#483]
  • Include citation linking to ICML paper in T-CREx documentation and docstrings. [#480]

Version [1.3.1] - 2024-09-24

Changed

  • Fixed a remaining bug in NeuroTreeExt extensions. [#475]

Version [1.3.0] - 2024-09-16

Changed

  • Fixed bug in NeuroTreeExt extensions. [#475]

Added

  • Added basic support for the T-CREx counterfactual generator. [#473]
  • Added docstrings for package extensions to documentation. [#475]

Version [1.2.0] - 2024-09-10

Added

  • Added documentation for generating counterfactuals consistent with the MINT framework. [#467]
  • Added tests for new evaluation metrics and JEM extension. [#471]
  • Added support for gradient-based causal algorithm-recourse (MNIT) as described in Karimi et al. (2020). This incorporates an input encoder that is based on a Structural Causal Model [#457]
  • Added out-of-the-box support for training joint energy models (JEM). [#454]
  • Added new evaluation metric to measure faithfulness of counterfactual explanations as in Altmeyer et al. (2024). [#454]
  • A tutorial in the documentation ("Explanation" section) explaining the faithfulness metric in detail. [#454]
  • Added support for an energy constraint as in Altmeyer et al. (2024). This is the first step towards adding functionality for ECCCo. [#387]

Changed

  • The fitresult field of Model now takes a concrete Fitresult type, for which some basic methods have been defined. This mutable struct has a field called other that accepts a dictionary Dict that can be filled with additional objects. [#454]
  • Regenerated pre-trained model artifacts. [#454]
  • Updated the tutorial on "Handling Data". [#454]

Removed

  • Removed bug in find_potential_neighbours method. [#454]

Version [1.1.6] - 2024-05-19

Removed

  • Removed the call to the Iris function in the test suite because of HTTPs issues. [#452]
  • Removed the mlj_models_catalogue because it served no obvious purpose. In the future, we may instead add meta information to the all_models_catalogue. [#444]

Added

  • New general Model struct that wraps empty concrete types. This adds a more general interface that is still flexible enough by simply using multiple dispatch on the empty concrete types. [#444]
  • A new incompatible(::AbstractGenerator, ::AbstractCounterfactualExplanation) function has been added to avoid running a counterfactual search if the generator is incompatible with any other specification (e.g. the model). [#444]

Changed

  • No longer exporting many of the deprecated functions. [#452]
  • Updated pre-trained model artifacts. [#444]
  • Some function signatures have been deprecated, e.g. NeuroTreeModel to NeuroTree, LaplaceReduxModel to LaplaceNN. [#444]
  • Support for DecisionTree.jl models and the FeatureTweakGenerator have been moved to an extension (DecisionTreeExt). [#444]
  • Updates to NeuroTreeModels extensions to incorporate breaking changes to package. [#444]
  • No longer running alloc test on Windows. [#441]
  • Slight change to doctests. [#447]

Version [v1.1.5] - 2024-04-30

Added

  • Unit tests: adds a simple performance benchmark to test that for a small problem, generating a counterfactual using the generic generator takes at most 4700 allocations. Only run on julia v1.10 and higher. [#436]

Changed

  • The find_potential_neighbours is now only triggered if one of the penalties of the generator requires access to samples from the target domain. This improves scalability because calling the function can be computationally costly (forward-pass). [#436]
  • The target variable encodings are now handled more efficiently. Previously certain tasks were repeated, which was not necessary. [#436]

Removed

  • Removed the assertion checking that the model ever predicts the target value. While this assertion is useful, it is not essential. For large enough models and datasets, this forward pass can be very costly. [#436]
  • Removed redundant distance_from_targets function. [#436]

Version [v1.1.4] - 2024-04-25

Changed

  • Refactors the encodings and decodings such that it is now more streamlined. Instead of conditional statements, encodings are now dispatched on the type of a new unifying data.input_encoder field. [#432]
  • Refactors the check for redundancy. This is now based on the convergence type and done right before the counterfactual search begins, if not redundant. [#432]

Added

  • Added additional unit tests. [#437]

Version [v1.1.3] - 2024-04-17

Added

  • Adds a section on Convergence to the documentation, Changelog.jl functionality and a few doc tests. [#429]

Changed

  • Changes style of taking gradients for the counterfactual search from implicit to explicit. [#430]
  • Removed all implicit imports. [#430]

Removed

  • Removed CUDA.jl dependency, because redundant. [#430]
  • Removed Parameters.jl dependency, because redundant. [#430]

Version [v1.1.2] - 2024-04-16

Changed

  • Replaces the GIF in the README and introduction of docs for a static image.

Version [v1.1.1] - 2024-04-15

Added

  • Added tests for LaplaceRedux extension. Bumped upper compat bound for LaplaceRedux.jl. [#428]

<!– Links generated by Changelog.jl –>

[#428]: https://github.com/juliatrustworthyai/CounterfactualExplanations.jl/issues/428 [#429]: https://github.com/juliatrustworthyai/CounterfactualExplanations.jl/issues/429

+Changelog · CounterfactualExplanations.jl
GitHub

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note: We try to adhere to these practices as of version [v1.1.1].

Version [1.3.4] - 2024-10-22

Changed

  • Fixed a bug in the find_potential_neighbours method.

Version [1.3.3] - 2024-09-30

Changed

  • Fixed a remaining bug in NeuroTreeExt extensions. [#475]

Version [1.3.2] - 2024-09-24

Added

  • Added support for using a random forest as a surrogate model for the T-CREx generator. [#483]

Changed

  • Improved the T-CREx documentation further by bringing example even closer to the example in the paper. [#483]
  • Include citation linking to ICML paper in T-CREx documentation and docstrings. [#480]

Version [1.3.1] - 2024-09-24

Changed

  • Fixed a remaining bug in NeuroTreeExt extensions. [#475]

Version [1.3.0] - 2024-09-16

Changed

  • Fixed bug in NeuroTreeExt extensions. [#475]

Added

  • Added basic support for the T-CREx counterfactual generator. [#473]
  • Added docstrings for package extensions to documentation. [#475]

Version [1.2.0] - 2024-09-10

Added

  • Added documentation for generating counterfactuals consistent with the MINT framework. [#467]
  • Added tests for new evaluation metrics and JEM extension. [#471]
  • Added support for gradient-based causal algorithm-recourse (MNIT) as described in Karimi et al. (2020). This incorporates an input encoder that is based on a Structural Causal Model [#457]
  • Added out-of-the-box support for training joint energy models (JEM). [#454]
  • Added new evaluation metric to measure faithfulness of counterfactual explanations as in Altmeyer et al. (2024). [#454]
  • A tutorial in the documentation ("Explanation" section) explaining the faithfulness metric in detail. [#454]
  • Added support for an energy constraint as in Altmeyer et al. (2024). This is the first step towards adding functionality for ECCCo. [#387]

Changed

  • The fitresult field of Model now takes a concrete Fitresult type, for which some basic methods have been defined. This mutable struct has a field called other that accepts a dictionary Dict that can be filled with additional objects. [#454]
  • Regenerated pre-trained model artifacts. [#454]
  • Updated the tutorial on "Handling Data". [#454]

Removed

  • Removed bug in find_potential_neighbours method. [#454]

Version [1.1.6] - 2024-05-19

Removed

  • Removed the call to the Iris function in the test suite because of HTTPs issues. [#452]
  • Removed the mlj_models_catalogue because it served no obvious purpose. In the future, we may instead add meta information to the all_models_catalogue. [#444]

Added

  • New general Model struct that wraps empty concrete types. This adds a more general interface that is still flexible enough by simply using multiple dispatch on the empty concrete types. [#444]
  • A new incompatible(::AbstractGenerator, ::AbstractCounterfactualExplanation) function has been added to avoid running a counterfactual search if the generator is incompatible with any other specification (e.g. the model). [#444]

Changed

  • No longer exporting many of the deprecated functions. [#452]
  • Updated pre-trained model artifacts. [#444]
  • Some function signatures have been deprecated, e.g. NeuroTreeModel to NeuroTree, LaplaceReduxModel to LaplaceNN. [#444]
  • Support for DecisionTree.jl models and the FeatureTweakGenerator have been moved to an extension (DecisionTreeExt). [#444]
  • Updates to NeuroTreeModels extensions to incorporate breaking changes to package. [#444]
  • No longer running alloc test on Windows. [#441]
  • Slight change to doctests. [#447]

Version [v1.1.5] - 2024-04-30

Added

  • Unit tests: adds a simple performance benchmark to test that for a small problem, generating a counterfactual using the generic generator takes at most 4700 allocations. Only run on julia v1.10 and higher. [#436]

Changed

  • The find_potential_neighbours is now only triggered if one of the penalties of the generator requires access to samples from the target domain. This improves scalability because calling the function can be computationally costly (forward-pass). [#436]
  • The target variable encodings are now handled more efficiently. Previously certain tasks were repeated, which was not necessary. [#436]

Removed

  • Removed the assertion checking that the model ever predicts the target value. While this assertion is useful, it is not essential. For large enough models and datasets, this forward pass can be very costly. [#436]
  • Removed redundant distance_from_targets function. [#436]

Version [v1.1.4] - 2024-04-25

Changed

  • Refactors the encodings and decodings such that it is now more streamlined. Instead of conditional statements, encodings are now dispatched on the type of a new unifying data.input_encoder field. [#432]
  • Refactors the check for redundancy. This is now based on the convergence type and done right before the counterfactual search begins, if not redundant. [#432]

Added

  • Added additional unit tests. [#437]

Version [v1.1.3] - 2024-04-17

Added

  • Adds a section on Convergence to the documentation, Changelog.jl functionality and a few doc tests. [#429]

Changed

  • Changes style of taking gradients for the counterfactual search from implicit to explicit. [#430]
  • Removed all implicit imports. [#430]

Removed

  • Removed CUDA.jl dependency, because redundant. [#430]
  • Removed Parameters.jl dependency, because redundant. [#430]

Version [v1.1.2] - 2024-04-16

Changed

  • Replaces the GIF in the README and introduction of docs for a static image.

Version [v1.1.1] - 2024-04-15

Added

  • Added tests for LaplaceRedux extension. Bumped upper compat bound for LaplaceRedux.jl. [#428]

<!– Links generated by Changelog.jl –>

[#428]: https://github.com/juliatrustworthyai/CounterfactualExplanations.jl/issues/428 [#429]: https://github.com/juliatrustworthyai/CounterfactualExplanations.jl/issues/429

diff --git a/dev/assets/resources/index.html b/dev/assets/resources/index.html index fed9b29fa..af7b904d8 100644 --- a/dev/assets/resources/index.html +++ b/dev/assets/resources/index.html @@ -1,2 +1,2 @@ -📚 Additional Resources · CounterfactualExplanations.jl
GitHub
+📚 Additional Resources · CounterfactualExplanations.jl
GitHub
diff --git a/dev/contribute/index.html b/dev/contribute/index.html index bfdbee6e4..d2c1a0d70 100644 --- a/dev/contribute/index.html +++ b/dev/contribute/index.html @@ -1,2 +1,2 @@ -🛠 Contribute · CounterfactualExplanations.jl
GitHub

🛠 Contribute

Contributions of any kind are very much welcome! Take a look at the issue to see what things we are currently working on. If you have an idea for a new feature or want to report a bug, please open a new issue.

Development

If your looking to contribute code, it may be helpful to check out the Explanation section of the docs.

Testing

Please always make sure to add tests for any new features or changes.

Documentation

If you add new features or change existing ones, please make sure to update the documentation accordingly. The documentation is written in Documenter.jl and is located in the docs/src folder.

Log Changes

As of version 1.1.1, we have tried to be more stringent about logging changes. Please make sure to add a note to the CHANGELOG.md file for any changes you make. It is sufficient to add a note under the Unreleased section.

General Pointers

There are also some general pointers for people looking to contribute to any of our Taija packages here.

Please follow the SciML ColPrac guide.

+🛠 Contribute · CounterfactualExplanations.jl
GitHub

🛠 Contribute

Contributions of any kind are very much welcome! Take a look at the issue to see what things we are currently working on. If you have an idea for a new feature or want to report a bug, please open a new issue.

Development

If your looking to contribute code, it may be helpful to check out the Explanation section of the docs.

Testing

Please always make sure to add tests for any new features or changes.

Documentation

If you add new features or change existing ones, please make sure to update the documentation accordingly. The documentation is written in Documenter.jl and is located in the docs/src folder.

Log Changes

As of version 1.1.1, we have tried to be more stringent about logging changes. Please make sure to add a note to the CHANGELOG.md file for any changes you make. It is sufficient to add a note under the Unreleased section.

General Pointers

There are also some general pointers for people looking to contribute to any of our Taija packages here.

Please follow the SciML ColPrac guide.

diff --git a/dev/contribute/performance/index.html b/dev/contribute/performance/index.html index 60ec619d6..9ca5dec67 100644 --- a/dev/contribute/performance/index.html +++ b/dev/contribute/performance/index.html @@ -12,4 +12,4 @@ # Search: generator = GenericGenerator() ce = generate_counterfactual(x, target, counterfactual_data, M, generator)
data_large = TaijaData.load_linearly_separable(100000)
-counterfactual_data_large = DataPreprocessing.CounterfactualData(data_large...)
@time generate_counterfactual(x, target, counterfactual_data, M, generator)
@time generate_counterfactual(x, target, counterfactual_data_large, M, generator)
+counterfactual_data_large = DataPreprocessing.CounterfactualData(data_large...)
@time generate_counterfactual(x, target, counterfactual_data, M, generator)
@time generate_counterfactual(x, target, counterfactual_data_large, M, generator)
diff --git a/dev/explanation/architecture/index.html b/dev/explanation/architecture/index.html index fbe93214a..958ba13ba 100644 --- a/dev/explanation/architecture/index.html +++ b/dev/explanation/architecture/index.html @@ -1,2 +1,2 @@ -Package Architecture · CounterfactualExplanations.jl
GitHub

Package Architecture

The diagram below provides an overview of the package architecture. It is built around two core modules that are designed to be as extensible as possible through dispatch: 1) Models is concerned with making any arbitrary model compatible with the package; 2) Generators is used to implement arbitrary counterfactual search algorithms.[1]

The core function of the package, generate_counterfactual, uses an instance of type AbstractModel produced by the Models module and an instance of type AbstractGenerator produced by the Generators module.

Metapackages from the Taija ecosystem provide additional functionality such as datasets, language interoperability, parallelization, and plotting. The CounterfactualExplanations package is designed to be used in conjunction with these metapackages, but can also be used as a standalone package.

[1] We have made an effort to keep the code base a flexible and extensible as possible, but cannot guarantee at this point that any counterfactual generator can be implemented without further adaptation.

+Package Architecture · CounterfactualExplanations.jl
GitHub

Package Architecture

The diagram below provides an overview of the package architecture. It is built around two core modules that are designed to be as extensible as possible through dispatch: 1) Models is concerned with making any arbitrary model compatible with the package; 2) Generators is used to implement arbitrary counterfactual search algorithms.[1]

The core function of the package, generate_counterfactual, uses an instance of type AbstractModel produced by the Models module and an instance of type AbstractGenerator produced by the Generators module.

Metapackages from the Taija ecosystem provide additional functionality such as datasets, language interoperability, parallelization, and plotting. The CounterfactualExplanations package is designed to be used in conjunction with these metapackages, but can also be used as a standalone package.

[1] We have made an effort to keep the code base a flexible and extensible as possible, but cannot guarantee at this point that any counterfactual generator can be implemented without further adaptation.

diff --git a/dev/explanation/categorical/index.html b/dev/explanation/categorical/index.html index 5e9956a48..1a0b3983a 100644 --- a/dev/explanation/categorical/index.html +++ b/dev/explanation/categorical/index.html @@ -117,4 +117,4 @@ 0.0 0.0 1.0 - 1.85 + 1.85 diff --git a/dev/explanation/evaluation/faithfulness/index.html b/dev/explanation/evaluation/faithfulness/index.html index 984d4e2d3..8555670ba 100644 --- a/dev/explanation/evaluation/faithfulness/index.html +++ b/dev/explanation/evaluation/faithfulness/index.html @@ -157,4 +157,4 @@ title = "ECCo Generator\nplaus.: $(round(plaus, digits=2))\nfaith.: $(round(faith, digits=2))" p3 = plot(img, title=title, axis=([], false)) -plot(p1, p2, p3; size=(600, 200), layout=(1, 3), topmargin=15mm)

References

Altmeyer, Patrick, Mojtaba Farmanbar, Arie van Deursen, and Cynthia CS Liem. 2024. “Faithful Model Explanations Through Energy-Constrained Conformal Counterfactuals.” In Proceedings of the AAAI Conference on Artificial Intelligence, 38:10829–37. 10.

LeCun, Yann. 1998. “The MNIST Database of Handwritten Digits.” http://yann.lecun.com/exdb/mnist/.

Slack, Dylan, Anna Hilgard, Himabindu Lakkaraju, and Sameer Singh. 2021. “Counterfactual Explanations Can Be Manipulated.” Advances in Neural Information Processing Systems 34.

+plot(p1, p2, p3; size=(600, 200), layout=(1, 3), topmargin=15mm)

References

Altmeyer, Patrick, Mojtaba Farmanbar, Arie van Deursen, and Cynthia CS Liem. 2024. “Faithful Model Explanations Through Energy-Constrained Conformal Counterfactuals.” In Proceedings of the AAAI Conference on Artificial Intelligence, 38:10829–37. 10.

LeCun, Yann. 1998. “The MNIST Database of Handwritten Digits.” http://yann.lecun.com/exdb/mnist/.

Slack, Dylan, Anna Hilgard, Himabindu Lakkaraju, and Sameer Singh. 2021. “Counterfactual Explanations Can Be Manipulated.” Advances in Neural Information Processing Systems 34.

diff --git a/dev/explanation/evaluation/overview/index.html b/dev/explanation/evaluation/overview/index.html index fd0c54cf2..0d508f7f2 100644 --- a/dev/explanation/evaluation/overview/index.html +++ b/dev/explanation/evaluation/overview/index.html @@ -1,2 +1,2 @@ -Overview · CounterfactualExplanations.jl
GitHub

Evaluation

Evaluation of counterfactual explanations is an integral part of the counterfactual explanation process. It is important to evaluate the quality of the generated counterfactual explanations to ensure that they are meaningful and useful. The tutorial provides an overview of the evaluation metrics and methods that can be used to evaluate counterfactual explanations. In this part of the documentation, we dive deeper into specific evaluation metrics and methods that can be used to evaluate counterfactual explanations.

+Overview · CounterfactualExplanations.jl
GitHub

Evaluation

Evaluation of counterfactual explanations is an integral part of the counterfactual explanation process. It is important to evaluate the quality of the generated counterfactual explanations to ensure that they are meaningful and useful. The tutorial provides an overview of the evaluation metrics and methods that can be used to evaluate counterfactual explanations. In this part of the documentation, we dive deeper into specific evaluation metrics and methods that can be used to evaluate counterfactual explanations.

diff --git a/dev/explanation/generators/clap_roar/index.html b/dev/explanation/generators/clap_roar/index.html index 7f78c15e2..ca72dd0ba 100644 --- a/dev/explanation/generators/clap_roar/index.html +++ b/dev/explanation/generators/clap_roar/index.html @@ -3,4 +3,4 @@ \text{extcost}(f(\mathbf{s}^\prime)) = l(M(f(\mathbf{s}^\prime)),y^\prime) \end{aligned}\]

for each counterfactual $k$ where $l$ denotes the loss function used to train $M$. This approach is based on the intuition that (endogenous) model shifts will be triggered by counterfactuals that increase classifier loss (Altmeyer et al. 2023).

Usage

The approach can be used in our package as follows:

generator = ClaPROARGenerator()
 ce = generate_counterfactual(x, target, counterfactual_data, M, generator)
-plot(ce)

Comparison to GenericGenerator

The figure below compares the outcome for the GenericGenerator and the ClaPROARGenerator.

References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In First IEEE Conference on Secure and Trustworthy Machine Learning. https://doi.org/10.1109/satml54575.2023.00036.

Upadhyay, Sohini, Shalmali Joshi, and Himabindu Lakkaraju. 2021. “Towards Robust and Reliable Algorithmic Recourse.” https://arxiv.org/abs/2102.13620.

+plot(ce)

Comparison to GenericGenerator

The figure below compares the outcome for the GenericGenerator and the ClaPROARGenerator.

References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In First IEEE Conference on Secure and Trustworthy Machine Learning. https://doi.org/10.1109/satml54575.2023.00036.

Upadhyay, Sohini, Shalmali Joshi, and Himabindu Lakkaraju. 2021. “Towards Robust and Reliable Algorithmic Recourse.” https://arxiv.org/abs/2102.13620.

diff --git a/dev/explanation/generators/clue/index.html b/dev/explanation/generators/clue/index.html index 43b2d1c81..335282930 100644 --- a/dev/explanation/generators/clue/index.html +++ b/dev/explanation/generators/clue/index.html @@ -7,4 +7,4 @@ ce = generate_counterfactual( x, target, counterfactual_data, M, generator; convergence=conv) -plot(ce)

Extra: The CLUE generator can also be used upon already having achieved a counterfactual with a different generator. In this case, you can use CLUE and make the counterfactual more robust.

Note: The above documentation is based on the information provided in the CLUE paper. Please refer to the original paper for more detailed explanations and implementation specifics.

References

Antorán, Javier, Umang Bhatt, Tameem Adel, Adrian Weller, and José Miguel Hernández-Lobato. 2020. “Getting a Clue: A Method for Explaining Uncertainty Estimates.” https://arxiv.org/abs/2006.06848.

+plot(ce)

Extra: The CLUE generator can also be used upon already having achieved a counterfactual with a different generator. In this case, you can use CLUE and make the counterfactual more robust.

Note: The above documentation is based on the information provided in the CLUE paper. Please refer to the original paper for more detailed explanations and implementation specifics.

References

Antorán, Javier, Umang Bhatt, Tameem Adel, Adrian Weller, and José Miguel Hernández-Lobato. 2020. “Getting a Clue: A Method for Explaining Uncertainty Estimates.” https://arxiv.org/abs/2006.06848.

diff --git a/dev/explanation/generators/dice/index.html b/dev/explanation/generators/dice/index.html index 7c6e24c60..a7189bdab 100644 --- a/dev/explanation/generators/dice/index.html +++ b/dev/explanation/generators/dice/index.html @@ -40,4 +40,4 @@ num_counterfactuals=n_cf, convergence=conv ) ) -end

The figure below shows the resulting counterfactual paths. As expected, the resulting counterfactuals are more dispersed across the feature domain for higher choices of $\lambda_2$

References

Mothilal, Ramaravind K, Amit Sharma, and Chenhao Tan. 2020. “Explaining Machine Learning Classifiers Through Diverse Counterfactual Explanations.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–17. https://doi.org/10.1145/3351095.3372850.

[1] With thanks to the respondents on Discourse

+end

The figure below shows the resulting counterfactual paths. As expected, the resulting counterfactuals are more dispersed across the feature domain for higher choices of $\lambda_2$

References

Mothilal, Ramaravind K, Amit Sharma, and Chenhao Tan. 2020. “Explaining Machine Learning Classifiers Through Diverse Counterfactual Explanations.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–17. https://doi.org/10.1145/3351095.3372850.

[1] With thanks to the respondents on Discourse

diff --git a/dev/explanation/generators/feature_tweak/index.html b/dev/explanation/generators/feature_tweak/index.html index f919b9706..0ef4209f8 100644 --- a/dev/explanation/generators/feature_tweak/index.html +++ b/dev/explanation/generators/feature_tweak/index.html @@ -31,4 +31,4 @@ colorbar=false, ) -display(plot(p1, p2; size=(800, 400)))

References

Tolomei, Gabriele, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017. “Interpretable Predictions of Tree-Based Ensembles via Actionable Feature Tweaking.” In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 465–74. https://doi.org/10.1145/3097983.3098039.

+display(plot(p1, p2; size=(800, 400)))

References

Tolomei, Gabriele, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017. “Interpretable Predictions of Tree-Based Ensembles via Actionable Feature Tweaking.” In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 465–74. https://doi.org/10.1145/3097983.3098039.

diff --git a/dev/explanation/generators/generic/index.html b/dev/explanation/generators/generic/index.html index 490e3632a..7cf1ab6fe 100644 --- a/dev/explanation/generators/generic/index.html +++ b/dev/explanation/generators/generic/index.html @@ -1,4 +1,4 @@ Generic · CounterfactualExplanations.jl
GitHub

GenericGenerator

We use the term generic to relate to the basic counterfactual generator proposed by Wachter, Mittelstadt, and Russell (2017) with $L1$-norm regularization. There is also a variant of this generator that uses the distance metric proposed in Wachter, Mittelstadt, and Russell (2017), which we call WachterGenerator.

Description

As the term indicates, this approach is simple: it forms the baseline approach for gradient-based counterfactual generators. Wachter, Mittelstadt, and Russell (2017) were among the first to realise that

[…] explanations can, in principle, be offered without opening the “black box.”

— Wachter, Mittelstadt, and Russell (2017)

Gradient descent is performed directly in the feature space. Concerning the cost heuristic, the authors choose to penalize the distance of counterfactuals from the factual value. This is based on the intuitive notion that larger feature perturbations require greater effort.

Usage

The approach can be used in our package as follows:

generator = GenericGenerator()
 ce = generate_counterfactual(x, target, counterfactual_data, M, generator)
-plot(ce)

References

Wachter, Sandra, Brent Mittelstadt, and Chris Russell. 2017. “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.” Harv. JL & Tech. 31: 841. https://doi.org/10.2139/ssrn.3063289.

+plot(ce)

References

Wachter, Sandra, Brent Mittelstadt, and Chris Russell. 2017. “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.” Harv. JL & Tech. 31: 841. https://doi.org/10.2139/ssrn.3063289.

diff --git a/dev/explanation/generators/gravitational/index.html b/dev/explanation/generators/gravitational/index.html index ce8aa2e32..381b01bbd 100644 --- a/dev/explanation/generators/gravitational/index.html +++ b/dev/explanation/generators/gravitational/index.html @@ -5,4 +5,4 @@ \text{extcost}(f(\mathbf{s}^\prime)) = \text{dist}(f(\mathbf{s}^\prime),\bar{x}^*) \end{aligned}\]

where $\bar{x}$ is some sensible point in the target domain, for example, the subsample average $\bar{x}^*=\text{mean}(x)$, $x \in \mathcal{D}_1$.

There is a tradeoff then, between the distance of counterfactuals from their factual value and the chosen point in the target domain. The chart below illustrates how the counterfactual outcome changes as the penalty $\lambda_2$ on the distance to the point in the target domain is increased from left to right (holding the other penalty term constant).

Usage

The approach can be used in our package as follows:

generator = GravitationalGenerator()
 ce = generate_counterfactual(x, target, counterfactual_data, M, generator)
-display(plot(ce))

Comparison to GenericGenerator

The figure below compares the outcome for the GenericGenerator and the GravitationalGenerator.

References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In First IEEE Conference on Secure and Trustworthy Machine Learning. https://doi.org/10.1109/satml54575.2023.00036.

+display(plot(ce))

Comparison to GenericGenerator

The figure below compares the outcome for the GenericGenerator and the GravitationalGenerator.

References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In First IEEE Conference on Secure and Trustworthy Machine Learning. https://doi.org/10.1109/satml54575.2023.00036.

diff --git a/dev/explanation/generators/greedy/index.html b/dev/explanation/generators/greedy/index.html index 0e652bd55..28903e9df 100644 --- a/dev/explanation/generators/greedy/index.html +++ b/dev/explanation/generators/greedy/index.html @@ -2,4 +2,4 @@ Greedy · CounterfactualExplanations.jl
GitHub

GreedyGenerator

We use the term greedy to describe the counterfactual generator introduced by Schut et al. (2021).

Description

The Greedy generator works under the premise of generating realistic counterfactuals by minimizing predictive uncertainty. Schut et al. (2021) show that for models that incorporates predictive uncertainty in their predictions, maximizing the predictive probability corresponds to minimizing the predictive uncertainty: by construction, the generated counterfactual will therefore be realistic (low epistemic uncertainty) and unambiguous (low aleatoric uncertainty).

For the counterfactual search Schut et al. (2021) propose using a Jacobian-based Saliency Map Attack(JSMA). It is greedy in the sense that it is an “iterative algorithm that updates the most salient feature, i.e. the feature that has the largest influence on the classification, by $\delta$ at each step” (Schut et al. 2021).

Usage

The approach can be used in our package as follows:

M = fit_model(counterfactual_data, :DeepEnsemble)
 generator = GreedyGenerator()
 ce = generate_counterfactual(x, target, counterfactual_data, M, generator)
-plot(ce)

References

Schut, Lisa, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

+plot(ce)

References

Schut, Lisa, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

diff --git a/dev/explanation/generators/growing_spheres/index.html b/dev/explanation/generators/growing_spheres/index.html index 43f54ceda..7403ccb2d 100644 --- a/dev/explanation/generators/growing_spheres/index.html +++ b/dev/explanation/generators/growing_spheres/index.html @@ -3,4 +3,4 @@ M = fit_model(counterfactual_data, :DeepEnsemble) ce = generate_counterfactual( x, target, counterfactual_data, M, generator) -plot(ce)

References

Laugel, Thibault, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2017. “Inverse Classification for Comparison-Based Interpretability in Machine Learning.” arXiv. https://doi.org/10.48550/arXiv.1712.08443.

+plot(ce)

References

Laugel, Thibault, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2017. “Inverse Classification for Comparison-Based Interpretability in Machine Learning.” arXiv. https://doi.org/10.48550/arXiv.1712.08443.

diff --git a/dev/explanation/generators/mint/index.html b/dev/explanation/generators/mint/index.html index eb824972d..b828dbc41 100644 --- a/dev/explanation/generators/mint/index.html +++ b/dev/explanation/generators/mint/index.html @@ -35,4 +35,4 @@ data_scm.input_encoder = fit_transformer(data_scm, CausalInference.SCM) ce = generate_counterfactual(x, 2, data_scm, M, GenericGenerator(); initialization=:identity)
CounterfactualExplanation
-Convergence: ❌ after 100 steps.
Note

The above documentation is based on the information provided in the MINT paper. Please refer to the original paper for more detailed explanations and implementation specifics.

References

Karimi, Amir-Hossein, Bernhard Schölkopf, and Isabel Valera. 2021. “Algorithmic Recourse: From Counterfactual Explanations to Interventions.” In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 353–62.

+Convergence: ❌ after 100 steps.
Note

The above documentation is based on the information provided in the MINT paper. Please refer to the original paper for more detailed explanations and implementation specifics.

References

Karimi, Amir-Hossein, Bernhard Schölkopf, and Isabel Valera. 2021. “Algorithmic Recourse: From Counterfactual Explanations to Interventions.” In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 353–62.

diff --git a/dev/explanation/generators/overview/index.html b/dev/explanation/generators/overview/index.html index 86f81df22..868e604e5 100644 --- a/dev/explanation/generators/overview/index.html +++ b/dev/explanation/generators/overview/index.html @@ -12,4 +12,4 @@ :generic => GenericGenerator :greedy => GreedyGenerator

The following sections provide brief descriptions of all of them.

Gradient-based Counterfactual Generators

At the time of writing, all generators are gradient-based: that is, counterfactuals are searched through gradient descent. In Altmeyer et al. (2023) we lay out a general methodological framework that can be applied to all of these generators:

\[\begin{aligned} \mathbf{s}^\prime &= \arg \min_{\mathbf{s}^\prime \in \mathcal{S}} \left\{ {\text{yloss}(M(f(\mathbf{s}^\prime)),y^*)}+ \lambda {\text{cost}(f(\mathbf{s}^\prime)) } \right\} -\end{aligned} \]

“Here $\mathbf{s}^\prime=\left\{s_k^\prime\right\}_K$ is a $K$-dimensional array of counterfactual states and $f: \mathcal{S} \mapsto \mathcal{X}$ maps from the counterfactual state space to the feature space.” (Altmeyer et al. 2023)

For most generators, the state space is the feature space ($f$ is the identity function) and the number of counterfactuals $K$ is one. Latent Space generators instead search counterfactuals in some latent space $\mathcal{S}$. In this case, $f$ corresponds to the decoder part of the generative model, that is the function that maps back from the latent space to inputs.

References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In First IEEE Conference on Secure and Trustworthy Machine Learning. https://doi.org/10.1109/satml54575.2023.00036.

+\end{aligned} \]

“Here $\mathbf{s}^\prime=\left\{s_k^\prime\right\}_K$ is a $K$-dimensional array of counterfactual states and $f: \mathcal{S} \mapsto \mathcal{X}$ maps from the counterfactual state space to the feature space.” (Altmeyer et al. 2023)

For most generators, the state space is the feature space ($f$ is the identity function) and the number of counterfactuals $K$ is one. Latent Space generators instead search counterfactuals in some latent space $\mathcal{S}$. In this case, $f$ corresponds to the decoder part of the generative model, that is the function that maps back from the latent space to inputs.

References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In First IEEE Conference on Secure and Trustworthy Machine Learning. https://doi.org/10.1109/satml54575.2023.00036.

diff --git a/dev/explanation/generators/probe/index.html b/dev/explanation/generators/probe/index.html index 5836818fb..3d21155eb 100644 --- a/dev/explanation/generators/probe/index.html +++ b/dev/explanation/generators/probe/index.html @@ -10,4 +10,4 @@ generator = CounterfactualExplanations.Generators.ProbeGenerator(opt=opt) conv = CounterfactualExplanations.Convergence.InvalidationRateConvergence(;invalidation_rate=0.5) ce = generate_counterfactual(x, target, counterfactual_data, M, generator, convergence=conv) -plot(ce)

Choosing different invalidation rates makes the counterfactual more or less robust. The following plot shows the counterfactuals generated for different invalidation rates.

References

Pawelczyk, Martin, Teresa Datta, Johannes van-den-Heuvel, Gjergji Kasneci, and Himabindu Lakkaraju. 2022. “Probabilistically Robust Recourse: Navigating the Trade-Offs Between Costs and Robustness in Algorithmic Recourse.” arXiv Preprint arXiv:2203.06768.

+plot(ce)

Choosing different invalidation rates makes the counterfactual more or less robust. The following plot shows the counterfactuals generated for different invalidation rates.

References

Pawelczyk, Martin, Teresa Datta, Johannes van-den-Heuvel, Gjergji Kasneci, and Himabindu Lakkaraju. 2022. “Probabilistically Robust Recourse: Navigating the Trade-Offs Between Costs and Robustness in Algorithmic Recourse.” arXiv Preprint arXiv:2203.06768.

diff --git a/dev/explanation/generators/revise/index.html b/dev/explanation/generators/revise/index.html index f82d84118..73b438ad3 100644 --- a/dev/explanation/generators/revise/index.html +++ b/dev/explanation/generators/revise/index.html @@ -52,4 +52,4 @@ # Define generator: generator = REVISEGenerator(opt=Flux.Adam(0.1)) # Generate recourse: -ce = generate_counterfactual(x, target, counterfactual_data, M, generator; convergence=conv)

The chart below shows the results:

References

Joshi, Shalmali, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019. “Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems.” https://arxiv.org/abs/1907.09615.

[1] In general, we believe that there may be a trade-off between creating counterfactuals that respect the DGP vs. counterfactuals reflect the behaviour of the black-model in question - both accurately and complete.

[2] We believe that there is another potentially crucial disadvantage of relying on a separate generative model: it reallocates the task of learning realistic explanations for the data from the black-box model to the generative model.

+ce = generate_counterfactual(x, target, counterfactual_data, M, generator; convergence=conv)

The chart below shows the results:

References

Joshi, Shalmali, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019. “Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems.” https://arxiv.org/abs/1907.09615.

[1] In general, we believe that there may be a trade-off between creating counterfactuals that respect the DGP vs. counterfactuals reflect the behaviour of the black-model in question - both accurately and complete.

[2] We believe that there is another potentially crucial disadvantage of relying on a separate generative model: it reallocates the task of learning realistic explanations for the data from the black-box model to the generative model.

diff --git a/dev/explanation/generators/tcrex/index.html b/dev/explanation/generators/tcrex/index.html index 59bd812bf..9a5610ac5 100644 --- a/dev/explanation/generators/tcrex/index.html +++ b/dev/explanation/generators/tcrex/index.html @@ -99,4 +99,4 @@ p4

(g) Local CE example

To generate a local explanation based on the global CE representation, we simply apply the CART decision tree classifier from the previous step to our factual:

optimal_rule = apply_tree(tree, vec(x))
 p5 = deepcopy(p2)
 scatter!(p5, [x[1]], [x[2]], ms=10, color=2+optimal_rule, label="Local CE (move to R$optimal_rule)")
-p5

References

Bewley, Tom, Salim I. Amoukou, Saumitra Mishra, Daniele Magazzeni, and Manuela Veloso. 2024. “Counterfactual Metarules for Local and Global Recourse.” In Proceedings of the 41st International Conference on Machine Learning, edited by Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, 235:3707–24. Proceedings of Machine Learning Research. PMLR. https://proceedings.mlr.press/v235/bewley24a.html.

+p5

References

Bewley, Tom, Salim I. Amoukou, Saumitra Mishra, Daniele Magazzeni, and Manuela Veloso. 2024. “Counterfactual Metarules for Local and Global Recourse.” In Proceedings of the 41st International Conference on Machine Learning, edited by Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, 235:3707–24. Proceedings of Machine Learning Research. PMLR. https://proceedings.mlr.press/v235/bewley24a.html.

diff --git a/dev/explanation/index.html b/dev/explanation/index.html index 54da46998..cd7b18a79 100644 --- a/dev/explanation/index.html +++ b/dev/explanation/index.html @@ -1,2 +1,2 @@ -Overview · CounterfactualExplanations.jl
GitHub

Explanation

In this section you will find detailed explanations about the methodology and code.

Explanation clarifies, deepens and broadens the reader’s understanding of a subject.

— Diátaxis

In other words, you come here because you are interested in understanding how all of this actually works 🤓.

+Overview · CounterfactualExplanations.jl
GitHub

Explanation

In this section you will find detailed explanations about the methodology and code.

Explanation clarifies, deepens and broadens the reader’s understanding of a subject.

— Diátaxis

In other words, you come here because you are interested in understanding how all of this actually works 🤓.

diff --git a/dev/explanation/optimisers/jsma/index.html b/dev/explanation/optimisers/jsma/index.html index 2c14c56e7..6aa7f5a67 100644 --- a/dev/explanation/optimisers/jsma/index.html +++ b/dev/explanation/optimisers/jsma/index.html @@ -2,4 +2,4 @@ JSMA · CounterfactualExplanations.jl
GitHub

Jacobian-based Saliency Map Attack

To search counterfactuals, Schut et al. (2021) propose to use a Jacobian-Based Saliency Map Attack (JSMA) inspired by the literature on adversarial attacks. It works by moving in the direction of the most salient feature at a fixed step size in each iteration. Schut et al. (2021) use this optimisation rule in the context of Bayesian classifiers and demonstrate good results in terms of plausibility — how realistic counterfactuals are — and redundancy — how sparse the proposed feature changes are.

JSMADescent

To implement this approach in a reusable manner, we have added JSMA as a Flux optimiser. In particular, we have added a class JSMADescent<:Flux.Optimise.AbstractOptimiser, for which we have overloaded the Flux.Optimise.apply! method. This makes it possible to reuse JSMADescent as an optimiser in composable generators.

The optimiser can be used with with any generator as follows:

using CounterfactualExplanations.Generators: JSMADescent
 generator = GenericGenerator() |>
     gen -> @with_optimiser(gen,JSMADescent(;η=0.1))
-ce = generate_counterfactual(x, target, counterfactual_data, M, generator)

The figure below compares the resulting counterfactual search outcome to the corresponding outcome with generic Descent.

plot(p1,p2,size=(1000,400))

Schut, Lisa, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

+ce = generate_counterfactual(x, target, counterfactual_data, M, generator)

The figure below compares the resulting counterfactual search outcome to the corresponding outcome with generic Descent.

plot(p1,p2,size=(1000,400))

Schut, Lisa, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

diff --git a/dev/explanation/optimisers/overview/index.html b/dev/explanation/optimisers/overview/index.html index e81417fe5..058d54815 100644 --- a/dev/explanation/optimisers/overview/index.html +++ b/dev/explanation/optimisers/overview/index.html @@ -1,2 +1,2 @@ -Overview · CounterfactualExplanations.jl
GitHub

Optimisation Rules

Counterfactual search is an optimization problem. Consequently, the choice of the optimisation rule affects the generated counterfactuals. In the short term, we aim to enable users to choose any of the available Flux optimisers. This has not been sufficiently tested yet, and you may run into issues.

Custom Optimisation Rules

Flux optimisers are specifically designed for deep learning, and in particular, for learning model parameters. In counterfactual search, the features are the free parameters that we are optimising over. To this end, some custom optimisation rules are necessary to incorporate ideas presented in the literature. In the following, we introduce those rules.

+Overview · CounterfactualExplanations.jl
GitHub

Optimisation Rules

Counterfactual search is an optimization problem. Consequently, the choice of the optimisation rule affects the generated counterfactuals. In the short term, we aim to enable users to choose any of the available Flux optimisers. This has not been sufficiently tested yet, and you may run into issues.

Custom Optimisation Rules

Flux optimisers are specifically designed for deep learning, and in particular, for learning model parameters. In counterfactual search, the features are the free parameters that we are optimising over. To this end, some custom optimisation rules are necessary to incorporate ideas presented in the literature. In the following, we introduce those rules.

diff --git a/dev/extensions/index.html b/dev/extensions/index.html index 7a27e200a..9f46447c5 100644 --- a/dev/extensions/index.html +++ b/dev/extensions/index.html @@ -1,2 +1,2 @@ -Overview · CounterfactualExplanations.jl
GitHub

⛓️ Extensions

In this section, you will find information about package extensions of the CounterfactualExplanations package. Extensions are a relatively new feature of Julia that allows users to conditionally load code based on the presence of other packages. This is useful for creating packages that extend the functionality of other packages, without requiring the user to install the package being extended.

+Overview · CounterfactualExplanations.jl
GitHub

⛓️ Extensions

In this section, you will find information about package extensions of the CounterfactualExplanations package. Extensions are a relatively new feature of Julia that allows users to conditionally load code based on the presence of other packages. This is useful for creating packages that extend the functionality of other packages, without requiring the user to install the package being extended.

diff --git a/dev/extensions/laplace_redux/index.html b/dev/extensions/laplace_redux/index.html index 0e8252311..b0b248fe9 100644 --- a/dev/extensions/laplace_redux/index.html +++ b/dev/extensions/laplace_redux/index.html @@ -14,4 +14,4 @@ decision_threshold=0.9, max_iter=100 ) ce = generate_counterfactual(x, target, data, M, generator; convergence=conv) -plot(ce, alpha=0.1)

References

Daxberger, Erik, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. 2021. “Laplace Redux-Effortless Bayesian Deep Learning.” Advances in Neural Information Processing Systems 34.

Immer, Alexander, Maciej Korzepa, and Matthias Bauer. 2020. “Improving Predictions of Bayesian Neural Networks via Local Linearization.” https://arxiv.org/abs/2008.08400.

Schut, Lisa, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

+plot(ce, alpha=0.1)

References

Daxberger, Erik, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. 2021. “Laplace Redux-Effortless Bayesian Deep Learning.” Advances in Neural Information Processing Systems 34.

Immer, Alexander, Maciej Korzepa, and Matthias Bauer. 2020. “Improving Predictions of Bayesian Neural Networks via Local Linearization.” https://arxiv.org/abs/2008.08400.

Schut, Lisa, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

diff --git a/dev/extensions/neurotree/index.html b/dev/extensions/neurotree/index.html index 4aaa3551c..c3505fb28 100644 --- a/dev/extensions/neurotree/index.html +++ b/dev/extensions/neurotree/index.html @@ -17,4 +17,4 @@ decision_threshold=0.9, max_iter=100 ) ce = generate_counterfactual(x, target, data, M, generator; convergence=conv) -plot(ce, alpha=0.1)

References

Grinsztajn, Léo, Edouard Oyallon, and Gaël Varoquaux. 2022. “Why Do Tree-Based Models Still Outperform Deep Learning on Tabular Data?” https://arxiv.org/abs/2207.08815.

+plot(ce, alpha=0.1)

References

Grinsztajn, Léo, Edouard Oyallon, and Gaël Varoquaux. 2022. “Why Do Tree-Based Models Still Outperform Deep Learning on Tabular Data?” https://arxiv.org/abs/2207.08815.

diff --git a/dev/how_to_guides/custom_generators/index.html b/dev/how_to_guides/custom_generators/index.html index 6b84405d7..e0f96909d 100644 --- a/dev/how_to_guides/custom_generators/index.html +++ b/dev/how_to_guides/custom_generators/index.html @@ -55,4 +55,4 @@ x, target, counterfactual_data, M, generator; num_counterfactuals=5) -plot(ce)

+plot(ce)

diff --git a/dev/how_to_guides/custom_models/index.html b/dev/how_to_guides/custom_models/index.html index df5f9a6a9..bee276fa9 100644 --- a/dev/how_to_guides/custom_models/index.html +++ b/dev/how_to_guides/custom_models/index.html @@ -51,4 +51,4 @@ # Counterfactual search: generator = GenericGenerator() ce = generate_counterfactual(x, target, counterfactual_data, M, generator) -plot(ce)

References

Innes, Mike. 2018. “Flux: Elegant Machine Learning with Julia.” Journal of Open Source Software 3 (25): 602. https://doi.org/10.21105/joss.00602.

+plot(ce)

References

Innes, Mike. 2018. “Flux: Elegant Machine Learning with Julia.” Journal of Open Source Software 3 (25): 602. https://doi.org/10.21105/joss.00602.

diff --git a/dev/how_to_guides/index.html b/dev/how_to_guides/index.html index d981f2887..d2e72cb54 100644 --- a/dev/how_to_guides/index.html +++ b/dev/how_to_guides/index.html @@ -1,2 +1,2 @@ -Overview · CounterfactualExplanations.jl
GitHub

How-To Guides

In this section, you will find a series of how-to-guides that showcase specific use cases of counterfactual explanations (CE).

How-to guides are directions that take the reader through the steps required to solve a real-world problem. How-to guides are goal-oriented.

— Diátaxis

In other words, you come here because you may have some particular problem in mind, would like to see how it can be solved using CE and then most likely head off again 🫡.

+Overview · CounterfactualExplanations.jl
GitHub

How-To Guides

In this section, you will find a series of how-to-guides that showcase specific use cases of counterfactual explanations (CE).

How-to guides are directions that take the reader through the steps required to solve a real-world problem. How-to guides are goal-oriented.

— Diátaxis

In other words, you come here because you may have some particular problem in mind, would like to see how it can be solved using CE and then most likely head off again 🫡.

diff --git a/dev/index.html b/dev/index.html index 804e59b54..56eb9e40a 100644 --- a/dev/index.html +++ b/dev/index.html @@ -69,4 +69,4 @@ author = {Patrick Altmeyer and Arie van Deursen and Cynthia C. s. Liem}, title = {Explaining Black-Box Models through Counterfactuals}, journal = {Proceedings of the JuliaCon Conferences} -}

📚 References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia CS Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 418–31. IEEE.

Altmeyer, Patrick, Mojtaba Farmanbar, Arie van Deursen, and Cynthia CS Liem. 2024. “Faithful Model Explanations Through Energy-Constrained Conformal Counterfactuals.” In Proceedings of the AAAI Conference on Artificial Intelligence, 38:10829–37. 10.

Antorán, Javier, Umang Bhatt, Tameem Adel, Adrian Weller, and José Miguel Hernández-Lobato. 2020. “Getting a Clue: A Method for Explaining Uncertainty Estimates.” https://arxiv.org/abs/2006.06848.

Bewley, Tom, Salim I. Amoukou, Saumitra Mishra, Daniele Magazzeni, and Manuela Veloso. 2024. “Counterfactual Metarules for Local and Global Recourse.” https://arxiv.org/abs/2405.18875.

Joshi, Shalmali, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019. “Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems.” https://arxiv.org/abs/1907.09615.

Kaggle. 2011. “Give Me Some Credit, Improve on the State of the Art in Credit Scoring by Predicting the Probability That Somebody Will Experience Financial Distress in the Next Two Years.” https://www.kaggle.com/c/GiveMeSomeCredit; Kaggle. https://www.kaggle.com/c/GiveMeSomeCredit.

Karimi, Amir-Hossein, Julius Von Kügelgen, Bernhard Schölkopf, and Isabel Valera. 2020. “Algorithmic Recourse Under Imperfect Causal Knowledge: A Probabilistic Approach.” https://arxiv.org/abs/2006.06831.

Laugel, Thibault, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2017. “Inverse Classification for Comparison-Based Interpretability in Machine Learning.” https://arxiv.org/abs/1712.08443.

Mothilal, Ramaravind K, Amit Sharma, and Chenhao Tan. 2020. “Explaining Machine Learning Classifiers Through Diverse Counterfactual Explanations.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–17. https://doi.org/10.1145/3351095.3372850.

Pawelczyk, Martin, Teresa Datta, Johannes van-den-Heuvel, Gjergji Kasneci, and Himabindu Lakkaraju. 2023. “Probabilistically Robust Recourse: Navigating the Trade-Offs Between Costs and Robustness in Algorithmic Recourse.” https://arxiv.org/abs/2203.06768.

Schut, Lisa, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

Tolomei, Gabriele, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017. “Interpretable Predictions of Tree-Based Ensembles via Actionable Feature Tweaking.” In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 465–74. https://doi.org/10.1145/3097983.3098039.

Wachter, Sandra, Brent Mittelstadt, and Chris Russell. 2017. “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.” Harv. JL & Tech. 31: 841. https://doi.org/10.2139/ssrn.3063289.

+}

📚 References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia CS Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 418–31. IEEE.

Altmeyer, Patrick, Mojtaba Farmanbar, Arie van Deursen, and Cynthia CS Liem. 2024. “Faithful Model Explanations Through Energy-Constrained Conformal Counterfactuals.” In Proceedings of the AAAI Conference on Artificial Intelligence, 38:10829–37. 10.

Antorán, Javier, Umang Bhatt, Tameem Adel, Adrian Weller, and José Miguel Hernández-Lobato. 2020. “Getting a Clue: A Method for Explaining Uncertainty Estimates.” https://arxiv.org/abs/2006.06848.

Bewley, Tom, Salim I. Amoukou, Saumitra Mishra, Daniele Magazzeni, and Manuela Veloso. 2024. “Counterfactual Metarules for Local and Global Recourse.” https://arxiv.org/abs/2405.18875.

Joshi, Shalmali, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019. “Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems.” https://arxiv.org/abs/1907.09615.

Kaggle. 2011. “Give Me Some Credit, Improve on the State of the Art in Credit Scoring by Predicting the Probability That Somebody Will Experience Financial Distress in the Next Two Years.” https://www.kaggle.com/c/GiveMeSomeCredit; Kaggle. https://www.kaggle.com/c/GiveMeSomeCredit.

Karimi, Amir-Hossein, Julius Von Kügelgen, Bernhard Schölkopf, and Isabel Valera. 2020. “Algorithmic Recourse Under Imperfect Causal Knowledge: A Probabilistic Approach.” https://arxiv.org/abs/2006.06831.

Laugel, Thibault, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2017. “Inverse Classification for Comparison-Based Interpretability in Machine Learning.” https://arxiv.org/abs/1712.08443.

Mothilal, Ramaravind K, Amit Sharma, and Chenhao Tan. 2020. “Explaining Machine Learning Classifiers Through Diverse Counterfactual Explanations.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–17. https://doi.org/10.1145/3351095.3372850.

Pawelczyk, Martin, Teresa Datta, Johannes van-den-Heuvel, Gjergji Kasneci, and Himabindu Lakkaraju. 2023. “Probabilistically Robust Recourse: Navigating the Trade-Offs Between Costs and Robustness in Algorithmic Recourse.” https://arxiv.org/abs/2203.06768.

Schut, Lisa, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

Tolomei, Gabriele, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017. “Interpretable Predictions of Tree-Based Ensembles via Actionable Feature Tweaking.” In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 465–74. https://doi.org/10.1145/3097983.3098039.

Wachter, Sandra, Brent Mittelstadt, and Chris Russell. 2017. “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.” Harv. JL & Tech. 31: 841. https://doi.org/10.2139/ssrn.3063289.

diff --git a/dev/reference/index.html b/dev/reference/index.html index c029b6c48..2788e8c5b 100644 --- a/dev/reference/index.html +++ b/dev/reference/index.html @@ -321,4 +321,4 @@ M::Models.Model, type::CounterfactualExplanations.NeuroTreeModel, X::AbstractArray, -)

Overloads the probs method for NeuroTree models.

source
CounterfactualExplanations.Models.train — Method
train(M::NeuroTreeModel, data::CounterfactualData; kwargs...)

Fits the model M to the data in the CounterfactualData object. This method is not called by the user directly.

Arguments

  • M::NeuroTreeModel: The wrapper for an NeuroTree model.
  • data::CounterfactualData: The CounterfactualData object containing the data to be used for training the model.

Returns

  • M::NeuroTreeModel: The fitted NeuroTree model.
source
+)

Overloads the probs method for NeuroTree models.

source
CounterfactualExplanations.Models.train — Method
train(M::NeuroTreeModel, data::CounterfactualData; kwargs...)

Fits the model M to the data in the CounterfactualData object. This method is not called by the user directly.

Arguments

  • M::NeuroTreeModel: The wrapper for an NeuroTree model.
  • data::CounterfactualData: The CounterfactualData object containing the data to be used for training the model.

Returns

  • M::NeuroTreeModel: The fitted NeuroTree model.
source
diff --git a/dev/release-notes/index.html b/dev/release-notes/index.html index af56dca26..2dea87a63 100644 --- a/dev/release-notes/index.html +++ b/dev/release-notes/index.html @@ -1,2 +1,2 @@ -Release Notes · CounterfactualExplanations.jl
GitHub

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note: We try to adhere to these practices as of version v1.1.1.

Version [1.3.4] - 2024-10-22

Changed

  • Fixed a bug in the find_potential_neighbours method.

Version [1.3.3] - 2024-09-30

Changed

  • Fixed a remaining bug in NeuroTreeExt extensions. #475

Version [1.3.2] - 2024-09-24

Added

  • Added support for using a random forest as a surrogate model for the T-CREx generator. #483

Changed

  • Improved the T-CREx documentation further by bringing example even closer to the example in the paper. #483
  • Include citation linking to ICML paper in T-CREx documentation and docstrings. #480

Version [1.3.1] - 2024-09-24

Changed

  • Fixed a remaining bug in NeuroTreeExt extensions. #475

Version [1.3.0] - 2024-09-16

Changed

  • Fixed bug in NeuroTreeExt extensions. #475

Added

  • Added basic support for the T-CREx counterfactual generator. #473
  • Added docstrings for package extensions to documentation. #475

Version [1.2.0] - 2024-09-10

Added

  • Added documentation for generating counterfactuals consistent with the MINT framework. #467
  • Added tests for new evaluation metrics and JEM extension. #471
  • Added support for gradient-based causal algorithm-recourse (MNIT) as described in Karimi et al. (2020). This incorporates an input encoder that is based on a Structural Causal Model #457
  • Added out-of-the-box support for training joint energy models (JEM). #454
  • Added new evaluation metric to measure faithfulness of counterfactual explanations as in Altmeyer et al. (2024). #454
  • A tutorial in the documentation ("Explanation" section) explaining the faithfulness metric in detail. #454
  • Added support for an energy constraint as in Altmeyer et al. (2024). This is the first step towards adding functionality for ECCCo. #387

Changed

  • The fitresult field of Model now takes a concrete Fitresult type, for which some basic methods have been defined. This mutable struct has a field called other that accepts a dictionary Dict that can be filled with additional objects. #454
  • Regenerated pre-trained model artifacts. #454
  • Updated the tutorial on "Handling Data". #454

Removed

  • Removed bug in find_potential_neighbours method. #454

Version [1.1.6] - 2024-05-19

Removed

  • Removed the call to the Iris function in the test suite because of HTTPs issues. #452
  • Removed the mlj_models_catalogue because it served no obvious purpose. In the future, we may instead add meta information to the all_models_catalogue. #444

Added

  • New general Model struct that wraps empty concrete types. This adds a more general interface that is still flexible enough by simply using multiple dispatch on the empty concrete types. #444
  • A new incompatible(::AbstractGenerator, ::AbstractCounterfactualExplanation) function has been added to avoid running a counterfactual search if the generator is incompatible with any other specification (e.g. the model). #444

Changed

  • No longer exporting many of the deprecated functions. #452
  • Updated pre-trained model artifacts. #444
  • Some function signatures have been deprecated, e.g. NeuroTreeModel to NeuroTree, LaplaceReduxModel to LaplaceNN. #444
  • Support for DecisionTree.jl models and the FeatureTweakGenerator have been moved to an extension (DecisionTreeExt). #444
  • Updates to NeuroTreeModels extensions to incorporate breaking changes to package. #444
  • No longer running alloc test on Windows. #441
  • Slight change to doctests. #447

Version v1.1.5 - 2024-04-30

Added

  • Unit tests: adds a simple performance benchmark to test that for a small problem, generating a counterfactual using the generic generator takes at most 4700 allocations. Only run on julia v1.10 and higher. #436

Changed

  • The find_potential_neighbours is now only triggered if one of the penalties of the generator requires access to samples from the target domain. This improves scalability because calling the function can be computationally costly (forward-pass). #436
  • The target variable encodings are now handled more efficiently. Previously certain tasks were repeated, which was not necessary. #436

Removed

  • Removed the assertion checking that the model ever predicts the target value. While this assertion is useful, it is not essential. For large enough models and datasets, this forward pass can be very costly. #436
  • Removed redundant distance_from_targets function. #436

Version v1.1.4 - 2024-04-25

Changed

  • Refactors the encodings and decodings such that it is now more streamlined. Instead of conditional statements, encodings are now dispatched on the type of a new unifying data.input_encoder field. #432
  • Refactors the check for redundancy. This is now based on the convergence type and done right before the counterfactual search begins, if not redundant. #432

Added

  • Added additional unit tests. #437

Version v1.1.3 - 2024-04-17

Added

  • Adds a section on Convergence to the documentation, Changelog.jl functionality and a few doc tests. #429

Changed

  • Changes style of taking gradients for the counterfactual search from implicit to explicit. #430
  • Removed all implicit imports. #430

Removed

  • Removed CUDA.jl dependency, because redundant. #430
  • Removed Parameters.jl dependency, because redundant. #430

Version v1.1.2 - 2024-04-16

Changed

  • Replaces the GIF in the README and introduction of docs for a static image.

Version v1.1.1 - 2024-04-15

Added

  • Added tests for LaplaceRedux extension. Bumped upper compat bound for LaplaceRedux.jl. #428
+Release Notes · CounterfactualExplanations.jl
GitHub

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note: We try to adhere to these practices as of version v1.1.1.

Version [1.3.4] - 2024-10-22

Changed

  • Fixed a bug in the find_potential_neighbours method.

Version [1.3.3] - 2024-09-30

Changed

  • Fixed a remaining bug in NeuroTreeExt extensions. #475

Version [1.3.2] - 2024-09-24

Added

  • Added support for using a random forest as a surrogate model for the T-CREx generator. #483

Changed

  • Improved the T-CREx documentation further by bringing example even closer to the example in the paper. #483
  • Include citation linking to ICML paper in T-CREx documentation and docstrings. #480

Version [1.3.1] - 2024-09-24

Changed

  • Fixed a remaining bug in NeuroTreeExt extensions. #475

Version [1.3.0] - 2024-09-16

Changed

  • Fixed bug in NeuroTreeExt extensions. #475

Added

  • Added basic support for the T-CREx counterfactual generator. #473
  • Added docstrings for package extensions to documentation. #475

Version [1.2.0] - 2024-09-10

Added

  • Added documentation for generating counterfactuals consistent with the MINT framework. #467
  • Added tests for new evaluation metrics and JEM extension. #471
  • Added support for gradient-based causal algorithm-recourse (MNIT) as described in Karimi et al. (2020). This incorporates an input encoder that is based on a Structural Causal Model #457
  • Added out-of-the-box support for training joint energy models (JEM). #454
  • Added new evaluation metric to measure faithfulness of counterfactual explanations as in Altmeyer et al. (2024). #454
  • A tutorial in the documentation ("Explanation" section) explaining the faithfulness metric in detail. #454
  • Added support for an energy constraint as in Altmeyer et al. (2024). This is the first step towards adding functionality for ECCCo. #387

Changed

  • The fitresult field of Model now takes a concrete Fitresult type, for which some basic methods have been defined. This mutable struct has a field called other that accepts a dictionary Dict that can be filled with additional objects. #454
  • Regenerated pre-trained model artifacts. #454
  • Updated the tutorial on "Handling Data". #454

Removed

  • Removed bug in find_potential_neighbours method. #454

Version [1.1.6] - 2024-05-19

Removed

  • Removed the call to the Iris function in the test suite because of HTTPs issues. #452
  • Removed the mlj_models_catalogue because it served no obvious purpose. In the future, we may instead add meta information to the all_models_catalogue. #444

Added

  • New general Model struct that wraps empty concrete types. This adds a more general interface that is still flexible enough by simply using multiple dispatch on the empty concrete types. #444
  • A new incompatible(::AbstractGenerator, ::AbstractCounterfactualExplanation) function has been added to avoid running a counterfactual search if the generator is incompatible with any other specification (e.g. the model). #444

Changed

  • No longer exporting many of the deprecated functions. #452
  • Updated pre-trained model artifacts. #444
  • Some function signatures have been deprecated, e.g. NeuroTreeModel to NeuroTree, LaplaceReduxModel to LaplaceNN. #444
  • Support for DecisionTree.jl models and the FeatureTweakGenerator have been moved to an extension (DecisionTreeExt). #444
  • Updates to NeuroTreeModels extensions to incorporate breaking changes to package. #444
  • No longer running alloc test on Windows. #441
  • Slight change to doctests. #447

Version v1.1.5 - 2024-04-30

Added

  • Unit tests: adds a simple performance benchmark to test that for a small problem, generating a counterfactual using the generic generator takes at most 4700 allocations. Only run on julia v1.10 and higher. #436

Changed

  • The find_potential_neighbours is now only triggered if one of the penalties of the generator requires access to samples from the target domain. This improves scalability because calling the function can be computationally costly (forward-pass). #436
  • The target variable encodings are now handled more efficiently. Previously certain tasks were repeated, which was not necessary. #436

Removed

  • Removed the assertion checking that the model ever predicts the target value. While this assertion is useful, it is not essential. For large enough models and datasets, this forward pass can be very costly. #436
  • Removed redundant distance_from_targets function. #436

Version v1.1.4 - 2024-04-25

Changed

  • Refactors the encodings and decodings such that it is now more streamlined. Instead of conditional statements, encodings are now dispatched on the type of a new unifying data.input_encoder field. #432
  • Refactors the check for redundancy. This is now based on the convergence type and done right before the counterfactual search begins, if not redundant. #432

Added

  • Added additional unit tests. #437

Version v1.1.3 - 2024-04-17

Added

  • Adds a section on Convergence to the documentation, Changelog.jl functionality and a few doc tests. #429

Changed

  • Changes style of taking gradients for the counterfactual search from implicit to explicit. #430
  • Removed all implicit imports. #430

Removed

  • Removed CUDA.jl dependency, because redundant. #430
  • Removed Parameters.jl dependency, because redundant. #430

Version v1.1.2 - 2024-04-16

Changed

  • Replaces the GIF in the README and introduction of docs for a static image.

Version v1.1.1 - 2024-04-15

Added

  • Added tests for LaplaceRedux extension. Bumped upper compat bound for LaplaceRedux.jl. #428
diff --git a/dev/tutorials/benchmarking/index.html b/dev/tutorials/benchmarking/index.html index 80e57dbaf..8b8b91565 100644 --- a/dev/tutorials/benchmarking/index.html +++ b/dev/tutorials/benchmarking/index.html @@ -180,4 +180,4 @@ 1 │ moons Generic 1.56555 2 │ moons Greedy 0.819269 3 │ circles Generic 1.83524 - 4 │ circles Greedy 0.498953 + 4 │ circles Greedy 0.498953 diff --git a/dev/tutorials/convergence/index.html b/dev/tutorials/convergence/index.html index 237781202..434154ba9 100644 --- a/dev/tutorials/convergence/index.html +++ b/dev/tutorials/convergence/index.html @@ -10,4 +10,4 @@ for (ce, titl) in zip([ce_gen, ce_dec, ce_max], ["Gradient Convergence", "Decision Threshold Convergence", "Max Iterations Convergence"]) push!(plts, plot(ce; title=titl, cbar=false)) end -plot(plts..., layout=(1,3), size=(1200, 380))

References

Altmeyer, Patrick, Mojtaba Farmanbar, Arie van Deursen, and Cynthia CS Liem. 2024. “Faithful Model Explanations Through Energy-Constrained Conformal Counterfactuals.” In Proceedings of the AAAI Conference on Artificial Intelligence, 38:10829–37. 10.

+plot(plts..., layout=(1,3), size=(1200, 380))

References

Altmeyer, Patrick, Mojtaba Farmanbar, Arie van Deursen, and Cynthia CS Liem. 2024. “Faithful Model Explanations Through Energy-Constrained Conformal Counterfactuals.” In Proceedings of the AAAI Conference on Artificial Intelligence, 38:10829–37. 10.

diff --git a/dev/tutorials/data_catalogue/index.html b/dev/tutorials/data_catalogue/index.html index 2b5e866f2..04d01c2d8 100644 --- a/dev/tutorials/data_catalogue/index.html +++ b/dev/tutorials/data_catalogue/index.html @@ -38,4 +38,4 @@ 10 10 10

We can also use a helper function to split the data into train and test sets:

train_data, test_data = 
-    CounterfactualExplanations.DataPreprocessing.train_test_split(counterfactual_data)
+ CounterfactualExplanations.DataPreprocessing.train_test_split(counterfactual_data) diff --git a/dev/tutorials/data_preprocessing/index.html b/dev/tutorials/data_preprocessing/index.html index 4f8ac5df9..0ea0a2a28 100644 --- a/dev/tutorials/data_preprocessing/index.html +++ b/dev/tutorials/data_preprocessing/index.html @@ -89,4 +89,4 @@ x = select_factual(counterfactual_data, chosen) ce = generate_counterfactual(x, target, counterfactual_data, M, generator)

The resulting counterfactual path is shown in the chart below. Since only the first feature can be perturbed, the sample can only move along the horizontal axis.

plot(ce)

Domain constraints

In some cases, we may also want to constrain the domain of some feature. For example, age as a feature is constrained to a range from 0 to some upper bound corresponding perhaps to the average life expectancy of humans. Below, for example, we impose a lower bound of $0.5$ for our two features.

counterfactual_data.mutability = [:both, :both]
 counterfactual_data.domain = [(0.5,Inf) for var in counterfactual_data.features_continuous]

This results in the counterfactual path shown below: since features are not allowed to be perturbed beyond the upper bound, the resulting counterfactual falls just short of the threshold probability $\gamma$.

ce = generate_counterfactual(x, target, counterfactual_data, M, generator)
-plot(ce)
+plot(ce) diff --git a/dev/tutorials/evaluation/index.html b/dev/tutorials/evaluation/index.html index beaf4cdc2..bd659f6f7 100644 --- a/dev/tutorials/evaluation/index.html +++ b/dev/tutorials/evaluation/index.html @@ -42,4 +42,4 @@ [[1.0], Float32[2.9578466], [[0.0, 0.0, 0.0, 0.0, 0.0]]] [[0.8], Float32[2.6089585], [[0.0, 0.0, 0.0, 0.0, 0.0]]] -Vector{Vector}[[[0.8], Float32[3.2487042], [[0.0, 0.0, 0.0, 0.0, 0.0]]], [[0.8], Float32[4.185718], [[0.0, 0.0, 0.0, 0.0, 0.0]]], [[1.0], Float32[4.0083566], [[0.0, 0.0, 0.0, 0.0, 0.0]]], [[1.0], Float32[2.9578466], [[0.0, 0.0, 0.0, 0.0, 0.0]]], [[0.8], Float32[2.6089585], [[0.0, 0.0, 0.0, 0.0, 0.0]]]]

This leads us to our next topic: Performance Benchmarks.

+Vector{Vector}[[[0.8], Float32[3.2487042], [[0.0, 0.0, 0.0, 0.0, 0.0]]], [[0.8], Float32[4.185718], [[0.0, 0.0, 0.0, 0.0, 0.0]]], [[1.0], Float32[4.0083566], [[0.0, 0.0, 0.0, 0.0, 0.0]]], [[1.0], Float32[2.9578466], [[0.0, 0.0, 0.0, 0.0, 0.0]]], [[0.8], Float32[2.6089585], [[0.0, 0.0, 0.0, 0.0, 0.0]]]]

This leads us to our next topic: Performance Benchmarks.

diff --git a/dev/tutorials/generators/index.html b/dev/tutorials/generators/index.html index 82b41d16e..9b720b44e 100644 --- a/dev/tutorials/generators/index.html +++ b/dev/tutorials/generators/index.html @@ -22,4 +22,4 @@ :greedy => GreedyGenerator

To specify the type of generator you want to use, you can simply instantiate it:

# Search:
 generator = GenericGenerator()
 ce = generate_counterfactual(x, target, counterfactual_data, M, generator)
-plot(ce)

We generally make an effort to follow the literature as closely as possible when implementing off-the-shelf generators.

References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In First IEEE Conference on Secure and Trustworthy Machine Learning. https://doi.org/10.1109/satml54575.2023.00036.

Joshi, Shalmali, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019. “Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems.” https://arxiv.org/abs/1907.09615.

Mothilal, Ramaravind K, Amit Sharma, and Chenhao Tan. 2020. “Explaining Machine Learning Classifiers Through Diverse Counterfactual Explanations.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–17. https://doi.org/10.1145/3351095.3372850.

+plot(ce)

We generally make an effort to follow the literature as closely as possible when implementing off-the-shelf generators.

References

Altmeyer, Patrick, Giovan Angela, Aleksander Buszydlik, Karol Dobiczek, Arie van Deursen, and Cynthia Liem. 2023. “Endogenous Macrodynamics in Algorithmic Recourse.” In First IEEE Conference on Secure and Trustworthy Machine Learning. https://doi.org/10.1109/satml54575.2023.00036.

Joshi, Shalmali, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019. “Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems.” https://arxiv.org/abs/1907.09615.

Mothilal, Ramaravind K, Amit Sharma, and Chenhao Tan. 2020. “Explaining Machine Learning Classifiers Through Diverse Counterfactual Explanations.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–17. https://doi.org/10.1145/3351095.3372850.

diff --git a/dev/tutorials/index.html b/dev/tutorials/index.html index 757e0e4b2..859038480 100644 --- a/dev/tutorials/index.html +++ b/dev/tutorials/index.html @@ -1,2 +1,2 @@ -Overview · CounterfactualExplanations.jl
GitHub

Tutorials

In this section, you will find a series of tutorials that should help you gain a basic understanding of Counterfactual Explanations and how to apply it in Julia using this package.

Tutorials are lessons that take the reader by the hand through a series of steps to complete a project of some kind. Tutorials are learning-oriented.

— Diátaxis

In other words, you come here because you are new to this topic and are looking for a first peek at the methodology and code 🫣.

+Overview · CounterfactualExplanations.jl
GitHub

Tutorials

In this section, you will find a series of tutorials that should help you gain a basic understanding of Counterfactual Explanations and how to apply it in Julia using this package.

Tutorials are lessons that take the reader by the hand through a series of steps to complete a project of some kind. Tutorials are learning-oriented.

— Diátaxis

In other words, you come here because you are new to this topic and are looking for a first peek at the methodology and code 🫣.

diff --git a/dev/tutorials/model_catalogue/index.html b/dev/tutorials/model_catalogue/index.html index 2c725a7bf..8670c6a86 100644 --- a/dev/tutorials/model_catalogue/index.html +++ b/dev/tutorials/model_catalogue/index.html @@ -22,4 +22,4 @@ n_hidden = 32, dropout = true )

The model_params can be supplied to the familiar API call:

M = fit_model(train_data, :MLP; model_params...)
CounterfactualExplanations.Models.Model(Chain(Dense(784 => 32, relu), Dropout(0.25, active=false), Dense(32 => 10)), :classification_multi, Chain(Dense(784 => 32, relu), Dropout(0.25, active=false), Dense(32 => 10)), MLP())

The model performance on our test set can be evaluated as follows:

model_evaluation(M, test_data)
1-element Vector{Float64}:
- 0.9185

Finally, let’s restore the default training parameters:

CounterfactualExplanations.reset!(flux_training_params)

References

Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. 2017. “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.” Advances in Neural Information Processing Systems 30.

LeCun, Yann. 1998. “The MNIST Database of Handwritten Digits.” http://yann.lecun.com/exdb/mnist/.

+ 0.9185

Finally, let’s restore the default training parameters:

CounterfactualExplanations.reset!(flux_training_params)

References

Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. 2017. “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.” Advances in Neural Information Processing Systems 30.

LeCun, Yann. 1998. “The MNIST Database of Handwritten Digits.” http://yann.lecun.com/exdb/mnist/.

diff --git a/dev/tutorials/models/index.html b/dev/tutorials/models/index.html index 373abf0c9..d3c4be095 100644 --- a/dev/tutorials/models/index.html +++ b/dev/tutorials/models/index.html @@ -41,4 +41,4 @@ Epoch 80 avg_loss(data) = 0.011847609f0 Epoch 100 -avg_loss(data) = 0.007242911f0

To prepare the fitted model for use with our package, we need to wrap it inside a container. For plain-vanilla models trained in Flux.jl, the corresponding constructor is called MLP. There is also a separate constructor called DeepEnsemble, which applies to Deep Ensembles. Deep Ensembles are a popular approach to approximate Bayesian Deep Learning and have been shown to generate good predictive uncertainty estimates (Lakshminarayanan, Pritzel, and Blundell 2017).

The appropriate API call to wrap our simple network in a container follows below:

M = MLP(nn)
CounterfactualExplanations.Models.Model(Chain(Dense(2 => 32, relu), Dropout(0.1, active=false), Dense(32 => 2)), :classification_binary, Chain(Dense(2 => 32, relu), Dropout(0.1, active=false), Dense(32 => 2)), MLP())

The likelihood function of the output variable is automatically inferred from the data. The generic plot() method can be called on the model and data to visualise the results:

plot(M, counterfactual_data)

Our model M is now ready for use with the package.

References

Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. 2017. “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.” Advances in Neural Information Processing Systems 30.

+avg_loss(data) = 0.007242911f0

To prepare the fitted model for use with our package, we need to wrap it inside a container. For plain-vanilla models trained in Flux.jl, the corresponding constructor is called MLP. There is also a separate constructor called DeepEnsemble, which applies to Deep Ensembles. Deep Ensembles are a popular approach to approximate Bayesian Deep Learning and have been shown to generate good predictive uncertainty estimates (Lakshminarayanan, Pritzel, and Blundell 2017).

The appropriate API call to wrap our simple network in a container follows below:

M = MLP(nn)
CounterfactualExplanations.Models.Model(Chain(Dense(2 => 32, relu), Dropout(0.1, active=false), Dense(32 => 2)), :classification_binary, Chain(Dense(2 => 32, relu), Dropout(0.1, active=false), Dense(32 => 2)), MLP())

The likelihood function of the output variable is automatically inferred from the data. The generic plot() method can be called on the model and data to visualise the results:

plot(M, counterfactual_data)

Our model M is now ready for use with the package.

References

Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. 2017. “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.” Advances in Neural Information Processing Systems 30.

diff --git a/dev/tutorials/parallelization/index.html b/dev/tutorials/parallelization/index.html index 53c26a57c..3bb5bbc6a 100644 --- a/dev/tutorials/parallelization/index.html +++ b/dev/tutorials/parallelization/index.html @@ -219,4 +219,4 @@ bmk = benchmark(counterfactual_data; parallelizer=parallelizer) -MPI.Finalize()

The file can be executed from the command line as follows:

mpiexecjl --project -n 4 julia -e 'include("docs/src/srcipts/mpi.jl")'
+MPI.Finalize()

The file can be executed from the command line as follows:

mpiexecjl --project -n 4 julia -e 'include("docs/src/srcipts/mpi.jl")'
diff --git a/dev/tutorials/simple_example/index.html b/dev/tutorials/simple_example/index.html index 7c784ba76..04a3018a7 100644 --- a/dev/tutorials/simple_example/index.html +++ b/dev/tutorials/simple_example/index.html @@ -10,4 +10,4 @@ x = select_factual(counterfactual_data, chosen)

Finally, we generate and visualize the generated counterfactual:

# Search:
 generator = WachterGenerator()
 ce = generate_counterfactual(x, target, counterfactual_data, M, generator)
-plot(ce)

+plot(ce)

diff --git a/dev/tutorials/whistle_stop/index.html b/dev/tutorials/whistle_stop/index.html index b9dd49738..28a1fce6e 100644 --- a/dev/tutorials/whistle_stop/index.html +++ b/dev/tutorials/whistle_stop/index.html @@ -26,4 +26,4 @@ ) ces[key] = ce plts = [plts..., plot(ce; title=key, colorbar=false)] -end

+end