Skip to content

Commit

Permalink
Merge pull request #91 from JuliaTrustworthyAI/mljflux-update
Browse files Browse the repository at this point in the history
Mljflux update
  • Loading branch information
pat-alt authored Sep 14, 2023
2 parents b9e2498 + 4b1b87b commit 3d8dad4
Show file tree
Hide file tree
Showing 15 changed files with 1,217 additions and 1,081 deletions.
4 changes: 2 additions & 2 deletions Project.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name = "ConformalPrediction"
uuid = "98bfc277-1877-43dc-819b-a3e38c30242f"
authors = ["Patrick Altmeyer"]
version = "0.1.11"
version = "0.1.12"

[deps]
CategoricalArrays = "324d7699-5711-5eae-9e2f-1d82baa6b597"
Expand Down Expand Up @@ -29,7 +29,7 @@ ComputationalResources = "0.3"
Flux = "0.13.16, 0.14"
MLJBase = "0.20, 0.21"
MLJEnsembles = "0.3.3"
MLJFlux = "0.2.10"
MLJFlux = "0.2.10, 0.3"
MLJModelInterface = "1"
MLUtils = "0.4.2"
NaturalSort = "1"
Expand Down
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@

## 🏃 Quick Tour

> First time here? Take a quick interactive [tour](https://binder.plutojl.org/v0.19.12/open?url=https%253A%252F%252Fraw.githubusercontent.com%252Fpat-alt%252FConformalPrediction.jl%252Fmain%252Fdocs%252Fpluto%252Fintro.jl) to see what this package can do: [![Binder](https://mybinder.org/badge_logo.svg)](https://binder.plutojl.org/v0.19.12/open?url=https%253A%252F%252Fraw.githubusercontent.com%252Fpat-alt%252FConformalPrediction.jl%252Fmain%252Fdocs%252Fpluto%252Fintro.jl)
> First time here? Take a quick interactive [tour](https://juliahub.com/ui/Notebooks/juliahub/Tutorials/ConformalPrediction.jl) to see what this package can do right on [JuliaHub](https://juliahub.com/ui/Notebooks/juliahub/Tutorials/ConformalPrediction.jl) (To run the notebook, hit login and then edit).
The button takes you to a [`Pluto.jl`](https://github.com/fonsp/Pluto.jl) 🎈 notebook hosted on [binder](https://mybinder.org/). In my own experience, this may take some time to load, certainly long enough to get yourself a hot beverage ☕. Alternatively, you can run the notebook locally or skip the tour for now and read on below.
This [`Pluto.jl`](https://github.com/fonsp/Pluto.jl) 🎈 notebook won the 2nd Price in the [JuliaCon 2023 Notebook Competition](https://info.juliahub.com/pluto-notebook-winner-23).

### Local Tour

Expand Down Expand Up @@ -111,11 +111,11 @@ ŷ[1:show_first]
```

5-element Vector{Tuple{Float64, Float64}}:
(0.0458889297242715, 1.9182762960257687)
(-1.9174452847238976, -0.04505791842240037)
(-1.2544275358451678, 0.6179598304563294)
(-0.2818835218505735, 1.5905038444509236)
(0.01299565032151917, 1.8853830166230163)
(-0.04087262272113379, 1.8635644669554758)
(0.04647464096907805, 1.9509117306456876)
(-0.24248802236397216, 1.6619490673126376)
(-0.07841928163933476, 1.8260178080372749)
(-0.02268628324126465, 1.881750806435345)

For simple models like this one, we can call a custom `Plots` recipe on our instance, fit result and data to generate the chart below:

Expand All @@ -139,20 +139,20 @@ println("SSC: $(round(_eval.measurement[2], digits=3))")
```

PerformanceEvaluation object with these fields:
measure, operation, measurement, per_fold,
model, measure, operation, measurement, per_fold,
per_observation, fitted_params_per_fold,
report_per_fold, train_test_rows
report_per_fold, train_test_rows, resampling, repeats
Extract:
┌──────────────────────────────────────────────┬───────────┬─────────────┬──────
│ measure │ operation │ measurement │ 1.9 ⋯
├──────────────────────────────────────────────┼───────────┼─────────────┼──────
│ ConformalPrediction.emp_coverage │ predict │ 0.948 │ 0.0 ⋯
│ ConformalPrediction.size_stratified_coverage │ predict │ 0.948 │ 0.0 ⋯
│ ConformalPrediction.emp_coverage │ predict │ 0.953 │ 0.0 ⋯
│ ConformalPrediction.size_stratified_coverage │ predict │ 0.953 │ 0.0 ⋯
└──────────────────────────────────────────────┴───────────┴─────────────┴──────
2 columns omitted

Empirical coverage: 0.948
SSC: 0.948
Empirical coverage: 0.953
SSC: 0.953

## 📚 Read on

Expand Down Expand Up @@ -240,14 +240,14 @@ Contributions are welcome! A good place to start is the [list](https://github.co

## 🙏 Thanks

To build this package I have read and re-read both Angelopoulos and Bates (2021) and Barber et al. (2021). The Awesome Conformal Prediction [repository](https://github.com/valeman/awesome-conformal-prediction) (Manokhin, n.d.) has also been a fantastic place to get started. Thanks also to [@aangelopoulos](https://github.com/aangelopoulos), [@valeman](https://github.com/valeman) and others for actively contributing to discussions on here. Quite a few people have also recently started using and contributing to the package for which I am very grateful. Finally, many thanks to Anthony Blaom ([@ablaom](https://github.com/ablaom)) for many helpful discussions about how to interface this package to `MLJ.jl`.
To build this package I have read and re-read both Angelopoulos and Bates (2021) and Barber et al. (2021). The Awesome Conformal Prediction [repository](https://github.com/valeman/awesome-conformal-prediction) (Manokhin 2022) has also been a fantastic place to get started. Thanks also to [@aangelopoulos](https://github.com/aangelopoulos), [@valeman](https://github.com/valeman) and others for actively contributing to discussions on here. Quite a few people have also recently started using and contributing to the package for which I am very grateful. Finally, many thanks to Anthony Blaom ([@ablaom](https://github.com/ablaom)) for many helpful discussions about how to interface this package to `MLJ.jl`.

## 🎓 References

Angelopoulos, Anastasios N., and Stephen Bates. 2021. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” <https://arxiv.org/abs/2107.07511>.
Angelopoulos, Anastasios N, and Stephen Bates. 2021. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” *arXiv Preprint arXiv:2107.07511*.

Barber, Rina Foygel, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. “Predictive Inference with the Jackknife+.” *The Annals of Statistics* 49 (1): 486–507. <https://doi.org/10.1214/20-AOS1965>.

Blaom, Anthony D., Franz Kiraly, Thibaut Lienart, Yiannis Simillides, Diego Arenas, and Sebastian J. Vollmer. 2020. “MLJ: A Julia Package for Composable Machine Learning.” *Journal of Open Source Software* 5 (55): 2704. <https://doi.org/10.21105/joss.02704>.

Manokhin, Valery. n.d. “Awesome Conformal Prediction.”
Manokhin, Valery. 2022. “Awesome Conformal Prediction.” Zenodo. <https://doi.org/10.5281/zenodo.6467205>.
68 changes: 34 additions & 34 deletions README_files/figure-commonmark/cell-12-output-1.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

2 comments on commit 3d8dad4

@pat-alt
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JuliaRegistrator
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Registration pull request created: JuliaRegistries/General/91413

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v0.1.12 -m "<description of version>" 3d8dad454282f82259c3bdb81a50ac79a5d645c3
git push origin v0.1.12

Please sign in to comment.