diff --git a/Project.toml b/Project.toml index 0357b2d..1e791c4 100644 --- a/Project.toml +++ b/Project.toml @@ -8,6 +8,7 @@ MLJ = "add582a8-e3ab-11e8-2d5e-e98b27df1bc7" MLJBase = "a7f614a8-145f-11e9-1d2a-a57a1082229d" MLJModelInterface = "e80e1ace-859a-464e-9ed9-23947d8ae3ea" Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +Term = "22787eb5-b846-44ae-b979-8e399b8463ab" [compat] MLJ = "0.18" diff --git a/_freeze/docs/src/classification/execute-results/md.json b/_freeze/docs/src/classification/execute-results/md.json new file mode 100644 index 0000000..938a281 --- /dev/null +++ b/_freeze/docs/src/classification/execute-results/md.json @@ -0,0 +1,10 @@ +{ + "hash": "a5e7b7c00cea9c148884a97a62ddf49b", + "result": { + "markdown": "\n\n::: {.cell execution_count=2}\n``` {.julia .cell-code}\nusing MLJ\nX, y = MLJ.make_blobs(1000, 2; centers=3, cluster_std=1.0)\ntrain, test = partition(eachindex(y), 0.4, 0.4, shuffle=true)\n```\n:::\n\n\n::: {.cell execution_count=3}\n``` {.julia .cell-code}\nEvoTreeClassifier = @load EvoTreeClassifier pkg=EvoTrees\nmodel = EvoTreeClassifier() \n```\n:::\n\n\n::: {.cell execution_count=4}\n``` {.julia .cell-code}\nusing ConformalPrediction\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach, rows=train)\n```\n:::\n\n\n::: {.cell execution_count=5}\n``` {.julia .cell-code}\nrows = rand(test, 10)\nXtest = selectrows(X, rows)\nytest = y[rows]\npredict(mach, Xtest)\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n```\n╭───────────────────────────────────────────────────────────────────╮\n│ │\n│ (1) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │\n│ (2) UnivariateFinite{Multiclass {#90CAF9}3} (3=>0.82{/#90CAF9}) │\n│ (3) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │\n│ (4) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │\n│ (5) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │\n│ (6) UnivariateFinite{Multiclass {#90CAF9}3} (3=>0.82{/#90CAF9}) │\n│ (7) UnivariateFinite{Multiclass {#90CAF9}3} (3=>0.82{/#90CAF9}) │\n│ (8) UnivariateFinite{Multiclass {#90CAF9}3} (2=>0.82{/#90CAF9}) │\n│ (9) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │\n│ (10) UnivariateFinite{Multiclass {#90CAF9}3} (3=>0.82{/#90CAF9}) │\n│ │\n│ │\n╰────────────────────────────────────────────────────── 10 items ───╯\n```\n:::\n:::\n\n\n", + "supporting": [ + "classification_files" + ], + "filters": [] + } +} \ No newline at end of file diff --git a/_freeze/docs/src/index/execute-results/md.json b/_freeze/docs/src/index/execute-results/md.json index c7c0df6..ab72e72 100644 --- a/_freeze/docs/src/index/execute-results/md.json +++ b/_freeze/docs/src/index/execute-results/md.json @@ -1,7 +1,7 @@ { "hash": "c56dcfed5fce5fece3f8dd90b08af0bd", "result": { - "markdown": "```@meta\nCurrentModule = ConformalPrediction\n```\n\n# ConformalPrediction\n\nDocumentation for [ConformalPrediction.jl](https://github.com/pat-alt/ConformalPrediction.jl).\n\n```@meta\nCurrentModule = ConformalPrediction\n```\n\n\n\n`ConformalPrediction.jl` is a package for Uncertainty Quantification (UQ) through Conformal Prediction (CP) in Julia. It is designed to work with supervised models trained in [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/). Conformal Prediction is distribution-free, easy-to-understand, easy-to-use and model-agnostic. \n\n## Installation 🚩\n\nYou can install the first stable release from the general registry:\n\n```julia\nusing Pkg\nPkg.add(\"ConformalPrediction\")\n```\n\nThe development version can be installed as follows:\n\n```julia\nusing Pkg\nPkg.add(url=\"https://github.com/pat-alt/ConformalPrediction.jl\")\n```\n\n## Status 🔁\n\nThis package is in its very early stages of development and therefore still subject to changes to the core architecture. The following approaches have been implemented in the development version:\n\n**Regression**:\n\n- Inductive \n- Naive Transductive \n- Jackknife \n- Jackknife+ \n- Jackknife-minmax\n- CV+\n- CV-minmax\n\n**Classification**:\n\n- Inductive (LABEL [@sadinle2019least])\n- Adaptive Inductive\n\nI have only tested it for a few of the supervised models offered by [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/).\n\n## Usage Example 🔍\n\nTo illustrate the intended use of the package, let's have a quick look at a simple regression problem. Using [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) we first generate some synthetic data and then determine indices for our training, calibration and test data:\n\n::: {.cell execution_count=2}\n``` {.julia .cell-code}\nusing MLJ\nX, y = MLJ.make_regression(1000, 2)\ntrain, test = partition(eachindex(y), 0.4, 0.4)\n```\n:::\n\n\nWe then import a decision tree ([DecisionTree](https://github.com/Evovest/DecisionTree.jl)) following the standard [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) procedure.\n\n::: {.cell execution_count=3}\n``` {.julia .cell-code}\nDecisionTreeRegressor = @load DecisionTreeRegressor pkg=DecisionTree\nmodel = DecisionTreeRegressor() \n```\n:::\n\n\nTo turn our conventional model into a conformal model, we just need to declare it as such by using `conformal_model` wrapper function. The generated conformal model instance can wrapped in data to create a *machine*. Finally, we proceed by fitting the machine on training data using the generic `fit!` method:\n\n::: {.cell execution_count=4}\n``` {.julia .cell-code}\nusing ConformalPrediction\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach, rows=train)\n```\n:::\n\n\nPredictions can then be computed using the generic `predict` method. The code below produces predictions for the first `n` samples. Each tuple contains the lower and upper bound for the prediction interval.\n\n::: {.cell execution_count=5}\n``` {.julia .cell-code}\nn = 10\nXtest = selectrows(X, first(test,n))\nytest = y[first(test,n)]\npredict(mach, Xtest)\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n```\n╭──────────────────────────────────────────────────────────────────╮\n│ │\n│ (1) ([-0.16035036780321532], [1.4939904924997824]) │\n│ (2) ([-1.086589388667894], [0.5677514716351038]) │\n│ (3) ([-1.086589388667894], [0.5677514716351038]) │\n│ (4) ([-1.6661164684544767], [-0.011775608151479156]) │\n│ (5) ([-3.0116018507211617], [-1.3572609904181638]) │\n│ (6) ([0.5337083913933376], [2.1880492516963352]) │\n│ (7) ([-1.2219266921060266], [0.43241416819697115]) │\n│ (8) ([-1.6867950029289869], [-0.032454142625989335]) │\n│ (9) ([-2.0599181285783263], [-0.4055772682753287]) │\n│ (10) ([-0.06499897951385392], [1.5893418807891437]) │\n│ │\n│ │\n╰───────────────────────────────────────────────────── 10 items ───╯\n```\n:::\n:::\n\n\n## Contribute 🛠\n\nContributions are welcome! Please follow the [SciML ColPrac guide](https://github.com/SciML/ColPrac).\n\n## References 🎓\n\n", + "markdown": "```@meta\nCurrentModule = ConformalPrediction\n```\n\n# ConformalPrediction\n\nDocumentation for [ConformalPrediction.jl](https://github.com/pat-alt/ConformalPrediction.jl).\n\n\n\n`ConformalPrediction.jl` is a package for Uncertainty Quantification (UQ) through Conformal Prediction (CP) in Julia. It is designed to work with supervised models trained in [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/). Conformal Prediction is distribution-free, easy-to-understand, easy-to-use and model-agnostic. \n\n## Installation 🚩\n\nYou can install the first stable release from the general registry:\n\n```{.julia}\nusing Pkg\nPkg.add(\"ConformalPrediction\")\n```\n\nThe development version can be installed as follows:\n\n```{.julia}\nusing Pkg\nPkg.add(url=\"https://github.com/pat-alt/ConformalPrediction.jl\")\n```\n\n## Status 🔁\n\nThis package is in its very early stages of development and therefore still subject to changes to the core architecture. The following approaches have been implemented in the development version:\n\n**Regression**:\n\n- Inductive \n- Naive Transductive \n- Jackknife \n- Jackknife+ \n- Jackknife-minmax\n- CV+\n- CV-minmax\n\n**Classification**:\n\n- Inductive (LABEL [@sadinle2019least])\n- Adaptive Inductive\n\nI have only tested it for a few of the supervised models offered by [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/).\n\n## Usage Example 🔍\n\nTo illustrate the intended use of the package, let's have a quick look at a simple regression problem. Using [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) we first generate some synthetic data and then determine indices for our training, calibration and test data:\n\n::: {.cell execution_count=2}\n``` {.julia .cell-code}\nusing MLJ\nX, y = MLJ.make_regression(1000, 2)\ntrain, test = partition(eachindex(y), 0.4, 0.4)\n```\n:::\n\n\nWe then import a decision tree ([`EvoTrees.jl`](https://github.com/Evovest/EvoTrees.jl)) following the standard [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) procedure.\n\n::: {.cell execution_count=3}\n``` {.julia .cell-code}\nEvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees\nmodel = EvoTreeRegressor() \n```\n:::\n\n\nTo turn our conventional model into a conformal model, we just need to declare it as such by using `conformal_model` wrapper function. The generated conformal model instance can wrapped in data to create a *machine*. Finally, we proceed by fitting the machine on training data using the generic `fit!` method:\n\n::: {.cell execution_count=4}\n``` {.julia .cell-code}\nusing ConformalPrediction\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach, rows=train)\n```\n:::\n\n\nPredictions can then be computed using the generic `predict` method. The code below produces predictions for the first `n` samples. Each tuple contains the lower and upper bound for the prediction interval.\n\n::: {.cell execution_count=5}\n``` {.julia .cell-code}\nn = 10\nXtest = selectrows(X, first(test,n))\nytest = y[first(test,n)]\npredict(mach, Xtest)\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n```\n╭────────────────────────────────────────────────────────────────╮\n│ │\n│ (1) ([-5.4338268970328265], [0.25968688121283146]) │\n│ (2) ([-3.327102611742035], [2.3664111665036227]) │\n│ (3) ([-6.487880423821674], [-0.7943666455760168]) │\n│ (4) ([1.5074870726446568], [7.201000850890314]) │\n│ (5) ([-6.668012325746515], [-0.9744985475008576]) │\n│ (6) ([-2.36920384417906], [3.3243099340665974]) │\n│ (7) ([-0.4783861002145251], [5.215127678031132]) │\n│ (8) ([-5.554310900530298], [0.1392028777153591]) │\n│ (9) ([-1.4607932119178935], [4.232720566327764]) │\n│ (10) ([-5.190158387746367], [0.5033553904992907]) │\n│ │\n│ │\n╰─────────────────────────────────────────────────── 10 items ───╯\n```\n:::\n:::\n\n\n## Contribute 🛠\n\nContributions are welcome! Please follow the [SciML ColPrac guide](https://github.com/SciML/ColPrac).\n\n## References 🎓\n\n", "supporting": [ "index_files" ], diff --git a/_freeze/docs/src/intro/execute-results/md.json b/_freeze/docs/src/intro/execute-results/md.json new file mode 100644 index 0000000..960f41c --- /dev/null +++ b/_freeze/docs/src/intro/execute-results/md.json @@ -0,0 +1,10 @@ +{ + "hash": "1b0ee553524705afa5795d1e05898476", + "result": { + "markdown": "\n\n`ConformalPrediction.jl` is a package for Uncertainty Quantification (UQ) through Conformal Prediction (CP) in Julia. It is designed to work with supervised models trained in [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/). Conformal Prediction is distribution-free, easy-to-understand, easy-to-use and model-agnostic. \n\n## Installation 🚩\n\nYou can install the first stable release from the general registry:\n\n```{.julia}\nusing Pkg\nPkg.add(\"ConformalPrediction\")\n```\n\nThe development version can be installed as follows:\n\n```{.julia}\nusing Pkg\nPkg.add(url=\"https://github.com/pat-alt/ConformalPrediction.jl\")\n```\n\n## Status 🔁\n\nThis package is in its very early stages of development and therefore still subject to changes to the core architecture. The following approaches have been implemented in the development version:\n\n**Regression**:\n\n- Inductive \n- Naive Transductive \n- Jackknife \n- Jackknife+ \n- Jackknife-minmax\n- CV+\n- CV-minmax\n\n**Classification**:\n\n- Inductive (LABEL [@sadinle2019least])\n- Adaptive Inductive\n\nI have only tested it for a few of the supervised models offered by [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/).\n\n## Usage Example 🔍\n\nTo illustrate the intended use of the package, let's have a quick look at a simple regression problem. Using [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) we first generate some synthetic data and then determine indices for our training, calibration and test data:\n\n::: {.cell execution_count=2}\n``` {.julia .cell-code}\nusing MLJ\nX, y = MLJ.make_regression(1000, 2)\ntrain, test = partition(eachindex(y), 0.4, 0.4)\n```\n:::\n\n\nWe then import a decision tree ([`EvoTrees.jl`](https://github.com/Evovest/EvoTrees.jl)) following the standard [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) procedure.\n\n::: {.cell execution_count=3}\n``` {.julia .cell-code}\nEvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees\nmodel = EvoTreeRegressor() \n```\n:::\n\n\nTo turn our conventional model into a conformal model, we just need to declare it as such by using `conformal_model` wrapper function. The generated conformal model instance can wrapped in data to create a *machine*. Finally, we proceed by fitting the machine on training data using the generic `fit!` method:\n\n::: {.cell execution_count=4}\n``` {.julia .cell-code}\nusing ConformalPrediction\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach, rows=train)\n```\n:::\n\n\nPredictions can then be computed using the generic `predict` method. The code below produces predictions for the first `n` samples. Each tuple contains the lower and upper bound for the prediction interval.\n\n::: {.cell execution_count=5}\n``` {.julia .cell-code}\nn = 10\nXtest = selectrows(X, first(test,n))\nytest = y[first(test,n)]\npredict(mach, Xtest)\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n```\n╭─────────────────────────────────────────────────────────────────╮\n│ │\n│ (1) ([-0.20063113789390163], [1.323655530145934]) │\n│ (2) ([-0.061147489871723804], [1.4631391781681118]) │\n│ (3) ([-1.4486105066363675], [0.07567616140346822]) │\n│ (4) ([-0.7160881365817455], [0.8081985314580902]) │\n│ (5) ([-1.7173644161988695], [-0.19307774815903367]) │\n│ (6) ([-1.2158809697881832], [0.3084056982516525]) │\n│ (7) ([-1.7173644161988695], [-0.19307774815903367]) │\n│ (8) ([0.26510754559144056], [1.7893942136312764]) │\n│ (9) ([-0.8716996456392521], [0.6525870224005836]) │\n│ (10) ([0.43084861624955606], [1.9551352842893919]) │\n│ │\n│ │\n╰──────────────────────────────────────────────────── 10 items ───╯\n```\n:::\n:::\n\n\n## Contribute 🛠\n\nContributions are welcome! Please follow the [SciML ColPrac guide](https://github.com/SciML/ColPrac).\n\n## References 🎓\n\n", + "supporting": [ + "intro_files" + ], + "filters": [] + } +} \ No newline at end of file diff --git a/_quarto.yml b/_quarto.yml index b749e6f..3ce739e 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -12,5 +12,5 @@ execute: echo: true eval: true output: false - +jupyter: julia-1.8 diff --git a/docs/Manifest.toml b/docs/Manifest.toml index 1038ce0..fb9607e 100644 --- a/docs/Manifest.toml +++ b/docs/Manifest.toml @@ -2,7 +2,7 @@ julia_version = "1.8.1" manifest_format = "2.0" -project_hash = "9f3db68ff535b141bf92e2e3a6ac483ef835a792" +project_hash = "ae240e8e0ebf5559f152baffe6c05f54b653e10d" [[deps.ANSIColoredPrinters]] git-tree-sha1 = "574baf8110975760d391c710b6341da1afa48d8c" @@ -22,9 +22,9 @@ uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c" version = "1.2.1" [[deps.AbstractTrees]] -git-tree-sha1 = "5c0b629df8a5566a06f5fef5100b53ea56e465a0" +git-tree-sha1 = "52b3b436f8f73133d7bc3a6c71ee7ed6ab2ab754" uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" -version = "0.4.2" +version = "0.4.3" [[deps.Adapt]] deps = ["LinearAlgebra"] @@ -44,9 +44,9 @@ version = "6.0.23" [[deps.ArrayInterfaceCore]] deps = ["LinearAlgebra", "SparseArrays", "SuiteSparse"] -git-tree-sha1 = "e9f7992287edfc27b3cbe0046c544bace004ca5b" +git-tree-sha1 = "e6cba4aadba7e8a7574ab2ba2fcfb307b4c4b02a" uuid = "30b0a656-2188-435a-8636-2ec0e6a096e2" -version = "0.1.22" +version = "0.1.23" [[deps.ArrayInterfaceOffsetArrays]] deps = ["ArrayInterface", "OffsetArrays", "Static"] @@ -69,6 +69,12 @@ version = "0.1.3" [[deps.Artifacts]] uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" +[[deps.AxisAlgorithms]] +deps = ["LinearAlgebra", "Random", "SparseArrays", "WoodburyMatrices"] +git-tree-sha1 = "66771c8d21c8ff5e3a93379480a2307ac36863f7" +uuid = "13072b0f-2c55-5437-9ae7-d433b7a33950" +version = "1.0.1" + [[deps.BFloat16s]] deps = ["LinearAlgebra", "Printf", "Random", "Test"] git-tree-sha1 = "a598ecb0d717092b5539dbbe890c98bac842b072" @@ -220,7 +226,7 @@ version = "0.3.2" deps = ["MLJ", "MLJBase", "MLJModelInterface", "Statistics"] path = ".." uuid = "98bfc277-1877-43dc-819b-a3e38c30242f" -version = "0.1.0" +version = "0.1.2" [[deps.ConstructionBase]] deps = ["LinearAlgebra"] @@ -294,9 +300,9 @@ version = "1.1.0" [[deps.DiffRules]] deps = ["IrrationalConstants", "LogExpFunctions", "NaNMath", "Random", "SpecialFunctions"] -git-tree-sha1 = "992a23afdb109d0d2f8802a30cf5ae4b1fe7ea68" +git-tree-sha1 = "8b7a4d23e22f5d44883671da70865ca98f2ebf9d" uuid = "b552c78f-8df3-52c6-915a-8e097449b14b" -version = "1.11.1" +version = "1.12.0" [[deps.Distances]] deps = ["LinearAlgebra", "SparseArrays", "Statistics", "StatsAPI"] @@ -316,9 +322,9 @@ version = "0.25.76" [[deps.DocStringExtensions]] deps = ["LibGit2"] -git-tree-sha1 = "5158c2b41018c5f7eb1470d558127ac274eca0c9" +git-tree-sha1 = "c36550cb29cbe373e95b3f40486b9a4148f89ffd" uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae" -version = "0.9.1" +version = "0.9.2" [[deps.Documenter]] deps = ["ANSIColoredPrinters", "Base64", "Dates", "DocStringExtensions", "IOCapture", "InteractiveUtils", "JSON", "LibGit2", "Logging", "Markdown", "REPL", "Test", "Unicode"] @@ -351,9 +357,9 @@ version = "0.3.0" [[deps.EvoTrees]] deps = ["BSON", "CUDA", "CategoricalArrays", "Distributions", "LoopVectorization", "MLJModelInterface", "NetworkLayout", "Random", "RecipesBase", "Statistics", "StatsBase", "Tables"] -git-tree-sha1 = "2e01454a464cdb4ad82cc9b824ef762254c0e2ca" +git-tree-sha1 = "966e236ded10551a44b6e25ce4bbea4c12be1557" uuid = "f6006082-12f8-11e9-0c9c-0d5d367ab1e5" -version = "0.12.0" +version = "0.12.4" [[deps.Expat_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] @@ -383,6 +389,18 @@ git-tree-sha1 = "74faea50c1d007c85837327f6775bea60b5492dd" uuid = "b22a6f82-2f65-5046-a5b2-351ab43fb4e5" version = "4.4.2+2" +[[deps.FFTW]] +deps = ["AbstractFFTs", "FFTW_jll", "LinearAlgebra", "MKL_jll", "Preferences", "Reexport"] +git-tree-sha1 = "90630efff0894f8142308e334473eba54c433549" +uuid = "7a1cc6ca-52ef-59f5-83cd-3a7055c09341" +version = "1.5.0" + +[[deps.FFTW_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "c6033cc3892d0ef5bb9cd29b7f2f0331ea5184ea" +uuid = "f5851436-0d7a-5f13-b9de-f02708fd171a" +version = "3.3.10+0" + [[deps.FileIO]] deps = ["Pkg", "Requires", "UUIDs"] git-tree-sha1 = "7be5f99f7d15578798f338f5433b6c432ea8037b" @@ -398,6 +416,12 @@ git-tree-sha1 = "802bfc139833d2ba893dd9e62ba1767c88d708ae" uuid = "1a297f60-69ca-5386-bcde-b61e274b549b" version = "0.13.5" +[[deps.FiniteDiff]] +deps = ["ArrayInterfaceCore", "LinearAlgebra", "Requires", "Setfield", "SparseArrays", "StaticArrays"] +git-tree-sha1 = "5a2cff9b6b77b33b89f3d97a4d367747adce647e" +uuid = "6a86dc24-6348-571c-b903-95158fe2bd41" +version = "2.15.0" + [[deps.FixedPointNumbers]] deps = ["Statistics"] git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc" @@ -464,9 +488,9 @@ version = "0.1.2" [[deps.GPUCompiler]] deps = ["ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "TimerOutputs", "UUIDs"] -git-tree-sha1 = "ebb892e1df16040a845e1d11087e4fbfe10323a8" +git-tree-sha1 = "323949b0bbdf38c93d2ea1f7d3e68ff163c3f081" uuid = "61eb1bfa-7361-4325-ad38-22787b887f55" -version = "0.16.4" +version = "0.16.5" [[deps.GR]] deps = ["Base64", "DelimitedFiles", "GR_jll", "HTTP", "JSON", "Libdl", "LinearAlgebra", "Pkg", "Preferences", "Printf", "Random", "Serialization", "Sockets", "Test", "UUIDs"] @@ -515,11 +539,23 @@ git-tree-sha1 = "53bb909d1151e57e2484c3d1b53e19552b887fb2" uuid = "42e2da0e-8278-4e71-bc24-59509adca0fe" version = "1.0.2" +[[deps.HDF5]] +deps = ["Compat", "HDF5_jll", "Libdl", "Mmap", "Random", "Requires"] +git-tree-sha1 = "19effd6b5af759c8aaeb9c77f89422d3f975ab65" +uuid = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f" +version = "0.16.12" + +[[deps.HDF5_jll]] +deps = ["Artifacts", "JLLWrappers", "LibCURL_jll", "Libdl", "OpenSSL_jll", "Pkg", "Zlib_jll"] +git-tree-sha1 = "4cc2bb72df6ff40b055295fdef6d92955f9dede8" +uuid = "0234f1f7-429e-5d53-9886-15a909be8d59" +version = "1.12.2+2" + [[deps.HTTP]] deps = ["Base64", "CodecZlib", "Dates", "IniFile", "Logging", "LoggingExtras", "MbedTLS", "NetworkOptions", "OpenSSL", "Random", "SimpleBufferStream", "Sockets", "URIs", "UUIDs"] -git-tree-sha1 = "e8c58d5f03b9d9eb9ed7067a2f34c7c371ab130b" +git-tree-sha1 = "a97d47758e933cd5fe5ea181d178936a9fc60427" uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3" -version = "1.4.1" +version = "1.5.1" [[deps.HarfBuzz_jll]] deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"] @@ -555,10 +591,22 @@ git-tree-sha1 = "f550e6e32074c939295eb5ea6de31849ac2c9625" uuid = "83e8ac13-25f8-5344-8a64-a9f2b223428f" version = "0.5.1" +[[deps.IntelOpenMP_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "d979e54b71da82f3a65b62553da4fc3d18c9004c" +uuid = "1d5cc7b8-4909-519e-a0f8-d0f5ad9712d0" +version = "2018.0.3+2" + [[deps.InteractiveUtils]] deps = ["Markdown"] uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" +[[deps.Interpolations]] +deps = ["Adapt", "AxisAlgorithms", "ChainRulesCore", "LinearAlgebra", "OffsetArrays", "Random", "Ratios", "Requires", "SharedArrays", "SparseArrays", "StaticArrays", "WoodburyMatrices"] +git-tree-sha1 = "842dd89a6cb75e02e85fdd75c760cdc43f5d6863" +uuid = "a98d9a8b-a2ab-59e6-89dd-64a1c18fca59" +version = "0.14.6" + [[deps.InverseFunctions]] deps = ["Test"] git-tree-sha1 = "49510dfcb407e572524ba94aeae2fced1f3feb0f" @@ -586,6 +634,12 @@ git-tree-sha1 = "d7df9a6fdd82a8cfdfe93a94fcce35515be634da" uuid = "b3c1a2ee-3fec-4384-bf48-272ea71de57c" version = "0.5.3" +[[deps.IterativeSolvers]] +deps = ["LinearAlgebra", "Printf", "Random", "RecipesBase", "SparseArrays"] +git-tree-sha1 = "1169632f425f79429f245113b775a0e3d121457c" +uuid = "42fd0dbc-a981-5370-80f2-aaf504508153" +version = "0.9.2" + [[deps.IteratorInterfaceExtensions]] git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856" uuid = "82899510-4779-5014-852e-03e436cf321d" @@ -615,6 +669,12 @@ git-tree-sha1 = "b53380851c6e6664204efb2e62cd24fa5c47e4ba" uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" version = "2.1.2+0" +[[deps.KernelDensity]] +deps = ["Distributions", "DocStringExtensions", "FFTW", "Interpolations", "StatsBase"] +git-tree-sha1 = "9816b296736292a80b9a3200eb7fbb57aaa3917a" +uuid = "5ab0869b-81aa-558d-bb23-cbf5423bbe9b" +version = "0.6.5" + [[deps.LAME_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "f6250b16881adf048549549fba48b1161acdac8c" @@ -664,9 +724,9 @@ version = "1.8.0" [[deps.LayoutPointers]] deps = ["ArrayInterface", "ArrayInterfaceOffsetArrays", "ArrayInterfaceStaticArrays", "LinearAlgebra", "ManualMemory", "SIMDTypes", "Static"] -git-tree-sha1 = "b67e749fb35530979839e7b4b606a97105fe4f1c" +git-tree-sha1 = "73e2e40eb02d6ccd191a8a9f8cee20db8d5df010" uuid = "10f19ff3-798f-405d-979b-55457f8fc047" -version = "0.1.10" +version = "0.1.11" [[deps.LazyArtifacts]] deps = ["Artifacts", "Pkg"] @@ -742,10 +802,22 @@ git-tree-sha1 = "7f3efec06033682db852f8b3bc3c1d2b0a0ab066" uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700" version = "2.36.0+0" +[[deps.LineSearches]] +deps = ["LinearAlgebra", "NLSolversBase", "NaNMath", "Parameters", "Printf"] +git-tree-sha1 = "7bbea35cec17305fc70a0e5b4641477dc0789d9d" +uuid = "d3d80556-e9d4-5f37-9878-2ab0fcc64255" +version = "7.2.0" + [[deps.LinearAlgebra]] deps = ["Libdl", "libblastrampoline_jll"] uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" +[[deps.LinearMaps]] +deps = ["LinearAlgebra", "SparseArrays", "Statistics"] +git-tree-sha1 = "d1b46faefb7c2f48fdec69e6f3cc34857769bc15" +uuid = "7a12625a-238d-50fd-b39a-03d52299707e" +version = "3.8.0" + [[deps.LogExpFunctions]] deps = ["ChainRulesCore", "ChangesOfVariables", "DocStringExtensions", "InverseFunctions", "IrrationalConstants", "LinearAlgebra"] git-tree-sha1 = "94d9c52ca447e23eac0c0f074effbcd38830deb5" @@ -763,9 +835,9 @@ version = "0.4.9" [[deps.LoopVectorization]] deps = ["ArrayInterface", "ArrayInterfaceCore", "ArrayInterfaceOffsetArrays", "ArrayInterfaceStaticArrays", "CPUSummary", "ChainRulesCore", "CloseOpenIntervals", "DocStringExtensions", "ForwardDiff", "HostCPUFeatures", "IfElse", "LayoutPointers", "LinearAlgebra", "OffsetArrays", "PolyesterWeave", "SIMDDualNumbers", "SIMDTypes", "SLEEFPirates", "SnoopPrecompile", "SpecialFunctions", "Static", "ThreadingUtilities", "UnPack", "VectorizationBase"] -git-tree-sha1 = "39af6a1e398a29f568dc9fe469f459ad3aacb03b" +git-tree-sha1 = "9f6030ca92d1a816e931abb657219c9fc4991a96" uuid = "bdcacae8-1622-11e9-2a5c-532679323890" -version = "0.12.133" +version = "0.12.136" [[deps.LossFunctions]] deps = ["InteractiveUtils", "Markdown", "RecipesBase"] @@ -773,6 +845,12 @@ git-tree-sha1 = "53cd63a12f06a43eef6f4aafb910ac755c122be7" uuid = "30fc2ffe-d236-52d8-8643-a9d8f7c094a7" version = "0.8.0" +[[deps.MKL_jll]] +deps = ["Artifacts", "IntelOpenMP_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"] +git-tree-sha1 = "2ce8695e1e699b68702c03402672a69f54b8aca9" +uuid = "856f044c-d86e-5d09-b602-aeab76dc8ba7" +version = "2022.2.0+0" + [[deps.MLJ]] deps = ["CategoricalArrays", "ComputationalResources", "Distributed", "Distributions", "LinearAlgebra", "MLJBase", "MLJEnsembles", "MLJIteration", "MLJModels", "MLJTuning", "OpenML", "Pkg", "ProgressMeter", "Random", "ScientificTypes", "Statistics", "StatsBase", "Tables"] git-tree-sha1 = "025706ea81e635ac530a1d3dd365af971805bf79" @@ -803,6 +881,12 @@ git-tree-sha1 = "024d0bd22bf4a5b273f626e89d742a9db95285ef" uuid = "614be32b-d00c-4edb-bd02-1eb411ab5e55" version = "0.5.0" +[[deps.MLJLinearModels]] +deps = ["DocStringExtensions", "IterativeSolvers", "LinearAlgebra", "LinearMaps", "MLJModelInterface", "Optim", "Parameters"] +git-tree-sha1 = "bfebb824a1b9a0c6d58e417e680f5e99317534e3" +uuid = "6ee0df7b-362f-4a72-a706-9e79364fb692" +version = "0.7.1" + [[deps.MLJModelInterface]] deps = ["Random", "ScientificTypesBase", "StatisticalTraits"] git-tree-sha1 = "0a36882e73833d60dac49b00d203f73acfd50b85" @@ -815,6 +899,12 @@ git-tree-sha1 = "147a8e7939601f8c37204addbbe29f2bcfb876a8" uuid = "d491faf4-2d78-11e9-2867-c94bc002c0b7" version = "0.15.14" +[[deps.MLJNaiveBayesInterface]] +deps = ["LogExpFunctions", "MLJModelInterface", "NaiveBayes"] +git-tree-sha1 = "65c8eb0f1da7380f1660201c7ca14d085602ad90" +uuid = "33e4bacb-b9e2-458e-9a13-5d9a90b235fa" +version = "0.1.6" + [[deps.MLJTuning]] deps = ["ComputationalResources", "Distributed", "Distributions", "LatinHypercubeSampling", "MLJBase", "ProgressMeter", "Random", "RecipesBase"] git-tree-sha1 = "77209966cc028c1d7730001dc32bffe17a198f29" @@ -844,9 +934,9 @@ uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" [[deps.MbedTLS]] deps = ["Dates", "MbedTLS_jll", "MozillaCACerts_jll", "Random", "Sockets"] -git-tree-sha1 = "6872f9594ff273da6d13c7c1a1545d5a8c7d0c1c" +git-tree-sha1 = "03a9b9718f5682ecb107ac9f7308991db4ce395b" uuid = "739be429-bea8-5141-9913-cc70e7f3736d" -version = "1.1.6" +version = "1.1.7" [[deps.MbedTLS_jll]] deps = ["Artifacts", "Libdl"] @@ -871,12 +961,24 @@ uuid = "a63ad114-7e13-5084-954f-fe012c677804" uuid = "14a3606d-f60d-562e-9121-12d972cd8159" version = "2022.2.1" +[[deps.NLSolversBase]] +deps = ["DiffResults", "Distributed", "FiniteDiff", "ForwardDiff"] +git-tree-sha1 = "50310f934e55e5ca3912fb941dec199b49ca9b68" +uuid = "d41bc354-129a-5804-8e4c-c37616107c6c" +version = "7.8.2" + [[deps.NaNMath]] deps = ["OpenLibm_jll"] git-tree-sha1 = "a7c3d1da1189a1c2fe843a3bfa04d18d20eb3211" uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3" version = "1.0.1" +[[deps.NaiveBayes]] +deps = ["Distributions", "HDF5", "Interpolations", "KernelDensity", "LinearAlgebra", "Random", "SparseArrays", "StatsBase"] +git-tree-sha1 = "830c601de91378e773e7286c3a3e8964d6248657" +uuid = "9bbee03b-0db5-5f46-924f-b5c9c21b8c60" +version = "0.5.4" + [[deps.NetworkLayout]] deps = ["GeometryBasics", "LinearAlgebra", "Random", "Requires", "SparseArrays"] git-tree-sha1 = "cac8fc7ba64b699c678094fa630f49b80618f625" @@ -917,9 +1019,9 @@ version = "0.3.0" [[deps.OpenSSL]] deps = ["BitFlags", "Dates", "MozillaCACerts_jll", "OpenSSL_jll", "Sockets"] -git-tree-sha1 = "ebe81469e9d7b471d7ddb611d9e147ea16de0add" +git-tree-sha1 = "3c3c4a401d267b04942545b1e964a20279587fd7" uuid = "4d8831e6-92b7-49fb-bdf8-b643e874388c" -version = "1.2.1" +version = "1.3.0" [[deps.OpenSSL_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] @@ -933,6 +1035,12 @@ git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1" uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e" version = "0.5.5+0" +[[deps.Optim]] +deps = ["Compat", "FillArrays", "ForwardDiff", "LineSearches", "LinearAlgebra", "NLSolversBase", "NaNMath", "Parameters", "PositiveFactorizations", "Printf", "SparseArrays", "StatsBase"] +git-tree-sha1 = "b9fe76d1a39807fdcf790b991981a922de0c3050" +uuid = "429524aa-4258-5aef-a3af-852621145aeb" +version = "1.7.3" + [[deps.Opus_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "51a08fb14ec28da2ec7a927c4337e4332c2a4720" @@ -997,9 +1105,9 @@ version = "1.3.1" [[deps.Plots]] deps = ["Base64", "Contour", "Dates", "Downloads", "FFMPEG", "FixedPointNumbers", "GR", "JLFzf", "JSON", "LaTeXStrings", "Latexify", "LinearAlgebra", "Measures", "NaNMath", "Pkg", "PlotThemes", "PlotUtils", "Printf", "REPL", "Random", "RecipesBase", "RecipesPipeline", "Reexport", "RelocatableFolders", "Requires", "Scratch", "Showoff", "SnoopPrecompile", "SparseArrays", "Statistics", "StatsBase", "UUIDs", "UnicodeFun", "Unzip"] -git-tree-sha1 = "524d9ff1b2f4473fef59678c06f9f77160a204b1" +git-tree-sha1 = "0a56829d264eb1bc910cf7c39ac008b5bcb5a0d9" uuid = "91a5bcdd-55d7-5caf-9e0b-520d859cae80" -version = "1.35.3" +version = "1.35.5" [[deps.PolyesterWeave]] deps = ["BitTwiddlingConvenienceFunctions", "CPUSummary", "IfElse", "Static", "ThreadingUtilities"] @@ -1013,6 +1121,12 @@ git-tree-sha1 = "a6062fe4063cdafe78f4a0a81cfffb89721b30e7" uuid = "2dfb63ee-cc39-5dd5-95bd-886bf059d720" version = "1.4.2" +[[deps.PositiveFactorizations]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "17275485f373e6673f7e7f97051f703ed5b15b20" +uuid = "85a6dd25-e78a-55b7-8502-1745935b8125" +version = "0.2.4" + [[deps.Preferences]] deps = ["TOML"] git-tree-sha1 = "47e5f437cc0e7ef2ce8406ce1e7e24d44915f88d" @@ -1048,9 +1162,9 @@ version = "5.15.3+1" [[deps.QuadGK]] deps = ["DataStructures", "LinearAlgebra"] -git-tree-sha1 = "3c009334f45dfd546a16a57960a821a1a023d241" +git-tree-sha1 = "97aa253e65b784fd13e83774cadc95b38011d734" uuid = "1fd47b50-473d-5c70-9696-f719f8f3bcdc" -version = "2.5.0" +version = "2.6.0" [[deps.REPL]] deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] @@ -1072,11 +1186,17 @@ git-tree-sha1 = "043da614cc7e95c703498a491e2c21f58a2b8111" uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143" version = "1.5.3" +[[deps.Ratios]] +deps = ["Requires"] +git-tree-sha1 = "dc84268fe0e3335a62e315a3a7cf2afa7178a734" +uuid = "c84ed2f1-dad5-54f0-aa8e-dbefe2724439" +version = "0.4.3" + [[deps.RecipesBase]] deps = ["SnoopPrecompile"] -git-tree-sha1 = "612a4d76ad98e9722c8ba387614539155a59e30c" +git-tree-sha1 = "d12e612bba40d189cead6ff857ddb67bd2e6a387" uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01" -version = "1.3.0" +version = "1.3.1" [[deps.RecipesPipeline]] deps = ["Dates", "NaNMath", "PlotUtils", "RecipesBase", "SnoopPrecompile"] @@ -1160,6 +1280,16 @@ version = "1.1.1" [[deps.Serialization]] uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" +[[deps.Setfield]] +deps = ["ConstructionBase", "Future", "MacroTools", "StaticArraysCore"] +git-tree-sha1 = "e2cc6d8c88613c05e1defb55170bf5ff211fbeac" +uuid = "efcf1570-3423-57d1-acb7-fd33fddbac46" +version = "1.1.1" + +[[deps.SharedArrays]] +deps = ["Distributed", "Mmap", "Random", "Serialization"] +uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" + [[deps.Showoff]] deps = ["Dates", "Grisu"] git-tree-sha1 = "91eddf657aca81df9ae6ceb20b959ae5653ad1de" @@ -1253,9 +1383,9 @@ version = "0.3.0" [[deps.StructArrays]] deps = ["Adapt", "DataAPI", "StaticArraysCore", "Tables"] -git-tree-sha1 = "8c6ac65ec9ab781af05b08ff305ddc727c25f680" +git-tree-sha1 = "13237798b407150a6d2e2bce5d793d7d9576e99e" uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a" -version = "0.6.12" +version = "0.6.13" [[deps.SuiteSparse]] deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"] @@ -1336,9 +1466,9 @@ version = "0.4.1" [[deps.UnicodePlots]] deps = ["ColorSchemes", "ColorTypes", "Contour", "Crayons", "Dates", "FileIO", "FreeType", "LinearAlgebra", "MarchingCubes", "NaNMath", "Printf", "Requires", "SnoopPrecompile", "SparseArrays", "StaticArrays", "StatsBase", "Unitful"] -git-tree-sha1 = "8a6dcd44129de81cc760b9d8af6fba188d3a01a6" +git-tree-sha1 = "390b2e8e5535f5beb50885d1a1059f460547d3a5" uuid = "b8865327-cd53-5732-bb35-84acbb429228" -version = "3.1.3" +version = "3.1.6" [[deps.Unitful]] deps = ["ConstructionBase", "Dates", "LinearAlgebra", "Random"] @@ -1353,9 +1483,9 @@ version = "0.2.0" [[deps.VectorizationBase]] deps = ["ArrayInterface", "CPUSummary", "HostCPUFeatures", "IfElse", "LayoutPointers", "Libdl", "LinearAlgebra", "SIMDTypes", "Static"] -git-tree-sha1 = "3bc5ea8fbf25f233c4c49c0a75f14b276d2f9a69" +git-tree-sha1 = "ba9d398034a2ba78059391492730889c6e45cf15" uuid = "3d5dd08c-fd9d-11e8-17fa-ed2836048c2f" -version = "0.21.51" +version = "0.21.54" [[deps.Wayland_jll]] deps = ["Artifacts", "Expat_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg", "XML2_jll"] @@ -1369,6 +1499,12 @@ git-tree-sha1 = "4528479aa01ee1b3b4cd0e6faef0e04cf16466da" uuid = "2381bf8a-dfd0-557d-9999-79630e7b1b91" version = "1.25.0+0" +[[deps.WoodburyMatrices]] +deps = ["LinearAlgebra", "SparseArrays"] +git-tree-sha1 = "de67fa59e33ad156a590055375a30b23c40299d3" +uuid = "efce3f68-66dc-5838-9240-27a6d6f5f9b6" +version = "0.5.5" + [[deps.XML2_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "Zlib_jll"] git-tree-sha1 = "58443b63fb7e465a8a7210828c91c08b92132dff" diff --git a/docs/Project.toml b/docs/Project.toml index f648e40..ad9ad43 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -1,11 +1,15 @@ [deps] ConformalPrediction = "98bfc277-1877-43dc-819b-a3e38c30242f" DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0" -DecisionTree = "f6006082-12f8-11e9-0c9c-0d5d367ab1e5" +DecisionTree = "7806a523-6efd-50cb-b5f6-3fa6f1930dbb" Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" +EvoTrees = "f6006082-12f8-11e9-0c9c-0d5d367ab1e5" MLJ = "add582a8-e3ab-11e8-2d5e-e98b27df1bc7" MLJBase = "a7f614a8-145f-11e9-1d2a-a57a1082229d" MLJDecisionTreeInterface = "c6f25543-311c-4c74-83dc-3ea6d1015661" +MLJLinearModels = "6ee0df7b-362f-4a72-a706-9e79364fb692" MLJModelInterface = "e80e1ace-859a-464e-9ed9-23947d8ae3ea" +MLJNaiveBayesInterface = "33e4bacb-b9e2-458e-9a13-5d9a90b235fa" +NaiveBayes = "9bbee03b-0db5-5f46-924f-b5c9c21b8c60" PlotThemes = "ccf2f8ad-2431-5c83-bf29-c5338b663b6a" Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80" diff --git a/docs/src/classification.md b/docs/src/classification.md new file mode 100644 index 0000000..546810e --- /dev/null +++ b/docs/src/classification.md @@ -0,0 +1,41 @@ + +``` julia +using MLJ +X, y = MLJ.make_blobs(1000, 2; centers=3, cluster_std=1.0) +train, test = partition(eachindex(y), 0.4, 0.4, shuffle=true) +``` + +``` julia +EvoTreeClassifier = @load EvoTreeClassifier pkg=EvoTrees +model = EvoTreeClassifier() +``` + +``` julia +using ConformalPrediction +conf_model = conformal_model(model) +mach = machine(conf_model, X, y) +fit!(mach, rows=train) +``` + +``` julia +rows = rand(test, 10) +Xtest = selectrows(X, rows) +ytest = y[rows] +predict(mach, Xtest) +``` + + ╭───────────────────────────────────────────────────────────────────╮ + │ │ + │ (1) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │ + │ (2) UnivariateFinite{Multiclass {#90CAF9}3} (3=>0.82{/#90CAF9}) │ + │ (3) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │ + │ (4) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │ + │ (5) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │ + │ (6) UnivariateFinite{Multiclass {#90CAF9}3} (3=>0.82{/#90CAF9}) │ + │ (7) UnivariateFinite{Multiclass {#90CAF9}3} (3=>0.82{/#90CAF9}) │ + │ (8) UnivariateFinite{Multiclass {#90CAF9}3} (2=>0.82{/#90CAF9}) │ + │ (9) UnivariateFinite{Multiclass {#90CAF9}3} (1=>0.82{/#90CAF9}) │ + │ (10) UnivariateFinite{Multiclass {#90CAF9}3} (3=>0.82{/#90CAF9}) │ + │ │ + │ │ + ╰────────────────────────────────────────────────────── 10 items ───╯ diff --git a/docs/src/classification.qmd b/docs/src/classification.qmd new file mode 100644 index 0000000..0dc31b7 --- /dev/null +++ b/docs/src/classification.qmd @@ -0,0 +1,32 @@ +```{julia} +#| echo: false +using Pkg; Pkg.activate("docs") +using Plots +theme(:wong) +``` + +```{julia} +using MLJ +X, y = MLJ.make_blobs(1000, 2; centers=3, cluster_std=1.0) +train, test = partition(eachindex(y), 0.4, 0.4, shuffle=true) +``` + +```{julia} +EvoTreeClassifier = @load EvoTreeClassifier pkg=EvoTrees +model = EvoTreeClassifier() +``` + +```{julia} +using ConformalPrediction +conf_model = conformal_model(model) +mach = machine(conf_model, X, y) +fit!(mach, rows=train) +``` + +```{julia} +#| output: true +rows = rand(test, 10) +Xtest = selectrows(X, rows) +ytest = y[rows] +predict(mach, Xtest) +``` \ No newline at end of file diff --git a/docs/src/contribute_files/figure-commonmark/mermaid-figure-1.png b/docs/src/contribute_files/figure-commonmark/mermaid-figure-1.png index dfc3a35..0ccffd0 100644 Binary files a/docs/src/contribute_files/figure-commonmark/mermaid-figure-1.png and b/docs/src/contribute_files/figure-commonmark/mermaid-figure-1.png differ diff --git a/docs/src/index.md b/docs/src/index.md index e796f71..bdd69f3 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -7,10 +7,6 @@ CurrentModule = ConformalPrediction Documentation for [ConformalPrediction.jl](https://github.com/pat-alt/ConformalPrediction.jl). -``` @meta -CurrentModule = ConformalPrediction -``` - `ConformalPrediction.jl` is a package for Uncertainty Quantification (UQ) through Conformal Prediction (CP) in Julia. It is designed to work with supervised models trained in [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/). Conformal Prediction is distribution-free, easy-to-understand, easy-to-use and model-agnostic. ## Installation 🚩 @@ -60,11 +56,11 @@ X, y = MLJ.make_regression(1000, 2) train, test = partition(eachindex(y), 0.4, 0.4) ``` -We then import a decision tree ([DecisionTree](https://github.com/Evovest/DecisionTree.jl)) following the standard [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) procedure. +We then import a decision tree ([`EvoTrees.jl`](https://github.com/Evovest/EvoTrees.jl)) following the standard [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) procedure. ``` julia -DecisionTreeRegressor = @load DecisionTreeRegressor pkg=DecisionTree -model = DecisionTreeRegressor() +EvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees +model = EvoTreeRegressor() ``` To turn our conventional model into a conformal model, we just need to declare it as such by using `conformal_model` wrapper function. The generated conformal model instance can wrapped in data to create a *machine*. Finally, we proceed by fitting the machine on training data using the generic `fit!` method: @@ -85,21 +81,21 @@ ytest = y[first(test,n)] predict(mach, Xtest) ``` - ╭──────────────────────────────────────────────────────────────────╮ - │ │ - │ (1) ([-0.16035036780321532], [1.4939904924997824]) │ - │ (2) ([-1.086589388667894], [0.5677514716351038]) │ - │ (3) ([-1.086589388667894], [0.5677514716351038]) │ - │ (4) ([-1.6661164684544767], [-0.011775608151479156]) │ - │ (5) ([-3.0116018507211617], [-1.3572609904181638]) │ - │ (6) ([0.5337083913933376], [2.1880492516963352]) │ - │ (7) ([-1.2219266921060266], [0.43241416819697115]) │ - │ (8) ([-1.6867950029289869], [-0.032454142625989335]) │ - │ (9) ([-2.0599181285783263], [-0.4055772682753287]) │ - │ (10) ([-0.06499897951385392], [1.5893418807891437]) │ - │ │ - │ │ - ╰───────────────────────────────────────────────────── 10 items ───╯ + ╭────────────────────────────────────────────────────────────────╮ + │ │ + │ (1) ([-5.4338268970328265], [0.25968688121283146]) │ + │ (2) ([-3.327102611742035], [2.3664111665036227]) │ + │ (3) ([-6.487880423821674], [-0.7943666455760168]) │ + │ (4) ([1.5074870726446568], [7.201000850890314]) │ + │ (5) ([-6.668012325746515], [-0.9744985475008576]) │ + │ (6) ([-2.36920384417906], [3.3243099340665974]) │ + │ (7) ([-0.4783861002145251], [5.215127678031132]) │ + │ (8) ([-5.554310900530298], [0.1392028777153591]) │ + │ (9) ([-1.4607932119178935], [4.232720566327764]) │ + │ (10) ([-5.190158387746367], [0.5033553904992907]) │ + │ │ + │ │ + ╰─────────────────────────────────────────────────── 10 items ───╯ ## Contribute 🛠 diff --git a/docs/src/intro.md b/docs/src/intro.md new file mode 100644 index 0000000..a0a13fa --- /dev/null +++ b/docs/src/intro.md @@ -0,0 +1,98 @@ + +`ConformalPrediction.jl` is a package for Uncertainty Quantification (UQ) through Conformal Prediction (CP) in Julia. It is designed to work with supervised models trained in [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/). Conformal Prediction is distribution-free, easy-to-understand, easy-to-use and model-agnostic. + +## Installation 🚩 + +You can install the first stable release from the general registry: + +``` julia +using Pkg +Pkg.add("ConformalPrediction") +``` + +The development version can be installed as follows: + +``` julia +using Pkg +Pkg.add(url="https://github.com/pat-alt/ConformalPrediction.jl") +``` + +## Status 🔁 + +This package is in its very early stages of development and therefore still subject to changes to the core architecture. The following approaches have been implemented in the development version: + +**Regression**: + +- Inductive +- Naive Transductive +- Jackknife +- Jackknife+ +- Jackknife-minmax +- CV+ +- CV-minmax + +**Classification**: + +- Inductive (LABEL (Sadinle, Lei, and Wasserman 2019)) +- Adaptive Inductive + +I have only tested it for a few of the supervised models offered by [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/). + +## Usage Example 🔍 + +To illustrate the intended use of the package, let’s have a quick look at a simple regression problem. Using [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) we first generate some synthetic data and then determine indices for our training, calibration and test data: + +``` julia +using MLJ +X, y = MLJ.make_regression(1000, 2) +train, test = partition(eachindex(y), 0.4, 0.4) +``` + +We then import a decision tree ([`EvoTrees.jl`](https://github.com/Evovest/EvoTrees.jl)) following the standard [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) procedure. + +``` julia +EvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees +model = EvoTreeRegressor() +``` + +To turn our conventional model into a conformal model, we just need to declare it as such by using `conformal_model` wrapper function. The generated conformal model instance can wrapped in data to create a *machine*. Finally, we proceed by fitting the machine on training data using the generic `fit!` method: + +``` julia +using ConformalPrediction +conf_model = conformal_model(model) +mach = machine(conf_model, X, y) +fit!(mach, rows=train) +``` + +Predictions can then be computed using the generic `predict` method. The code below produces predictions for the first `n` samples. Each tuple contains the lower and upper bound for the prediction interval. + +``` julia +n = 10 +Xtest = selectrows(X, first(test,n)) +ytest = y[first(test,n)] +predict(mach, Xtest) +``` + + ╭─────────────────────────────────────────────────────────────────╮ + │ │ + │ (1) ([-0.20063113789390163], [1.323655530145934]) │ + │ (2) ([-0.061147489871723804], [1.4631391781681118]) │ + │ (3) ([-1.4486105066363675], [0.07567616140346822]) │ + │ (4) ([-0.7160881365817455], [0.8081985314580902]) │ + │ (5) ([-1.7173644161988695], [-0.19307774815903367]) │ + │ (6) ([-1.2158809697881832], [0.3084056982516525]) │ + │ (7) ([-1.7173644161988695], [-0.19307774815903367]) │ + │ (8) ([0.26510754559144056], [1.7893942136312764]) │ + │ (9) ([-0.8716996456392521], [0.6525870224005836]) │ + │ (10) ([0.43084861624955606], [1.9551352842893919]) │ + │ │ + │ │ + ╰──────────────────────────────────────────────────── 10 items ───╯ + +## Contribute 🛠 + +Contributions are welcome! Please follow the [SciML ColPrac guide](https://github.com/SciML/ColPrac). + +## References 🎓 + +Sadinle, Mauricio, Jing Lei, and Larry Wasserman. 2019. “Least Ambiguous Set-Valued Classifiers with Bounded Error Levels.” *Journal of the American Statistical Association* 114 (525): 223–34. diff --git a/docs/src/intro.qmd b/docs/src/intro.qmd index d19f110..3d30837 100644 --- a/docs/src/intro.qmd +++ b/docs/src/intro.qmd @@ -1,7 +1,7 @@ ```{julia} #| echo: false using Pkg; Pkg.activate("docs") -using Plots, PlotThemes +using Plots theme(:wong) ``` @@ -11,14 +11,14 @@ theme(:wong) You can install the first stable release from the general registry: -```julia +```{.julia} using Pkg Pkg.add("ConformalPrediction") ``` The development version can be installed as follows: -```julia +```{.julia} using Pkg Pkg.add(url="https://github.com/pat-alt/ConformalPrediction.jl") ``` @@ -54,11 +54,11 @@ X, y = MLJ.make_regression(1000, 2) train, test = partition(eachindex(y), 0.4, 0.4) ``` -We then import a decision tree ([DecisionTree](https://github.com/Evovest/DecisionTree.jl)) following the standard [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) procedure. +We then import a decision tree ([`EvoTrees.jl`](https://github.com/Evovest/EvoTrees.jl)) following the standard [MLJ](https://alan-turing-institute.github.io/MLJ.jl/dev/) procedure. ```{julia} -DecisionTreeRegressor = @load DecisionTreeRegressor pkg=DecisionTree -model = DecisionTreeRegressor() +EvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees +model = EvoTreeRegressor() ``` To turn our conventional model into a conformal model, we just need to declare it as such by using `conformal_model` wrapper function. The generated conformal model instance can wrapped in data to create a *machine*. Finally, we proceed by fitting the machine on training data using the generic `fit!` method: diff --git a/src/ConformalModels/inductive_classification.jl b/src/ConformalModels/inductive_classification.jl index 1d2390b..6a1bc6a 100644 --- a/src/ConformalModels/inductive_classification.jl +++ b/src/ConformalModels/inductive_classification.jl @@ -27,13 +27,15 @@ function MMI.fit(conf_model::SimpleInductiveClassifier, verbosity, X, y) # Data Splitting: train, calibration = partition(eachindex(y), conf_model.train_ratio) - Xtrain = MLJ.matrix(X)[train,:] + Xtrain = selectrows(X, train) ytrain = y[train] - Xcal = MLJ.matrix(X)[calibration,:] + Xtrain, ytrain = MMI.reformat(conf_model.model, Xtrain, ytrain) + Xcal = selectrows(X, calibration) ycal = y[calibration] + Xcal, ycal = MMI.reformat(conf_model.model, Xcal, ycal) # Training: - fitresult, cache, report = MMI.fit(conf_model.model, verbosity, MMI.reformat(conf_model.model, Xtrain, ytrain)...) + fitresult, cache, report = MMI.fit(conf_model.model, verbosity, Xtrain, ytrain) # Nonconformity Scores: ŷ = pdf.(MMI.predict(conf_model.model, fitresult, Xcal), ycal) @@ -55,12 +57,20 @@ where ``\mathcal{D}_{\text{calibration}}`` denotes the designated calibration da """ function MMI.predict(conf_model::SimpleInductiveClassifier, fitresult, Xnew) p̂ = MMI.predict(conf_model.model, fitresult, MMI.reformat(conf_model.model, Xnew)...) - L = p̂.decoder.classes - ŷ = pdf(p̂, L) v = conf_model.scores q̂ = Statistics.quantile(v, conf_model.coverage) - ŷ = map(x -> collect(key => 1.0-val <= q̂ ? val : missing for (key,val) in zip(L,x)),eachrow(ŷ)) - return ŷ + p̂ = map(p̂) do pp + L = p̂.decoder.classes + probas = pdf.(pp, L) + is_in_set = 1.0 .- probas .<= q̂ + if !all(is_in_set .== false) + pp = UnivariateFinite(L[is_in_set], probas[is_in_set]) + else + pp = missing + end + return pp + end + return p̂ end # Adaptive @@ -90,16 +100,18 @@ function MMI.fit(conf_model::AdaptiveInductiveClassifier, verbosity, X, y) # Data Splitting: train, calibration = partition(eachindex(y), conf_model.train_ratio) - Xtrain = MLJ.matrix(X)[train,:] + Xtrain = selectrows(X, train) ytrain = y[train] - Xcal = MLJ.matrix(X)[calibration,:] + Xtrain, ytrain = MMI.reformat(conf_model.model, Xtrain, ytrain) + Xcal = selectrows(X, calibration) ycal = y[calibration] + Xcal, ycal = MMI.reformat(conf_model.model, Xcal, ycal) # Training: - fitresult, cache, report = MMI.fit(conf_model.model, verbosity, MMI.reformat(conf_model.model, Xtrain, ytrain)...) + fitresult, cache, report = MMI.fit(conf_model.model, verbosity, Xtrain, ytrain) # Nonconformity Scores: - p̂ = MMI.predict(conf_model.model, fitresult, MMI.reformat(conf_model.model, Xcal)...) + p̂ = MMI.predict(conf_model.model, fitresult, Xcal) L = p̂.decoder.classes ŷ = pdf(p̂, L) # compute probabilities for all classes scores = map(eachrow(ŷ),eachrow(ycal)) do ŷᵢ, ycalᵢ @@ -126,11 +138,19 @@ where ``\mathcal{D}_{\text{calibration}}`` denotes the designated calibration da """ function MMI.predict(conf_model::AdaptiveInductiveClassifier, fitresult, Xnew) p̂ = MMI.predict(conf_model.model, fitresult, MMI.reformat(conf_model.model, Xnew)...) - L = p̂.decoder.classes - ŷ = pdf(p̂, L) v = conf_model.scores q̂ = Statistics.quantile(v, conf_model.coverage) - ŷ = map(x -> collect(key => 1.0-val <= q̂ ? val : missing for (key,val) in zip(L,x)),eachrow(ŷ)) - return ŷ + p̂ = map(p̂) do pp + L = p̂.decoder.classes + probas = pdf.(pp, L) + is_in_set = 1.0 .- probas .<= q̂ + if !all(is_in_set .== false) + pp = UnivariateFinite(L[is_in_set], probas[is_in_set]) + else + pp = missing + end + return pp + end + return p̂ end diff --git a/src/ConformalModels/inductive_regression.jl b/src/ConformalModels/inductive_regression.jl index 554ddcd..6fe62b8 100644 --- a/src/ConformalModels/inductive_regression.jl +++ b/src/ConformalModels/inductive_regression.jl @@ -26,13 +26,15 @@ function MMI.fit(conf_model::SimpleInductiveRegressor, verbosity, X, y) # Data Splitting: train, calibration = partition(eachindex(y), conf_model.train_ratio) - Xtrain = MLJ.matrix(X)[train,:] + Xtrain = selectrows(X, train) ytrain = y[train] - Xcal = MLJ.matrix(X)[calibration,:] + Xtrain, ytrain = MMI.reformat(conf_model.model, Xtrain, ytrain) + Xcal = selectrows(X, calibration) ycal = y[calibration] + Xcal, ycal = MMI.reformat(conf_model.model, Xcal, ycal) # Training: - fitresult, cache, report = MMI.fit(conf_model.model, verbosity, MMI.reformat(conf_model.model, Xtrain, ytrain)...) + fitresult, cache, report = MMI.fit(conf_model.model, verbosity, Xtrain, ytrain) # Nonconformity Scores: ŷ = MMI.predict(conf_model.model, fitresult, Xcal) diff --git a/src/ConformalModels/transductive_classification.jl b/src/ConformalModels/transductive_classification.jl index abd050e..20e34f8 100644 --- a/src/ConformalModels/transductive_classification.jl +++ b/src/ConformalModels/transductive_classification.jl @@ -24,8 +24,11 @@ A typical choice for the heuristic function is ``h(\hat\mu(X_i), Y_i)=1-\hat\mu( """ function MMI.fit(conf_model::NaiveClassifier, verbosity, X, y) + # Setup: + X, y = MMI.reformat(conf_model.model, X, y) + # Training: - fitresult, cache, report = MMI.fit(conf_model.model, verbosity, MMI.reformat(conf_model.model, X, y)...) + fitresult, cache, report = MMI.fit(conf_model.model, verbosity, X, y) # Nonconformity Scores: ŷ = pdf.(MMI.predict(conf_model.model, fitresult, X),y) @@ -48,10 +51,18 @@ The naive approach typically produces prediction regions that undercover due to """ function MMI.predict(conf_model::NaiveClassifier, fitresult, Xnew) p̂ = MMI.predict(conf_model.model, fitresult, MMI.reformat(conf_model.model, Xnew)...) - L = p̂.decoder.classes - ŷ = pdf(p̂, L) v = conf_model.scores q̂ = Statistics.quantile(v, conf_model.coverage) - ŷ = map(x -> collect(key => 1.0-val <= q̂ ? val : missing for (key,val) in zip(L,x)),eachrow(ŷ)) - return ŷ + p̂ = map(p̂) do pp + L = p̂.decoder.classes + probas = pdf.(pp, L) + is_in_set = 1.0 .- probas .<= q̂ + if !all(is_in_set .== false) + pp = UnivariateFinite(L[is_in_set], probas[is_in_set]) + else + pp = missing + end + return pp + end + return p̂ end \ No newline at end of file diff --git a/src/ConformalModels/transductive_regression.jl b/src/ConformalModels/transductive_regression.jl index a455a27..1f9686f 100644 --- a/src/ConformalModels/transductive_regression.jl +++ b/src/ConformalModels/transductive_regression.jl @@ -29,8 +29,11 @@ A typical choice for the heuristic function is ``h(\hat\mu(X_i),Y_i)=|Y_i-\hat\m """ function MMI.fit(conf_model::NaiveRegressor, verbosity, X, y) + # Setup: + X, y = MMI.reformat(conf_model.model, X, y) + # Training: - fitresult, cache, report = MMI.fit(conf_model.model, verbosity, MMI.reformat(conf_model.model, X, y)...) + fitresult, cache, report = MMI.fit(conf_model.model, verbosity, X, y) # Nonconformity Scores: ŷ = MMI.predict(conf_model.model, fitresult, X) @@ -87,8 +90,11 @@ where ``\hat\mu_{-i}(X_i)`` denotes the leave-one-out prediction for ``X_i``. In """ function MMI.fit(conf_model::JackknifeRegressor, verbosity, X, y) + # Setup: + X, y = MMI.reformat(conf_model.model, X, y) + # Training: - fitresult, cache, report = MMI.fit(conf_model.model, verbosity, MMI.reformat(conf_model.model, X, y)...) + fitresult, cache, report = MMI.fit(conf_model.model, verbosity, X, y) # Nonconformity Scores: T = size(y, 1)