Adds full support for conformal training

.github/workflows/CI.yml

@@ -19,7 +19,6 @@ jobs:
       fail-fast: false
       matrix:
         version:
-          - '1.6'
           - '1.7'
           - '1.8'
           - '1.9'

Project.toml

@@ -8,6 +8,7 @@ CategoricalArrays = "324d7699-5711-5eae-9e2f-1d82baa6b597"
 ChainRules = "082447d4-558c-5d27-93f4-14fc19e9eca2"
 ComputationalResources = "ed09eef8-17a6-5b46-8889-db040fac31e3"
 Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c"
+InferOpt = "4846b161-c94e-4150-8dac-c7ae193c601f"
 LazyArtifacts = "4af54fe1-eca0-43a8-85a7-787d91b784e3"
 LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
 MLJBase = "a7f614a8-145f-11e9-1d2a-a57a1082229d"
@@ -21,22 +22,24 @@ Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
 StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
 Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
+cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"
 
 [compat]
 CategoricalArrays = "0.10"
 ChainRules = "1.49.0"
 ComputationalResources = "0.3"
 Flux = "0.13.16, 0.14"
-MLJBase = "0.20, 0.21"
-MLJEnsembles = "0.3.3"
-MLJFlux = "0.2.10, 0.3"
+LazyArtifacts = "1"
+MLJBase = "0.20, 0.21, 1"
+MLJEnsembles = "0.3.3, 0.4"
+MLJFlux = "0.2.10, 0.3, 0.4"
 MLJModelInterface = "1"
 MLUtils = "0.4.2"
 NaturalSort = "1"
 ProgressMeter = "1"
 StatsBase = "0.33, 0.34.0"
 Tables = "1"
-julia = "1.7, 1.8, 1.9"
+julia = "1.6, 1.7, 1.8, 1.9"
 
 [extras]
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

_freeze/docs/src/tutorials/plotting/execute-results/md.json

@@ -1,9 +1,10 @@
 {
-  "hash": "c6e3193657b1eee4f55df45354477754",
+  "hash": "b9cf54ba3e24e6647a3f7153438e30a5",
   "result": {
-    "markdown": "---\ntitle: Visualization using `Plots.jl` recipes\n---\n\n\n\n```@meta\nCurrentModule = ConformalPrediction\n```\n\n\n\n\nThis tutorial demonstrates how various custom `Plots.jl` recipes can be used to visually analyze conformal predictors.\n\n::: {.cell execution_count=2}\n``` {.julia .cell-code}\nusing ConformalPrediction\n```\n:::\n\n\n## Regression\n\n### Visualizing Prediction Intervals\n\nFor conformal regressors, the [`Plots.plot(conf_model::ConformalPrediction.ConformalInterval, fitresult, X, y; kwrgs...)`](@ref) can be used to visualize the prediction intervals for given data points.\n\n#### Univariate Input\n\n::: {.cell execution_count=3}\n``` {.julia .cell-code}\nusing MLJ\nX, y = make_regression(100, 1; noise=0.3)\n```\n:::\n\n\n::: {.cell execution_count=4}\n``` {.julia .cell-code}\nEvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees\nmodel = EvoTreeRegressor() \nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=5}\n``` {.julia .cell-code}\nplot(mach.model, mach.fitresult, X, y; input_var=1)\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n![](plotting_files/figure-commonmark/cell-6-output-1.svg){}\n:::\n:::\n\n\n#### Multivariate Input\n\n::: {.cell execution_count=6}\n``` {.julia .cell-code}\nusing MLJ\nX, y = @load_boston\nschema(X)\n```\n:::\n\n\n::: {.cell execution_count=7}\n``` {.julia .cell-code}\nEvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees\nmodel = EvoTreeRegressor() \nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=8}\n``` {.julia .cell-code}\ninput_vars = [:Crim, :Age, :Tax]\nnvars = length(input_vars)\nplt_list = []\nfor input_var in input_vars\n    plt = plot(mach.model, mach.fitresult, X, y; input_var=input_var, title=input_var)\n    push!(plt_list, plt)\nend\nplot(plt_list..., layout=(1,nvars), size=(nvars*200, 200))\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n![](plotting_files/figure-commonmark/cell-9-output-1.svg){}\n:::\n:::\n\n\n### Visualizing Set Size\n\nTo visualize the set size distribution, the [`Plots.bar(conf_model::ConformalPrediction.ConformalModel, fitresult, X; label=\"\", xtickfontsize=6, kwrgs...)`](@ref) can be used. For regression models the prediction interval widths are stratified into discrete bins.a\n\n::: {.cell execution_count=9}\n``` {.julia .cell-code}\nbar(mach.model, mach.fitresult, X)\n```\n\n::: {.cell-output .cell-output-display execution_count=10}\n![](plotting_files/figure-commonmark/cell-10-output-1.svg){}\n:::\n:::\n\n\n::: {.cell execution_count=10}\n``` {.julia .cell-code}\nEvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees\nmodel = EvoTreeRegressor() \nconf_model = conformal_model(model, method=:jackknife_plus)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=11}\n``` {.julia .cell-code}\nbar(mach.model, mach.fitresult, X)\n```\n\n::: {.cell-output .cell-output-display execution_count=12}\n![](plotting_files/figure-commonmark/cell-12-output-1.svg){}\n:::\n:::\n\n\n## Classification\n\n::: {.cell execution_count=12}\n``` {.julia .cell-code}\nKNNClassifier = @load KNNClassifier pkg=NearestNeighborModels\nmodel = KNNClassifier(;K=3)\n```\n:::\n\n\n### Visualizing Predictions\n\n#### Stacked Area Charts\n\nStacked area charts can be used to visualize prediction sets for any conformal classifier.a\n\n::: {.cell execution_count=13}\n``` {.julia .cell-code}\nusing MLJ\nn_input = 4\nX, y = make_blobs(100, n_input)\n```\n:::\n\n\n::: {.cell execution_count=14}\n``` {.julia .cell-code}\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=15}\n``` {.julia .cell-code}\nplt_list = []\nfor i in 1:n_input\n    plt = areaplot(mach.model, mach.fitresult, X, y; input_var=i, title=\"Input $i\")\n    push!(plt_list, plt)\nend\nplot(plt_list..., size=(220*n_input,200), layout=(1, n_input))\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n![](plotting_files/figure-commonmark/cell-16-output-1.svg){}\n:::\n:::\n\n\n#### Contour Plots for Two-Dimensional Inputs\n\nFor conformal classifiers with exactly two input variables, the [`Plots.contourf(conf_model::ConformalPrediction.ConformalProbabilisticSet, fitresult, X, y; kwrgs...)`](@ref) method can be used to visualize conformal predictions in the two-dimensional feature space.a\n\n::: {.cell execution_count=16}\n``` {.julia .cell-code}\nusing MLJ\nX, y = make_blobs(100, 2)\n```\n:::\n\n\n::: {.cell execution_count=17}\n``` {.julia .cell-code}\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=18}\n``` {.julia .cell-code}\np1 = contourf(mach.model, mach.fitresult, X, y)\np2 = contourf(mach.model, mach.fitresult, X, y; plot_set_size=true)\nplot(p1, p2, size=(700,300))\n```\n\n::: {.cell-output .cell-output-display execution_count=19}\n![](plotting_files/figure-commonmark/cell-19-output-1.svg){}\n:::\n:::\n\n\n### Visualizing Set Size\n\nTo visualize the set size distribution, the [`Plots.bar(conf_model::ConformalPrediction.ConformalModel, fitresult, X; label=\"\", xtickfontsize=6, kwrgs...)`](@ref) can be used. Recall that for more adaptive predictors the distribution of set sizes is typically spread out more widely, which reflects that “the procedure is effectively distinguishing between easy and hard inputs” [@angelopoulos2021gentle]. This is desirable: when for a given sample it is difficult to make predictions, this should be reflected in the set size (or interval width in the regression case). Since ‘difficult’ lies on some spectrum that ranges from ‘very easy’ to ‘very difficult’ the set size should very across the spectrum of ‘empty set’ to ‘all labels included’.\n\n::: {.cell execution_count=19}\n``` {.julia .cell-code}\nX, y = make_moons(500; noise=0.15)\nKNNClassifier = @load KNNClassifier pkg=NearestNeighborModels\nmodel = KNNClassifier(;K=50) \n```\n:::\n\n\n::: {.cell execution_count=20}\n``` {.julia .cell-code}\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=21}\n``` {.julia .cell-code}\np1 = contourf(mach.model, mach.fitresult, X, y; plot_set_size=true)\np2 = bar(mach.model, mach.fitresult, X)\nplot(p1, p2, size=(700,300))\n```\n\n::: {.cell-output .cell-output-display execution_count=22}\n![](plotting_files/figure-commonmark/cell-22-output-1.svg){}\n:::\n:::\n\n\n::: {.cell execution_count=22}\n``` {.julia .cell-code}\nconf_model = conformal_model(model, method=:adaptive_inductive)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=23}\n``` {.julia .cell-code}\np1 = contourf(mach.model, mach.fitresult, X, y; plot_set_size=true)\np2 = bar(mach.model, mach.fitresult, X)\nplot(p1, p2, size=(700,300))\n```\n\n::: {.cell-output .cell-output-display execution_count=24}\n![](plotting_files/figure-commonmark/cell-24-output-1.svg){}\n:::\n:::\n\n\n",
+    "engine": "jupyter",
+    "markdown": "---\ntitle: Visualization using `TaijaPlotting.jl`\n---\n\n\n\n```@meta\nCurrentModule = ConformalPrediction\n```\n\n\n\n\nThis tutorial demonstrates how various custom plotting methods can be used to visually analyze conformal predictors.\n\n::: {.cell execution_count=2}\n``` {.julia .cell-code}\nusing ConformalPrediction\nusing Plots, TaijaPlotting\n```\n:::\n\n\n## Regression\n\n### Visualizing Prediction Intervals\n\nFor conformal regressors, the [`TaijaPlotting.plot(conf_model::ConformalPrediction.ConformalInterval, fitresult, X, y; kwrgs...)`](@ref) can be used to visualize the prediction intervals for given data points.\n\n#### Univariate Input\n\n::: {.cell execution_count=3}\n``` {.julia .cell-code}\nusing MLJ\nX, y = make_regression(100, 1; noise=0.3)\n```\n:::\n\n\n::: {.cell execution_count=4}\n``` {.julia .cell-code}\nEvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees\nmodel = EvoTreeRegressor() \nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=5}\n``` {.julia .cell-code}\nplot(mach.model, mach.fitresult, X, y; input_var=1)\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n![](plotting_files/figure-commonmark/cell-6-output-1.svg){}\n:::\n:::\n\n\n#### Multivariate Input\n\n::: {.cell execution_count=6}\n``` {.julia .cell-code}\nusing MLJ\nX, y = @load_boston\nschema(X)\n```\n:::\n\n\n::: {.cell execution_count=7}\n``` {.julia .cell-code}\nEvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees\nmodel = EvoTreeRegressor() \nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=8}\n``` {.julia .cell-code}\ninput_vars = [:Crim, :Age, :Tax]\nnvars = length(input_vars)\nplt_list = []\nfor input_var in input_vars\n    plt = plot(mach.model, mach.fitresult, X, y; input_var=input_var, title=input_var)\n    push!(plt_list, plt)\nend\nplot(plt_list..., layout=(1,nvars), size=(nvars*200, 200))\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n![](plotting_files/figure-commonmark/cell-9-output-1.svg){}\n:::\n:::\n\n\n### Visualizing Set Size\n\nTo visualize the set size distribution, the [`TaijaPlotting.bar(conf_model::ConformalPrediction.ConformalModel, fitresult, X; label=\"\", xtickfontsize=6, kwrgs...)`](@ref) can be used. For regression models the prediction interval widths are stratified into discrete bins.a\n\n::: {.cell execution_count=9}\n``` {.julia .cell-code}\nbar(mach.model, mach.fitresult, X)\n```\n\n::: {.cell-output .cell-output-display execution_count=10}\n![](plotting_files/figure-commonmark/cell-10-output-1.svg){}\n:::\n:::\n\n\n::: {.cell execution_count=10}\n``` {.julia .cell-code}\nEvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees\nmodel = EvoTreeRegressor() \nconf_model = conformal_model(model, method=:jackknife_plus)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=11}\n``` {.julia .cell-code}\nbar(mach.model, mach.fitresult, X)\n```\n\n::: {.cell-output .cell-output-display execution_count=12}\n![](plotting_files/figure-commonmark/cell-12-output-1.svg){}\n:::\n:::\n\n\n## Classification\n\n::: {.cell execution_count=12}\n``` {.julia .cell-code}\nKNNClassifier = @load KNNClassifier pkg=NearestNeighborModels\nmodel = KNNClassifier(;K=3)\n```\n:::\n\n\n### Visualizing Predictions\n\n#### Stacked Area Charts\n\nStacked area charts can be used to visualize prediction sets for any conformal classifier.a\n\n::: {.cell execution_count=13}\n``` {.julia .cell-code}\nusing MLJ\nn_input = 4\nX, y = make_blobs(100, n_input)\n```\n:::\n\n\n::: {.cell execution_count=14}\n``` {.julia .cell-code}\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=15}\n``` {.julia .cell-code}\nplt_list = []\nfor i in 1:n_input\n    plt = areaplot(mach.model, mach.fitresult, X, y; input_var=i, title=\"Input $i\")\n    push!(plt_list, plt)\nend\nplot(plt_list..., size=(220*n_input,200), layout=(1, n_input))\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n![](plotting_files/figure-commonmark/cell-16-output-1.svg){}\n:::\n:::\n\n\n#### Contour Plots for Two-Dimensional Inputs\n\nFor conformal classifiers with exactly two input variables, the [`TaijaPlotting.contourf(conf_model::ConformalPrediction.ConformalProbabilisticSet, fitresult, X, y; kwrgs...)`](@ref) method can be used to visualize conformal predictions in the two-dimensional feature space.a\n\n::: {.cell execution_count=16}\n``` {.julia .cell-code}\nusing MLJ\nX, y = make_blobs(100, 2)\n```\n:::\n\n\n::: {.cell execution_count=17}\n``` {.julia .cell-code}\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=18}\n``` {.julia .cell-code}\np1 = contourf(mach.model, mach.fitresult, X, y)\np2 = contourf(mach.model, mach.fitresult, X, y; plot_set_size=true)\nplot(p1, p2, size=(700,300))\n```\n\n::: {.cell-output .cell-output-display execution_count=19}\n![](plotting_files/figure-commonmark/cell-19-output-1.svg){}\n:::\n:::\n\n\n### Visualizing Set Size\n\nTo visualize the set size distribution, the [`TaijaPlotting.bar(conf_model::ConformalPrediction.ConformalModel, fitresult, X; label=\"\", xtickfontsize=6, kwrgs...)`](@ref) can be used. Recall that for more adaptive predictors the distribution of set sizes is typically spread out more widely, which reflects that “the procedure is effectively distinguishing between easy and hard inputs” [@angelopoulos2021gentle]. This is desirable: when for a given sample it is difficult to make predictions, this should be reflected in the set size (or interval width in the regression case). Since ‘difficult’ lies on some spectrum that ranges from ‘very easy’ to ‘very difficult’ the set size should very across the spectrum of ‘empty set’ to ‘all labels included’.\n\n::: {.cell execution_count=19}\n``` {.julia .cell-code}\nX, y = make_moons(500; noise=0.15)\nKNNClassifier = @load KNNClassifier pkg=NearestNeighborModels\nmodel = KNNClassifier(;K=50) \n```\n:::\n\n\n::: {.cell execution_count=20}\n``` {.julia .cell-code}\nconf_model = conformal_model(model)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=21}\n``` {.julia .cell-code}\np1 = contourf(mach.model, mach.fitresult, X, y; plot_set_size=true)\np2 = bar(mach.model, mach.fitresult, X)\nplot(p1, p2, size=(700,300))\n```\n\n::: {.cell-output .cell-output-display execution_count=22}\n![](plotting_files/figure-commonmark/cell-22-output-1.svg){}\n:::\n:::\n\n\n::: {.cell execution_count=22}\n``` {.julia .cell-code}\nconf_model = conformal_model(model, method=:adaptive_inductive)\nmach = machine(conf_model, X, y)\nfit!(mach)\n```\n:::\n\n\n::: {.cell execution_count=23}\n``` {.julia .cell-code}\np1 = contourf(mach.model, mach.fitresult, X, y; plot_set_size=true)\np2 = bar(mach.model, mach.fitresult, X)\nplot(p1, p2, size=(700,300))\n```\n\n::: {.cell-output .cell-output-display execution_count=24}\n![](plotting_files/figure-commonmark/cell-24-output-1.svg){}\n:::\n:::\n\n\n",
     "supporting": [
-      "plotting_files/figure-commonmark"
+      "plotting_files"
     ],
     "filters": []
   }