Develop

_freeze/docs/src/contribute/execute-results/md.json

@@ -0,0 +1,10 @@
+{
+  "hash": "c8191078af8739a251639903687b487e",
+  "result": {
+    "markdown": "# Contributor's Guide\n\n```@meta\nCurrentModule = ConformalPrediction\n```\n\n## Contents\n\n```@contents\nPages = [\"contribute.md\"]\nDepth = 2\n```\n\n## Contributing to `ConformalPrediction.jl`\n\nContributions are welcome! Please follow the [SciML ColPrac guide](https://github.com/SciML/ColPrac).\n\n## Architecture\n\nThe diagram below demonstrates the package architecture at the time of writing. This is still subject to change, so any thoughts and comments are very much welcome. \n\nThe goal is to make this package as compatible as possible with MLJ to tab into existing functionality. The basic idea is to subtype MLJ `Supervised` models and then use concrete types to implement different approaches to conformal prediction. For each of these concrete types the compulsory `MMI.fit` and `MMI.predict` methods need be implemented (see [here](https://alan-turing-institute.github.io/MLJ.jl/v0.18/adding_models_for_general_use/#Supervised-models)). \n\n\n```{mermaid}\n%%| echo: false\nflowchart TB\n    mmi[MLJModelInterface]\n    subgraph ConformalModel\n        interval[ConformalInterval]\n        set[ConformalSet]\n        prob[ConformalProbabilistic]\n        struct1([NaiveRegressor])\n        struct2([...])\n        fit((MMI.fit))\n        predict((MMI.predict))\n        interval --> struct1\n        set --> struct2\n        struct1 & struct2 --dispatch--> fit & predict\n    end\n    \n    mmi --<:MMI.Interval--> interval\n    mmi --<:MMI.Supervised--> set\n    mmi --<:MMI.Probabilistic--> prob\n```\n\n\n### Abstract Suptypes\n\nCurrently I intend to work with three different abstract subtypes: \n\n```@docs\nConformalInterval\nConformalSet\nConformalProbabilistic\n```\n\n### `fit` and `predict`\n\nThe `fit` and `predict` methods are compulsory in order to prepare models for general use with MLJ. They also serve us to implement the logic underlying the various approaches to conformal prediction. \n\nTo understand how this currently works, let's look at the [`JackknifeRegressor`](@ref) as an example. Below are the two docstrings documenting both methods. \n\n\n```{md}\n!!! note \"Source Code\"\n    Hovering over the bottom-right corner will reveal buttons that take \n```\n\n\n```@docs\nfit(conf_model::JackknifeRegressor, verbosity, X, y)\n```\n\n```@docs\npredict(conf_model::JackknifeRegressor, fitresult, Xnew)\n```\n\n",
+    "supporting": [
+      "contribute_files/figure-commonmark"
+    ],
+    "filters": []
+  }
+}

docs/_metadata.yml

@@ -1,5 +1,5 @@
 format: 
   commonmark:
     variant: -raw_html
-    wrap: none
+    wrap: preserve
     self-contained: true

docs/make.jl

@@ -2,11 +2,13 @@ using ConformalPrediction
 using Documenter
 
 ex_meta = quote
-    # Import module:
+    # Import module(s):
     using ConformalPrediction
+    using MLJ
+    using MLJModelInterface
+    const MMI = MLJModelInterface
 
     # Data:
-    using MLJ
     X, y = MLJ.make_regression(1000, 2)
     train, calibration, test = partition(eachindex(y), 0.4, 0.4)
 
@@ -29,8 +31,9 @@ makedocs(;
         assets=String[],
     ),
     pages=[
-        "Home" => "index.md",
-        "Reference" => "reference.md",
+        "🏠 Home" => "index.md",
+        "🛠 Contribute" => "contribute.md",
+        "📖 Library" => "reference.md",
     ],
 )
 

docs/src/contribute.md

@@ -0,0 +1,49 @@
+
+# Contributor’s Guide
+
+``` @meta
+CurrentModule = ConformalPrediction
+```
+
+## Contents
+
+``` @contents
+Pages = ["contribute.md"]
+Depth = 2
+```
+
+## Contributing to `ConformalPrediction.jl`
+
+Contributions are welcome! Please follow the [SciML ColPrac guide](https://github.com/SciML/ColPrac).
+
+## Architecture
+
+The diagram below demonstrates the package architecture at the time of writing. This is still subject to change, so any thoughts and comments are very much welcome.
+
+The goal is to make this package as compatible as possible with MLJ to tab into existing functionality. The basic idea is to subtype MLJ `Supervised` models and then use concrete types to implement different approaches to conformal prediction. For each of these concrete types the compulsory `MMI.fit` and `MMI.predict` methods need be implemented (see [here](https://alan-turing-institute.github.io/MLJ.jl/v0.18/adding_models_for_general_use/#Supervised-models)).
+
+![](contribute_files/figure-commonmark/mermaid-figure-1.png)
+
+### Abstract Suptypes
+
+Currently I intend to work with three different abstract subtypes:
+
+``` @docs
+ConformalInterval
+ConformalSet
+ConformalProbabilistic
+```
+
+### `fit` and `predict`
+
+The `fit` and `predict` methods are compulsory in order to prepare models for general use with MLJ. They also serve us to implement the logic underlying the various approaches to conformal prediction.
+
+To understand how this currently works, let’s look at the [`AdaptiveInductiveClassifier`](@ref) as an example. Below are the two docstrings documenting both methods. Hovering over the bottom-right corner will reveal buttons that take
+
+``` @docs
+fit(conf_model::AdaptiveInductiveClassifier, verbosity, X, y)
+```
+
+``` @docs
+predict(conf_model::AdaptiveInductiveClassifier, fitresult, Xnew)
+```

docs/src/contribute.qmd

@@ -0,0 +1,72 @@
+# Contributor's Guide
+
+```@meta
+CurrentModule = ConformalPrediction
+```
+
+## Contents
+
+```@contents
+Pages = ["contribute.md"]
+Depth = 2
+```
+
+## Contributing to `ConformalPrediction.jl`
+
+Contributions are welcome! Please follow the [SciML ColPrac guide](https://github.com/SciML/ColPrac).
+
+## Architecture
+
+The diagram below demonstrates the package architecture at the time of writing. This is still subject to change, so any thoughts and comments are very much welcome. 
+
+The goal is to make this package as compatible as possible with MLJ to tab into existing functionality. The basic idea is to subtype MLJ `Supervised` models and then use concrete types to implement different approaches to conformal prediction. For each of these concrete types the compulsory `MMI.fit` and `MMI.predict` methods need be implemented (see [here](https://alan-turing-institute.github.io/MLJ.jl/v0.18/adding_models_for_general_use/#Supervised-models)). 
+
+```{mermaid}
+%%| echo: false
+flowchart TB
+    mmi[MLJModelInterface]
+    subgraph ConformalModel
+        interval[ConformalInterval]
+        set[ConformalSet]
+        prob[ConformalProbabilistic]
+        struct1([NaiveRegressor])
+        struct2([...])
+        fit((MMI.fit))
+        predict((MMI.predict))
+        interval --> struct1
+        set --> struct2
+        struct1 & struct2 --dispatch--> fit & predict
+    end
+
+    mmi --<:MMI.Interval--> interval
+    mmi --<:MMI.Supervised--> set
+    mmi --<:MMI.Probabilistic--> prob
+```
+
+### Abstract Suptypes
+
+Currently I intend to work with three different abstract subtypes: 
+
+```@docs
+ConformalInterval
+ConformalSet
+ConformalProbabilistic
+```
+
+### `fit` and `predict`
+
+The `fit` and `predict` methods are compulsory in order to prepare models for general use with MLJ. They also serve us to implement the logic underlying the various approaches to conformal prediction. 
+
+To understand how this currently works, let's look at the [`AdaptiveInductiveClassifier`](@ref) as an example. Below are the two docstrings documenting both methods. Hovering over the bottom-right corner will reveal buttons that take 
+
+```@docs
+fit(conf_model::AdaptiveInductiveClassifier, verbosity, X, y)
+```
+
+```@docs
+predict(conf_model::AdaptiveInductiveClassifier, fitresult, Xnew)
+```
+
+
+
+

docs/src/reference.md

@@ -2,12 +2,18 @@
 CurrentModule = ConformalPrediction
 ```
 
-# All functions and types
+## Content
+
+```@contents
+Pages = ["reference.md"]
+```
+
+## Index
 
 ```@index
 ```
 
-## Exported functions
+## Public Interface
 
 ```@autodocs
 Modules = [

src/ConformalModels/ConformalModels.jl

@@ -4,41 +4,49 @@ using MLJ
 import MLJModelInterface as MMI
 import MLJModelInterface: predict, fit, save, restore
 
-"An abstract base type for conformal models."
-abstract type ConformalModel <: MMI.Model end
+"An abstract base type for conformal models that produce interval-values predictions. This includes most conformal regression models."
+abstract type ConformalInterval <: MMI.Interval end
 
-@doc raw"""
-An abstract base time of Inductive Conformal Models. These models rely on data splitting. In particular, we partition the training data as ``\{1,...,n\}=\mathcal{D}_{\text{train}} \cup \mathcal{D}_{\text{calibration}}``.
-"""
-abstract type InductiveConformalModel <: ConformalModel end
+"An abstract base type for conformal models that produce set-values predictions. This includes most conformal classification models."
+abstract type ConformalSet <: MMI.Supervised end        # ideally we'd have MMI.Set
 
-@doc raw"""
-An abstract base time of Transductive Conformal Models. These models do not rely on data splitting. In particular, nonconformity scores are computed using the entire trainign data set ``\{1,...,n\}=\mathcal{D}_{\text{train}}``.
-"""
-abstract type TransductiveConformalModel <: ConformalModel end
+"An abstract base type for conformal models that produce probabilistic predictions. This includes some conformal classifier like Venn-ABERS."
+abstract type ConformalProbabilistic <: MMI.Probabilistic end
 
-export ConformalModel, InductiveConformalModel, TransductiveConformalModel
+const ConformalModel = Union{ConformalInterval, ConformalSet, ConformalProbabilistic}
+
+export ConformalInterval, ConformalSet, ConformalProbabilistic, ConformalModel
 
 include("conformal_models.jl")
 
 # Regression Models:
 include("inductive_regression.jl")
-export InductiveConformalRegressor
 export SimpleInductiveRegressor
 include("transductive_regression.jl")
-export TransductiveConformalRegressor
 export NaiveRegressor, JackknifeRegressor, JackknifePlusRegressor, JackknifeMinMaxRegressor, CVPlusRegressor, CVMinMaxRegressor
 
 # Classification Models
 include("inductive_classification.jl")
-export InductiveConformalClassifier
 export SimpleInductiveClassifier, AdaptiveInductiveClassifier
 include("transductive_classification.jl")
-export TransductiveConformalClassifier
 export NaiveClassifier
 
-const ConformalClassifier = Union{InductiveConformalClassifier, TransductiveConformalClassifier}
-const ConformalRegressor = Union{InductiveConformalRegressor, TransductiveConformalRegressor}
+# Type unions:
+const InductiveModel = Union{
+    SimpleInductiveRegressor,
+    SimpleInductiveClassifier,
+    AdaptiveInductiveClassifier
+}
+
+const TransductiveModel = Union{
+    NaiveRegressor,
+    JackknifeRegressor,
+    JackknifePlusRegressor,
+    JackknifeMinMaxRegressor,
+    CVPlusRegressor,
+    CVMinMaxRegressor,
+    NaiveClassifier
+}
 
 "A container listing all available methods for conformal prediction."
 const available_models = Dict(
@@ -67,7 +75,9 @@ const available_models = Dict(
 )
 export available_models
 
+include("model_traits.jl")
+
 # Other general methods:
-export conformal_model, qplus
+export conformal_model, fit, predict
 
 end

src/ConformalModels/inductive_classification.jl

@@ -1,9 +1,6 @@
-"A base type for Inductive Conformal Classifiers."
-abstract type InductiveConformalClassifier <: InductiveConformalModel end
-
 # Simple
 "The `SimpleInductiveClassifier` is the simplest approach to Inductive Conformal Classification. Contrary to the [`NaiveClassifier`](@ref) it computes nonconformity scores using a designated calibration dataset."
-mutable struct SimpleInductiveClassifier{Model <: Supervised} <: InductiveConformalClassifier
+mutable struct SimpleInductiveClassifier{Model <: Supervised} <: ConformalSet
     model::Model
     coverage::AbstractFloat
     scores::Union{Nothing,AbstractArray}
@@ -18,7 +15,13 @@ end
 @doc raw"""
     MMI.fit(conf_model::SimpleInductiveClassifier, verbosity, X, y)
 
-Wrapper function to fit the underlying MLJ model.
+For the [`SimpleInductiveClassifier`](@ref) nonconformity scores are computed as follows:
+
+``
+S_i = s(X_i, Y_i) = h(X_i, Y_i), \ i \in \mathcal{D}_{\text{calibration}}
+``
+
+A typical choice for the heuristic function is ``h(X_i,Y_i)=1-\hat\mu(X_i)_{Y_i}`` where ``\hat\mu(X_i)_{Y_i}`` denotes the softmax output of the true class and ``\hat\mu`` denotes the model fitted on training data ``\mathcal{D}_{\text{train}}``. The simple approach only takes the softmax probability of the true label into account.
 """
 function MMI.fit(conf_model::SimpleInductiveClassifier, verbosity, X, y)
 
@@ -45,10 +48,10 @@ end
 For the [`SimpleInductiveClassifier`](@ref) prediction sets are computed as follows,
 
 ``
-\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{1 - \hat\mu(X_i)\} \right\}, \ i \in \mathcal{D}_{\text{calibration}}
+\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i\} \right\}, \ i \in \mathcal{D}_{\text{calibration}}
 ``
 
-where ``\mathcal{D}_{\text{calibration}}`` denotes the designated calibration data and ``\hat\mu`` denotes the model fitted on training data ``\mathcal{D}_{\text{train}}``.
+where ``\mathcal{D}_{\text{calibration}}`` denotes the designated calibration data.
 """
 function MMI.predict(conf_model::SimpleInductiveClassifier, fitresult, Xnew)
     p̂ = MMI.predict(conf_model.model, fitresult, MMI.reformat(conf_model.model, Xnew)...)
@@ -61,8 +64,8 @@ function MMI.predict(conf_model::SimpleInductiveClassifier, fitresult, Xnew)
 end
 
 # Adaptive
-"The `AdaptiveInductiveClassifier` is the simplest approach to Inductive Conformal Classification. Contrary to the [`NaiveClassifier`](@ref) it computes nonconformity scores using a designated calibration dataset."
-mutable struct AdaptiveInductiveClassifier{Model <: Supervised} <: InductiveConformalClassifier
+"The `AdaptiveInductiveClassifier` is an improvement to the [`SimpleInductiveClassifier`](@ref) and the [`NaiveClassifier`](@ref). Contrary to the [`NaiveClassifier`](@ref) it computes nonconformity scores using a designated calibration dataset like the [`SimpleInductiveClassifier`](@ref). Contrary to the [`SimpleInductiveClassifier`](@ref) it utilizes the softmax output of all classes."
+mutable struct AdaptiveInductiveClassifier{Model <: Supervised} <: ConformalSet
     model::Model
     coverage::AbstractFloat
     scores::Union{Nothing,AbstractArray}
@@ -77,7 +80,11 @@ end
 @doc raw"""
     MMI.fit(conf_model::AdaptiveInductiveClassifier, verbosity, X, y)
 
-Wrapper function to fit the underlying MLJ model.
+For the [`AdaptiveInductiveClassifier`](@ref) nonconformity scores are computed by cumulatively summing the ranked scores of each label in descending order until reaching the true label ``Y_i``:
+
+``
+S_i = s(X_i,Y_i) = \sum_{j=1}^k  \hat\mu(X_i)_{\pi_j} \ \text{where } \ Y_i=\pi_k,  i \in \mathcal{D}_{\text{calibration}}
+``
 """
 function MMI.fit(conf_model::AdaptiveInductiveClassifier, verbosity, X, y)
 
@@ -109,19 +116,13 @@ end
 @doc raw"""
     MMI.predict(conf_model::AdaptiveInductiveClassifier, fitresult, Xnew)
 
-For the [`AdaptiveInductiveClassifier`](@ref) nonconformity scores are computed by cumulatively summing the ranked scores of each label in descending order until reaching the true label ``y_i``:
-
-``
-s_i(X_i,y_i) = \sum_{j=1}^k  \hat\mu(X_i)_{\pi_j} \ \text{where } \ y_i=\pi_k,  i \in \mathcal{D}_{\text{calibration}}
-``
-
-Prediction sets are then computed as follows,
+For the [`AdaptiveInductiveClassifier`](@ref) prediction sets are computed as follows,
 
 ``
-\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{s_i(X_i,y_i)\} \right\},  i \in \mathcal{D}_{\text{calibration}}
+\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i\} \right\},  i \in \mathcal{D}_{\text{calibration}}
 ``
 
-where ``\mathcal{D}_{\text{calibration}}`` denotes the designated calibration data and ``\hat\mu`` denotes the model fitted on training data ``\mathcal{D}_{\text{train}}``.
+where ``\mathcal{D}_{\text{calibration}}`` denotes the designated calibration data.
 """
 function MMI.predict(conf_model::AdaptiveInductiveClassifier, fitresult, Xnew)
     p̂ = MMI.predict(conf_model.model, fitresult, MMI.reformat(conf_model.model, Xnew)...)

src/ConformalModels/inductive_regression.jl

@@ -1,8 +1,5 @@
-"A base type for Inductive Conformal Regressors."
-abstract type InductiveConformalRegressor <: InductiveConformalModel end
-
 "The `SimpleInductiveRegressor` is the simplest approach to Inductive Conformal Regression. Contrary to the [`NaiveRegressor`](@ref) it computes nonconformity scores using a designated calibration dataset."
-mutable struct SimpleInductiveRegressor{Model <: Supervised} <: InductiveConformalRegressor
+mutable struct SimpleInductiveRegressor{Model <: Supervised} <: ConformalInterval
     model::Model
     coverage::AbstractFloat
     scores::Union{Nothing,AbstractArray}
@@ -17,7 +14,13 @@ end
 @doc raw"""
     MMI.fit(conf_model::SimpleInductiveRegressor, verbosity, X, y)
 
-Wrapper function to fit the underlying MLJ model.
+For the [`SimpleInductiveRegressor`](@ref) nonconformity scores are computed as follows:
+
+``
+S_i = s(X_i, Y_i) = h(X_i, Y_i), \ i \in \mathcal{D}_{\text{calibration}}
+``
+
+A typical choice for the heuristic function is ``h(X_i,Y_i)=|Y_i-\hat\mu(X_i)|`` where ``\hat\mu`` denotes the model fitted on training data ``\mathcal{D}_{\text{train}}``.
 """
 function MMI.fit(conf_model::SimpleInductiveRegressor, verbosity, X, y)
 
@@ -45,10 +48,10 @@ end
 For the [`SimpleInductiveRegressor`](@ref) prediction intervals are computed as follows,
 
 ``
-\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{|Y_i - \hat\mu(X_i)| \}, \ i \in \mathcal{D}_{\text{calibration}}
+\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i \}, \ i \in \mathcal{D}_{\text{calibration}}
 ``
 
-where ``\mathcal{D}_{\text{calibration}}`` denotes the designated calibration data and ``\hat\mu`` denotes the model fitted on training data ``\mathcal{D}_{\text{train}}``.
+where ``\mathcal{D}_{\text{calibration}}`` denotes the designated calibration data.
 """
 function MMI.predict(conf_model::SimpleInductiveRegressor, fitresult, Xnew)
     ŷ = MMI.predict(conf_model.model, fitresult, MMI.reformat(conf_model.model, Xnew)...)