Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conformal inference documentation #23

Merged
merged 24 commits into from
Aug 17, 2023
Merged

Conformal inference documentation #23

merged 24 commits into from
Aug 17, 2023

Conversation

topepo
Copy link
Member

@topepo topepo commented Jun 29, 2023

For regression models.

@netlify
Copy link

netlify bot commented Jun 29, 2023

Deploy Preview for tidymodels-org ready!

Name Link
🔨 Latest commit a216d24
🔍 Latest deploy log https://app.netlify.com/sites/tidymodels-org/deploys/64dd90d88076100008d8c365
😎 Deploy Preview https://deploy-preview-23--tidymodels-org.netlify.app/learn/models/conformal-regression
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

topepo and others added 5 commits June 29, 2023 14:18
Merge commit 'af9341d2c4f6496b99e60750f73bd7120711f610'

#Conflicts:
#	docs/404.html
#	docs/about/davis/index.html
#	docs/about/index.html
#	docs/about/max/index.html
#	docs/books/fes/index.html
#	docs/books/index.html
#	docs/books/moderndive/index.html
#	docs/books/smltar/index.html
#	docs/books/tidytext/index.html
#	docs/books/tmwr/index.html
#	docs/contribute/index.html
#	docs/find/all/index.html
#	docs/find/broom/index.html
#	docs/find/index.html
#	docs/find/parsnip/index.html
#	docs/find/recipes/index.html
#	docs/help/index.html
#	docs/index.html
#	docs/learn/develop/broom/index.html
#	docs/learn/develop/metrics/index.html
#	docs/learn/develop/models/index.html
#	docs/learn/develop/parameters/index.html
#	docs/learn/develop/recipes/index.html
#	docs/learn/index.html
#	docs/learn/models/calibration/index.html
#	docs/learn/models/coefficients/index.html
#	docs/learn/models/parsnip-nnet/index.html
#	docs/learn/models/parsnip-ranger-glmnet/index.html
#	docs/learn/models/pls/index.html
#	docs/learn/models/sub-sampling/index.html
#	docs/learn/models/time-series/index.html
#	docs/learn/statistics/bootstrap/index.html
#	docs/learn/statistics/infer/index.html
#	docs/learn/statistics/k-means/index.html
#	docs/learn/statistics/tidy-analysis/index.html
#	docs/learn/statistics/xtabs/index.html
#	docs/learn/work/bayes-opt/index.html
#	docs/learn/work/case-weights/index.html
#	docs/learn/work/nested-resampling/index.html
#	docs/learn/work/tune-svm/index.html
#	docs/learn/work/tune-text/index.html
#	docs/packages/index.html
#	docs/search.json
#	docs/start/case-study/index.html
#	docs/start/index.html
#	docs/start/models/index.html
#	docs/start/recipes/index.html
#	docs/start/resampling/index.html
#	docs/start/tuning/index.html
@topepo topepo marked this pull request as ready for review July 18, 2023 17:31
@topepo topepo requested a review from EmilHvitfeldt July 18, 2023 21:58
Copy link
Member

@EmilHvitfeldt EmilHvitfeldt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! a number of comments

learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
@EmilHvitfeldt
Copy link
Member

Once we are happy with it, I can re-render it with the right quarto version so the diff is smaller

@topepo topepo requested a review from hfrick July 20, 2023 13:49
Copy link
Member

@hfrick hfrick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One very stats-y question and a couple of comments. I like the visualizations, I think that makes it all pretty easy to follow!

learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
geom_vline(xintercept = quants, col = "red", lty = 2)
```

If we were to get a new sample beyond these data, we would be able to say that we are about 90% sure that the new sample does not conform to the original distribution. This works under the assumption that the data are exchangeable.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If we were to get a new sample beyond these data, we would be able to say that we are about 90% sure that the new sample does not conform to the original distribution. This works under the assumption that the data are exchangeable.
If we were to get a new sample beyond these boundaries, we would be able to say that we are about 90% sure that the new sample does not conform to the original distribution. This works under the assumption that the data are exchangeable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting stuck on this sentence.

If we are talking confidence intervals, I think that sentence is skirting between "the interval covers the parameter" and "the parameter lies in the interval". But since we are talking about prediction intervals, does this still apply?

What I'm trying to say: is this sentence statistically correct? I can't judge that immediately and my training on how to talk about p-values is kicking in - but it might be misplaced here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rewrote this (emphasis added here to show the differences):

If we compute a prediction on a new unknown sample, we could center this reference distribution around its predicted value. For some significance level, we now know the range of sample values that "conform" to the variance seen in the reference distribution. That range can define our prediction interval.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of "predicted values" instead of "sample values" was incorrect.

learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Outdated Show resolved Hide resolved
learn/models/conformal-regression/index.qmd Show resolved Hide resolved
@topepo topepo merged commit d11f2e9 into main Aug 17, 2023
@topepo topepo deleted the conformal-reg branch August 17, 2023 03:16
@topepo topepo restored the conformal-reg branch August 17, 2023 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants