Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor the concept of public and private test set #1474

Closed
mfeurer opened this issue May 13, 2022 · 0 comments · Fixed by #1517
Closed

Refactor the concept of public and private test set #1474

mfeurer opened this issue May 13, 2022 · 0 comments · Fixed by #1517
Assignees
Labels
maintenance Internal maintenance
Milestone

Comments

@mfeurer
Copy link
Contributor

mfeurer commented May 13, 2022

Auto-sklearn was built for the AutoML challenges where there were public and private test sets by the names "validation set" and "test set". Since these challenges we have substantially refactored Auto-sklearn and no longer allow the user to pass in the public test set aka "validation set". This leads to confusion as "validation set" is really a misnomer, as inside a machine learning system it usually denotes the dataset on which a model is selected. Therefore we should drop all references that still exist to this old concept of a "validation set", as already being done by @eddiebergman in #1434, which should only be in the evaluation submodule`.

@mfeurer mfeurer added the maintenance Internal maintenance label May 13, 2022
@eddiebergman eddiebergman changed the title Refactor the concept of public and privat test set Refactor the concept of public and private test set May 13, 2022
@eddiebergman eddiebergman added this to the V0.15 milestone Jun 10, 2022
@eddiebergman eddiebergman self-assigned this Jun 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment