Dataset size reduction fixed, updated TargetValidator to match signatures

[eddiebergman]

This PR's main goal was to have reduce_dataset_size_if_too_large to use the new splitter but this ended up being easier to correctly type, fix and test as seperated functions.

Part of this was confirming the types it would recieve by going through and fixing the type from entry point AutoML.fit() through to where it called reduced_dataset_size_if_too_large. This lead to having to fix typing in InputValidator as it happens before reduce_dataset_size_if_too_large, where there was some oddities with type casting.

Major changes

• Autosklearn now subsamples less aggressively than before. Previous version didn't account for the new dtype byte sizes once precision was reduced.
• Autosklearn now subsamples using a CustomStratifiedShuffleSplit that accounts for unique labels.
• TargetValidator no longer accepts sparse y, in line with the rest of autosklearn.
• FeatureValidator no correctly returns the types it specifies from fit and transform
  • fit() returns specifically a FeatureValidator and not generically a BaseEstimator
  • transform now correctly specifies the types it can return, also changing list -> DataFrame to list -> np.ndarray
• TargetValidator now correctly returns the types it specifies from fit, _fit, transform, inverse_transform, improving typing in automl.py.
  • fit() and _fit() return specifically a TargetValidator and not generically a BaseEstimator
  • transform and inverse_transform no longer accept y of type spmatrix and always return an ndarray

Minor changes

• AutoML.subsample_if_too_large -> autosklearn.util.data.reduce_dataset_size_if_too_large
• This function was also split up into to separate functions subasmple and reduce_precision
• CustomStratifiedShuffleSplit was fixed to respect training size
• Added specific test to CustomStratifiedShuffleSplit to check for unique labels and respecting training_size
• Type fixes from Automl.fit():start up until Automl.fit():reduce_dataset_size_if_too_large
  • Also removed the typing prefix, eg. typing.List -> List to prevent clutter on long signatures and doc
• Update tests for TargetValidator in line with it's cleaned typing and fixes.
• Use @overload in InputValidator.transform so mypy correctly knows when to expect one return value vs two

Notes:

The types in general have to be updated and confirmed as pointed out in issue #1264

New funcs

# autosklearn/util/data.py

def reduce_dataset_size_if_too_large(
    X: Union[np.ndarray, spmatrix],
    y: np.ndarray,
    memory_limit: int,
    is_classification: bool,
    random_state: Union[int, np.random.RandomState] = None,
    operations: List[str] = ['precision', 'subsample'],
    multiplier: Union[float, int] = 10,
) -> Tuple[Union[np.ndarray, spmatrix], np.ndarray]:
    ...
    
 def reduce_precision(
    X: Union[np.ndarray, spmatrix]
) -> Tuple[Union[np.ndarray, spmatrix], Type]:

def subsample(
    X: SUPPORTED_FEAT_TYPES,
    y: Union[List, np.ndarray, pd.DataFrame, pd.Series],
    is_classification: bool,
    sample_size: Union[float, int],
) -> ) -> Tuple[SUPPORTED_FEAT_TYPES, Union[List, np.ndarray, pd.DataFrame, pd.Series]]:

[codecov]

Codecov Report

Merging #1250 (a1cc277) into development (887bb10) will increase coverage by 0.29%.
The diff coverage is 96.35%.

@@               Coverage Diff               @@
##           development    #1250      +/-   ##
===============================================
+ Coverage        88.02%   88.31%   +0.29%     
===============================================
  Files              140      140              
  Lines            10973    11070      +97     
===============================================
+ Hits              9659     9777     +118     
+ Misses            1314     1293      -21     

Impacted Files	Coverage Δ
autosklearn/evaluation/splitter.py	94.38% <90.00%> (+34.62%)	⬆️
autosklearn/data/validation.py	97.77% <91.66%> (+0.05%)	⬆️
autosklearn/data/target_validator.py	96.66% <94.87%> (-0.42%)	⬇️
autosklearn/automl.py	86.75% <95.45%> (-0.12%)	⬇️
autosklearn/util/data.py	82.08% <98.00%> (+41.54%)	⬆️
autosklearn/data/feature_validator.py	97.50% <100.00%> (ø)
autosklearn/estimators.py	93.45% <100.00%> (+0.03%)	⬆️
autosklearn/experimental/askl2.py	83.33% <100.00%> (ø)
autosklearn/util/logging_.py	88.96% <0.00%> (-1.38%)	⬇️
autosklearn/ensemble_builder.py	85.82% <0.00%> (-1.02%)	⬇️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 887bb10...a1cc277. Read the comment docs.

[mfeurer]

Two more questions :(

Also, do you know what's going on with the unit tests?

[eddiebergman]

Two more questions :(

Also, do you know what's going on with the unit tests?

Hmm, not sure why this didn't fail before but spmatrix.todense gives back an np.matrix. It should have been spmatrix.toarray to give a np.ndarray

[mfeurer]

Just one conflict away from being able to merge this :)

[eddiebergman]

Conflict resolves, just waiting on tests and analysis

[mfeurer]

Just two more minor questions, then we're good to go :)