Change treatment of generic column type object
#1415
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ng/feature_reduction` to `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction`. also rename corresponding feature reduction class FeatureReduction to TextFeatureReduction. `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction/truncated_svd.py:TextFeatureReduction` This includes adapting all *csv and *json participating in metalearning The "real" changes are limited to 1. truncated_svd.py 2. feature_type_text.py
…ng/feature_reduction` to `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction`. also rename corresponding feature reduction class FeatureReduction to TextFeatureReduction. `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction/truncated_svd.py:TextFeatureReduction` This includes adapting all *csv and *json participating in metalearning The "real" changes are limited to 1. truncated_svd.py 2. feature_type_text.py
The `object` type will be treated as `string` in the future.
The `object` type will be treated as `string` in the future.
Codecov Report
@@ Coverage Diff @@
## development #1415 +/- ##
===============================================
+ Coverage 84.51% 84.52% +0.01%
===============================================
Files 146 146
Lines 11283 11285 +2
Branches 1929 1929
===============================================
+ Hits 9536 9539 +3
+ Misses 1232 1230 -2
- Partials 515 516 +1 |
The `object` type will be treated as `string` in the future.
mfeurer
previously requested changes
Mar 3, 2022
| @@ -327,7 +325,7 @@ def get_feat_type_from_columns( | |||
| else: | |||
| raise ValueError( | |||
| "Input Column {} has unsupported dtype {}. " | |||
| "Supported column types are categorical/bool/numerical dtypes. " | |||
| "Supported column types are categorical/bool/numerical/string dtypes. " # noqa: E501 | |||
There was a problem hiding this comment.
I don't think this is necessary. Please reformat the string so the lines fit within the line limit.
| ) | ||
| X[column] = X[column].astype("string") |
There was a problem hiding this comment.
Does this work for random objects? Could we have a test that the feature validator correctly handles random objects? In general, could you please extend the tests under test/test_data/test_feature_validator.py?
There was a problem hiding this comment.
i will check the behavior and then update test/test_data/test_feature_validator.py
There was a problem hiding this comment.
class Dummy:
def __init__(self, x):
self.x = x
def __call__(self):
print(self.x)
def dummy_func(self):
for i in range(100):
print("do something 100 times")
dummy = Dummy(1)
dummy_2 = Dummy(2)
dummy_3 = Dummy(3)
df = pd.DataFrame({"Test object": [dummy], "Test list of objects": [[dummy_2, dummy_3]]})
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1 entries, 0 to 0
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Test object 1 non-null object
1 Test list of objects 1 non-null object
dtypes: object(2)
memory usage: 144.0+ bytes
df = df.astype({"Test object": "string", "Test list of objects": "string"})
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1 entries, 0 to 0
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Test object 1 non-null string
1 Test list of objects 1 non-null string
dtypes: string(2)
memory usage: 144.0 bytes
print(df)
Test object Test list of objects
0 <__main__.Dummy object at 0x7fcfbcf880a0> [<__main__.Dummy object at 0x7fcff7cb2ac0>, <_...The `object` type will be treated as `string` in the future. add new test case to `test_feature_validator.py`
The `object` type will be treated as `string` in the future. add new test case to `test_feature_validator.py`
The `object` type will be treated as `string` in the future. add new test case to `test_feature_validator.py`
eddiebergman
approved these changes
Mar 15, 2022
just merge, i will add the docstring in a new PR |
Louquinze
dismissed
mfeurer’s stale review
Currently on vacation, so i can not merge the PR
eddiebergman
pushed a commit
that referenced
this pull request
Aug 18, 2022
* rename `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/feature_reduction` to `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction`. also rename corresponding feature reduction class FeatureReduction to TextFeatureReduction. `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction/truncated_svd.py:TextFeatureReduction` This includes adapting all *csv and *json participating in metalearning The "real" changes are limited to 1. truncated_svd.py 2. feature_type_text.py * rename `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/feature_reduction` to `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction`. also rename corresponding feature reduction class FeatureReduction to TextFeatureReduction. `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction/truncated_svd.py:TextFeatureReduction` This includes adapting all *csv and *json participating in metalearning The "real" changes are limited to 1. truncated_svd.py 2. feature_type_text.py * change treatment of generic column dtype `object` for pandas dataframes. The `object` type will be treated as `string` in the future. * change treatment of generic column dtype `object` for pandas dataframes. The `object` type will be treated as `string` in the future. * change treatment of generic column dtype `object` for pandas dataframes. The `object` type will be treated as `string` in the future. * change treatment of generic column dtype `object` for pandas dataframes. The `object` type will be treated as `string` in the future. add new test case to `test_feature_validator.py` * change treatment of generic column dtype `object` for pandas dataframes. The `object` type will be treated as `string` in the future. add new test case to `test_feature_validator.py` * change treatment of generic column dtype `object` for pandas dataframes. The `object` type will be treated as `string` in the future. add new test case to `test_feature_validator.py`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the treatment of columns with dtype
object. This columns will be treated asstring.