Text Processing

[Louquinze]

Pull Request for Hackathon on Wednesday 17.11.

• rebased the forked branch and merge all current changes into it
• last test might still fail

[codecov]

Codecov Report

Merging #1300 (bc6e883) into development (a9fbd5c) will decrease coverage by 0.43%.
The diff coverage is 100.00%.

❗ Current head bc6e883 differs from pull request most recent head ce1c0d1. Consider uploading reports for the commit ce1c0d1 to get more accurate results

@@               Coverage Diff               @@
##           development    #1300      +/-   ##
===============================================
- Coverage        88.07%   87.63%   -0.44%     
===============================================
  Files              140      146       +6     
  Lines            10993    11285     +292     
===============================================
+ Hits              9682     9890     +208     
- Misses            1311     1395      +84     

Impacted Files	Coverage Δ
...osklearn/metalearning/metafeatures/metafeatures.py	94.47% <ø> (-0.12%)	⬇️
...a_preprocessing/feature_reduction/truncated_svd.py	68.75% <ø> (ø)
...line/components/data_preprocessing/feature_type.py	89.21% <ø> (+1.09%)	⬆️
...components/data_preprocessing/feature_type_text.py	90.00% <ø> (ø)
...nents/data_preprocessing/text_encoding/__init__.py	84.37% <ø> (ø)
...reprocessing/text_encoding/bag_of_word_encoding.py	70.00% <ø> (ø)
...ing/text_encoding/bag_of_word_encoding_distinct.py	58.33% <ø> (ø)
...data_preprocessing/text_encoding/tfidf_encoding.py	71.15% <ø> (ø)
autosklearn/data/feature_validator.py	96.63% <100.00%> (-0.87%)	⬇️
...ponents/feature_preprocessing/select_percentile.py	84.61% <0.00%> (-7.70%)	⬇️
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9fbd5c...ce1c0d1. Read the comment docs.

[mfeurer]

I'm good with this. @eddiebergman could you please have a look at your remaining comments?

[eddiebergman]

Still needs dropna to be done, X[feature] could have na's in them which CountVectorizer can't handle.

[eddiebergman]

Just a note on the sum thing @mfeurer was concerned about, it does exactly what the was done previously and now reimplemented. I don't really mind how it's implemented but it changes nothing but make the code shorter.

[Louquinze]

t

I think it is because sum(*) is a python native method and therefore it is not save how it behaves. but as far as i know "+=" is a numpy operation if both sides are numpy array. But i can also change it to np.sum if this is better

[eddiebergman]

sum actually just calls the underlying __add__ operation, i.e. it called np.ndarray.__add__. This is similar how len just called the underlying objects __len__ method.

sum is basically short hand for

ans = sum(items)
ans = item[0] + item[1] + ... + item[n]

Edit: Turns out i"m completely wrong on this. Use np.sum(iterable) for np arrays.
https://stackoverflow.com/a/49908528

Essentially there was talk of making it work as I described but they didn't go ahead with it.

[Louquinze]

i get the following message
Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.fromiter(generator)) or the python sum builtin instead. return np.sum(self.preprocessor.transform(X[feature]) for feature in X.columns)

[eddiebergman]

i get the following message Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.fromiter(generator)) or the python sum builtin instead. return np.sum(self.preprocessor.transform(X[feature]) for feature in X.columns)

Just do it manually as is, it's fine. Can deal with performance later if needed.

[mfeurer]

Just do it manually as is, it's fine. Can deal with performance later if needed.

I agree on that. Actually, I think the current way to do it is the best way because it is agnostic on the type of array implementation. It would work for numpy arrays, pandas dataframes and scipy sparse matrices. Other solutions such as the built-in sum() are more complicated to understand, and np.sum() explicitly invokes numpy.

Anyway, I'd be happy to merge this version. Anything left from your side @eddiebergman ?

[eddiebergman]

All good from my side :)

... side note: to keep things general with addition (and how I thought sum actually worked)

reduce(operator.add, xs)