Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stable key order for map vectorizers #88

Merged
merged 4 commits into from
Aug 24, 2018
Merged

Stable key order for map vectorizers #88

merged 4 commits into from
Aug 24, 2018

Conversation

tovbinm
Copy link
Collaborator

@tovbinm tovbinm commented Aug 24, 2018

Related issues
Map vectorizers do not order the keys properly.

Describe the proposed solution

  • Sorted map keys in MapVectorizerFuns.getKeyValues function.
  • bonus: use op specs base classes for the vectorizers

Describe alternatives you've considered
N/A

@codecov
Copy link

codecov bot commented Aug 24, 2018

Codecov Report

Merging #88 into master will increase coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #88      +/-   ##
==========================================
+ Coverage   85.97%   85.98%   +0.01%     
==========================================
  Files         294      294              
  Lines        9547     9548       +1     
  Branches      352      336      -16     
==========================================
+ Hits         8208     8210       +2     
+ Misses       1339     1338       -1
Impacted Files Coverage Δ
...p/stages/impl/feature/SmartTextMapVectorizer.scala 100% <100%> (ø) ⬆️
...force/op/stages/impl/feature/OPMapVectorizer.scala 97.82% <100%> (+0.74%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7135926...f60c7cb. Read the comment docs.

@tovbinm tovbinm merged commit fb38b67 into master Aug 24, 2018
@tovbinm tovbinm deleted the mt/sort-map-vecs branch August 24, 2018 23:14
@@ -124,14 +123,14 @@ class IntegralMapVectorizer[T <: OPMap[Long]](uid: String = UID[IntegralMapVecto
def setFillWithMode(shouldFill: Boolean): this.type = set(withConstant, !shouldFill)

override def fillByKey(dataset: Dataset[Seq[T#Value]]): Seq[Map[String, Double]] = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it looks like fillByKey in RealMapVectorizer and IntegralMapVectorizer have a very small diff. Maybe refactor.

ericwayman pushed a commit that referenced this pull request Feb 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants