Code

Sparkdeduplication#429

Code

Open

axnow wants to merge 21 commits intoCeON:masterCeON/CoAnSys:masterfrom axnow:sparkdeduplicationCopy head branch name to clipboard

+1,841

Commits

Commits on Mar 17, 2017

• Initial version of spark-based document deduplication. It contains

  Show description for eafc85d
  axnow

  committed
  eafc85d

Commits on Apr 4, 2017

• Work on the algorithm which splits large clusters among the cluster

  Show description for e95a8cc
  axnow

  committed
  e95a8cc

Commits on Apr 7, 2017

• Complete version with tiled comparison task.

  Show description for efb3fc2
  axnow

  committed
  efb3fc2

Commits on Apr 14, 2017

• Work on tiled optimalization.

  Show description for fab0915
  axnow

  committed
  fab0915

Commits on Jun 23, 2017

• Stable version, does proper job within 2.5h on full data set.

  Show description for 6180bba
  axnow

  committed
  6180bba

Commits on Jun 26, 2017

• Added options parsing from command line to control app behaviour.

  Show description for 28834d2
  axnow

  committed
  28834d2

Commits on Jul 11, 2017

• Added dependency for the scopt.

  axnow

  committed
  3a749d4

Commits on Jul 14, 2017

• Task tiling class rewritten to scala, with tests.

  axnow

  committed
  0cfdcbb

Commits on Jul 23, 2017

• Scala version.

  axnow

  committed
  2e0c205

Commits on Jul 24, 2017

• Fixed oozie workflow building.

  axnow

  committed
  1a2c5dd

• Cleaning up project files

  axnow

  committed
  ca4592b

Commits on Jul 25, 2017

• Initial version of spark-based document deduplication. It contains

  Show description for ece39dc
  axnow

  committed
  ece39dc

• Work on the algorithm which splits large clusters among the cluster

  Show description for c056a0b
  axnow

  committed
  c056a0b

• Complete version with tiled comparison task.

  Show description for e7ad7aa
  axnow

  committed
  e7ad7aa

• Work on tiled optimalization.

  Show description for 0cf6672
  axnow

  committed
  0cf6672

• Stable version, does proper job within 2.5h on full data set.

  Show description for 013b53c
  axnow

  committed
  013b53c

• Added options parsing from command line to control app behaviour.

  Show description for ac56042
  axnow

  committed
  ac56042

• Added dependency for the scopt.

  Show description for 81f6509
  axnow

  committed
  81f6509

• Scala version.

  axnow

  committed
  cd1014c

• Fixed oozie workflow building.

  Show description for 221cd52
  axnow

  committed
  221cd52

• Merge branch 'sparkdeduplication' of https://github.com/axnow/CoAnSys into sparkdeduplication

  axnow

  committed
  20da6f8