Code

Integrate OpusCleaner#163

Code

Merged

eu9ene merged 49 commits intomozilla:mainmozilla/translations:mainfrom eu9ene:opus_cleanereu9ene/bergamot-training:opus_cleanerCopy head branch name to clipboardSep 26, 2023

+979-68

Commits

Commits on Aug 17, 2023

• Initial integration of opus cleaner

  eu9ene

  committed
  fac320c

  Copy full SHA for fac320c

Commits on Aug 18, 2023

• Support custom filters

  eu9ene

  committed
  c5f2332

  Copy full SHA for c5f2332

• Use opus cleaner in pipeline

  eu9ene

  committed
  0910cd5

  Copy full SHA for 0910cd5

• Fix env

  eu9ene

  committed
  3df375b

  Copy full SHA for 3df375b

• Fix filter generation

  eu9ene

  committed
  966d3a1

  Copy full SHA for 966d3a1

• Merge branch 'main' into opus_cleaner

  eu9ene

  committed
  50bceb1

  Copy full SHA for 50bceb1

• Add more rules

  eu9ene

  committed
  d3ba834

  Copy full SHA for d3ba834

• Fix elrc filter

  eu9ene

  committed
  3f51c1c

  Copy full SHA for 3f51c1c

• Fix env

  eu9ene

  committed
  6554e50

  Copy full SHA for 6554e50

• Fix frequent patterns filter

  eu9ene

  committed
  b01df51

  Copy full SHA for b01df51

Commits on Sep 7, 2023

• Switch to reading from stdin

  eu9ene

  committed
  40a82d6

  Copy full SHA for 40a82d6

• Add a feature flag for opus cleaner

  eu9ene

  committed
  d691985

  Copy full SHA for d691985

Commits on Sep 8, 2023

• Fix condition

  eu9ene

  committed
  216a485

  Copy full SHA for 216a485

• Add extra test for non empty files

  eu9ene

  committed
  6b62166

  Copy full SHA for 6b62166

Commits on Sep 20, 2023

• Merge branch 'main' into opus_cleaner

  Show description for cbc86f0
  eu9ene

  committed
  cbc86f0

  Copy full SHA for cbc86f0

• Integrate with TC

  eu9ene

  committed
  21b0025

  Copy full SHA for 21b0025

• Run linter

  eu9ene

  committed
  831ed74

  Copy full SHA for 831ed74

• Fix step config

  eu9ene

  committed
  8c98421

  Copy full SHA for 8c98421

• Fix step config

  eu9ene

  committed
  3316feb

  Copy full SHA for 3316feb

• Fix step config

  eu9ene

  committed
  25026e4

  Copy full SHA for 25026e4

• Fix step config

  eu9ene

  committed
  85dce3a

  Copy full SHA for 85dce3a

Commits on Sep 21, 2023

• Merge branch 'main' into opus_cleaner

  eu9ene

  authored
  5972407

  Copy full SHA for 5972407

Commits on Sep 22, 2023

• Fix command

  eu9ene

  committed
  929fe8e

  Copy full SHA for 929fe8e

• Merge remote-tracking branch 'origin/opus_cleaner' into opus_cleaner

  eu9ene

  committed
  8787191

  Copy full SHA for 8787191

• Fix path

  eu9ene

  committed
  80f04d0

  Copy full SHA for 80f04d0

• Update OpusCleaner

  eu9ene

  committed
  84060a7

  Copy full SHA for 84060a7

• Remove warning

  eu9ene

  committed
  1d7c1ad

  Copy full SHA for 1d7c1ad

• Log filtered length

  eu9ene

  committed
  78777a6

  Copy full SHA for 78777a6

• Add opuscleaner logs

  eu9ene

  committed
  b655c8d

  Copy full SHA for b655c8d

• Add comments

  eu9ene

  committed
  afdbec8

  Copy full SHA for afdbec8

• Fix using custom filters

  eu9ene

  committed
  5324302

  Copy full SHA for 5324302

• Merge branch 'main' into opus_cleaner

  eu9ene

  committed
  aea4901

  Copy full SHA for aea4901

• Extract function

  eu9ene

  committed
  82599ef

  Copy full SHA for 82599ef

• Change the CI target back

  eu9ene

  committed
  6bd44d6

  Copy full SHA for 6bd44d6

• Fix file path

  eu9ene

  committed
  b06107c

  Copy full SHA for b06107c

Commits on Sep 26, 2023

• Replace conda with poetry

  eu9ene

  committed
  cb13c31

  Copy full SHA for cb13c31

• Add doc

  eu9ene

  committed
  53b82ad

  Copy full SHA for 53b82ad

• Add more comments

  eu9ene

  committed
  cb6c414

  Copy full SHA for cb6c414

• Rename example filter

  eu9ene

  committed
  9377a63

  Copy full SHA for 9377a63

• Test corpus

  eu9ene

  committed
  0ddcd8d

  Copy full SHA for 0ddcd8d

• Fix filter name

  eu9ene

  committed
  b73e1ad

  Copy full SHA for b73e1ad

• Use opus dataset instead of mtdata

  eu9ene

  committed
  282284d

  Copy full SHA for 282284d

• Make CI faster

  eu9ene

  committed
  09d8ee9

  Copy full SHA for 09d8ee9

• Add sections to makefile

  eu9ene

  committed
  82aa769

  Copy full SHA for 82aa769

• Fix custom filter search

  eu9ene

  committed
  20fdab4

  Copy full SHA for 20fdab4

• Redirect stderr to stdout

  eu9ene

  committed
  81f3482

  Copy full SHA for 81f3482

• Fix usage of custom config

  eu9ene

  committed
  b24d255

  Copy full SHA for b24d255

• Fix config name

  eu9ene

  committed
  aa48442

  Copy full SHA for aa48442

• Change back to all

  eu9ene

  committed
  02d5ebd

  Copy full SHA for 02d5ebd