Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust OpusTrainer settings in CI #455

Open
Tracked by #453
eu9ene opened this issue Feb 16, 2024 · 0 comments
Open
Tracked by #453

Adjust OpusTrainer settings in CI #455

eu9ene opened this issue Feb 16, 2024 · 0 comments
Labels
cost & perf Speeding up and lowering cost for the pipeline

Comments

@eu9ene
Copy link
Collaborator

eu9ene commented Feb 16, 2024

OpuTrainer buffers too much data to prepare for real training, but our dataset for CI is tiny. This leads to reading the dataset many times and a very long preprocessing. We should adjust settings like --chunk-size --batch-size and --workers.

Example student task from CI:
https://firefox-ci-tc.services.mozilla.com/tasks/AfSTqkqEQ5-vtmxJjkS3ZA/runs/0/logs/public/logs/live.log

Full help:

usage: opustrainer-train [-h] --config CONFIG [--state STATE] [--sync] [--temporary-directory TEMPORARY_DIRECTORY] [--do-not-resume] [--no-shuffle]
                         [--batch-size BATCH_SIZE] [--chunk-size CHUNK_SIZE] [--workers WORKERS] [--log-level LOG_LEVEL] [--log-file LOG_FILE]
                         ...

Feeds marian tsv data for training.

positional arguments:
  trainer               Trainer program that gets fed the input. If empty it is read from config.

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG, -c CONFIG
                        YML configuration input.
  --state STATE, -s STATE
                        YML state file, defaults to ${CONFIG}.state.
  --sync                Do not shuffle async
  --temporary-directory TEMPORARY_DIRECTORY, -T TEMPORARY_DIRECTORY
                        Temporary dir, used for shuffling and tracking state
  --do-not-resume, -d   Do not resume from the previous training state
  --no-shuffle, -n      Do not shuffle, for debugging
  --batch-size BATCH_SIZE, -b BATCH_SIZE
                        Batch size
  --chunk-size CHUNK_SIZE, -B CHUNK_SIZE
                        Chunk size of batches fed to modifiers
  --workers WORKERS, -j WORKERS
                        Number of workers
  --log-level LOG_LEVEL
                        Set log level. Available levels: DEBUG, INFO, WARNING, ERROR, CRITICAL. Default is INFO
  --log-file LOG_FILE, -l LOG_FILE
                        Target location for logging. Always logs to stderr and optionally to a file.

@eu9ene eu9ene added the cost & perf Speeding up and lowering cost for the pipeline label Feb 16, 2024
@eu9ene eu9ene changed the title Adjust OpusTrainer setting in CI Adjust OpusTrainer settings in CI Feb 16, 2024
nordzilla pushed a commit that referenced this issue Sep 19, 2024
Bumps [bergamot-translator-tests](https://github.com/browsermt/bergamot-translator-tests) from `7984d14` to `a04432d`.
- [Commits](browsermt/bergamot-translator-tests@7984d14...a04432d)

---
updated-dependencies:
- dependency-name: bergamot-translator-tests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cost & perf Speeding up and lowering cost for the pipeline
Projects
None yet
Development

No branches or pull requests

1 participant