You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
OpuTrainer buffers too much data to prepare for real training, but our dataset for CI is tiny. This leads to reading the dataset many times and a very long preprocessing. We should adjust settings like --chunk-size--batch-size and --workers.
usage: opustrainer-train [-h] --config CONFIG [--state STATE] [--sync] [--temporary-directory TEMPORARY_DIRECTORY] [--do-not-resume] [--no-shuffle]
[--batch-size BATCH_SIZE] [--chunk-size CHUNK_SIZE] [--workers WORKERS] [--log-level LOG_LEVEL] [--log-file LOG_FILE]
...
Feeds marian tsv data for training.
positional arguments:
trainer Trainer program that gets fed the input. If empty it is read from config.
optional arguments:
-h, --help show this help message and exit
--config CONFIG, -c CONFIG
YML configuration input.
--state STATE, -s STATE
YML state file, defaults to ${CONFIG}.state.
--sync Do not shuffle async
--temporary-directory TEMPORARY_DIRECTORY, -T TEMPORARY_DIRECTORY
Temporary dir, used for shuffling and tracking state
--do-not-resume, -d Do not resume from the previous training state
--no-shuffle, -n Do not shuffle, for debugging
--batch-size BATCH_SIZE, -b BATCH_SIZE
Batch size
--chunk-size CHUNK_SIZE, -B CHUNK_SIZE
Chunk size of batches fed to modifiers
--workers WORKERS, -j WORKERS
Number of workers
--log-level LOG_LEVEL
Set log level. Available levels: DEBUG, INFO, WARNING, ERROR, CRITICAL. Default is INFO
--log-file LOG_FILE, -l LOG_FILE
Target location for logging. Always logs to stderr and optionally to a file.
The text was updated successfully, but these errors were encountered:
OpuTrainer buffers too much data to prepare for real training, but our dataset for CI is tiny. This leads to reading the dataset many times and a very long preprocessing. We should adjust settings like
--chunk-size
--batch-size
and--workers
.Example student task from CI:
https://firefox-ci-tc.services.mozilla.com/tasks/AfSTqkqEQ5-vtmxJjkS3ZA/runs/0/logs/public/logs/live.log
Full help:
The text was updated successfully, but these errors were encountered: