Add the ability to run starting from a specific task (fixes #227) #377
firefoxci-taskcluster / collect-mono-src-ru-en
succeeded
Feb 13, 2024 in 1h 29m 40s
FirefoxCI (pull_request)
collect mono src ru-en
Details
View task in Taskcluster
View logs in Taskcluster
[taskcluster 2024-02-13 21:15:19.359Z] Task ID: IfddEhLmRNG1ccEFPQDAHw
[taskcluster 2024-02-13 21:15:19.359Z] Worker ID: 4777733558928565637
[taskcluster 2024-02-13 21:15:19.359Z] Worker Group: us-central1
[taskcluster 2024-02-13 21:15:19.359Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-02-13 21:15:19.359Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-02-13 21:15:19.359Z] Worker Version: 38.0.5
[taskcluster 2024-02-13 21:15:19.359Z] Public IP: 35.222.170.77
[taskcluster 2024-02-13 21:15:19.359Z] Hostname: translations-1-b-linux-large-gcp-300gb-ivrduyy3tzyw9g5fmdiifw
[taskcluster 2024-02-13 21:15:19.359Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-Gudb01f2QISFkQNad5U_NA" -> /builds/worker/checkouts
[taskcluster 2024-02-13 21:15:19.825Z] Image 'public/image.tar.zst' from task 'Gudb01f2QISFkQNad5U_NA' loaded. Using image ID sha256:654f27369c79de5913668b0c72131803788b72c6634577cb3426bfdc8fa57c40.
[taskcluster 2024-02-13 21:15:19.872Z] === Task Starting ===
[setup 2024-02-13T21:15:20.201Z] run-task started in /builds/worker
[setup 2024-02-13T21:15:20.201Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c zstd -d --rm $MOZ_FETCHES_DIR/file* && $VCS_PATH/pipeline/translate/collect.sh fetches artifacts/mono.en.zst $MOZ_FETCHES_DIR/mono.ru.zst
[setup 2024-02-13T21:15:20.201Z] Python version: 3.10.12
[cache 2024-02-13T21:15:20.203Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-02-13T21:15:20.203Z] volume /builds/worker/checkouts is a cache
[setup 2024-02-13T21:15:20.203Z] running as worker:worker
[vcs 2024-02-13T21:15:20.203Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-02-13T21:15:20.204Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/bhearsum/firefox-translations-training', 'start-specific']
[vcs 2024-02-13T21:15:20.406Z] From https://github.com/bhearsum/firefox-translations-training
[vcs 2024-02-13T21:15:20.406Z] * branch start-specific -> FETCH_HEAD
[vcs 2024-02-13T21:15:20.410Z] executing ['git', 'checkout', '-f', '-B', 'start-specific', '37fbf272d7eb316897377144111a3ef057becfd4']
[vcs 2024-02-13T21:15:20.414Z] Reset branch 'start-specific'
[vcs 2024-02-13T21:15:20.414Z] executing ['git', 'submodule', 'init']
[vcs 2024-02-13T21:15:20.431Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-02-13T21:15:20.462Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-02-13T21:15:20.472Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-02-13T21:15:20.481Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-02-13T21:15:20.492Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-02-13T21:15:20.506Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-02-13T21:15:20.516Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-02-13T21:15:20.517Z] cleaning git checkout...
[vcs 2024-02-13T21:15:20.517Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-02-13T21:15:20.519Z] removing []
[vcs 2024-02-13T21:15:20.519Z] successfully cleaned git checkout!
[vcs 2024-02-13T21:15:20.520Z] TinderboxPrint:<a href='https://github.com/bhearsum/firefox-translations-training/commit/37fbf272d7eb316897377144111a3ef057becfd4' title='Built from firefox-translations-training commit 37fbf272d7eb316897377144111a3ef057becfd4'>37fbf272d7eb316897377144111a3ef057becfd4</a>
[setup 2024-02-13T21:15:20.520Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-02-13T21:15:20.520Z] fetching artifacts
[fetches 2024-02-13T21:15:20.520Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EgKyq3SpRDe1QDjywQJWXg/artifacts/public/build/mono.ru.zst to /builds/worker/fetches/mono.ru.zst
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EgKyq3SpRDe1QDjywQJWXg/artifacts/public/build/mono.ru.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOzUfFSdTGKRpg-9jHDU7Q/artifacts/public/build/file.1.out.zst to /builds/worker/fetches/file.1.out.zstattempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/b0FIRywYQ7STYHVYprm58Q/artifacts/public/build/file.2.out.zst to /builds/worker/fetches/file.2.out.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOzUfFSdTGKRpg-9jHDU7Q/artifacts/public/build/file.1.out.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/b0FIRywYQ7STYHVYprm58Q/artifacts/public/build/file.2.out.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/b0FIRywYQ7STYHVYprm58Q/artifacts/public/build/file.2.out.zst resolved to 38139 bytes with sha256 1735ee5b06bac6b98d689dd45f81329eccb33169427e1ad56e52fefcc59d6c15 in 0.146s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOzUfFSdTGKRpg-9jHDU7Q/artifacts/public/build/file.1.out.zst resolved to 37894 bytes with sha256 0b748fdfa3811f87a386cb81a701c6f2b0408f24f22bddba0a2ab0ae7dde25f0 in 0.160s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EgKyq3SpRDe1QDjywQJWXg/artifacts/public/build/mono.ru.zst resolved to 577499 bytes with sha256 341251ac2d5d92d71f50180d8e028a3dbdfcad4bc2d683924bb345641af3c131 in 0.173s
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.1755646430000013, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-02-13T21:15:20.769Z] finished fetching artifacts
[task 2024-02-13T21:15:20.769Z] executing ['bash', '-c', 'zstd -d --rm $MOZ_FETCHES_DIR/file* && $VCS_PATH/pipeline/translate/collect.sh fetches artifacts/mono.en.zst $MOZ_FETCHES_DIR/mono.ru.zst']
[task 2024-02-13T21:15:20.771Z]
[task 2024-02-13T21:15:20.773Z] Decompress: 1/ 2 files. Current: .../file.1.out.zst : 0 MB...
[task 2024-02-13T21:15:20.775Z]
[task 2024-02-13T21:15:20.775Z]
[task 2024-02-13T21:15:20.775Z]
[task 2024-02-13T21:15:20.775Z] 2 files decompressed : 5348007 bytes total
[task 2024-02-13T21:15:20.777Z] + set -euo pipefail
[task 2024-02-13T21:15:20.777Z] + chunks_dir=fetches
[task 2024-02-13T21:15:20.777Z] + output_path=artifacts/mono.en.zst
[task 2024-02-13T21:15:20.777Z] + mono_path=/builds/worker/fetches/mono.ru.zst
[task 2024-02-13T21:15:20.777Z] + COMPRESSION_CMD=zstdmt
[task 2024-02-13T21:15:20.777Z] + echo '### Collecting translations'
[task 2024-02-13T21:15:20.777Z] ### Collecting translations
[task 2024-02-13T21:15:20.777Z] + find fetches -name '*.out'
[task 2024-02-13T21:15:20.777Z] + sort -t . -k2,2n
[task 2024-02-13T21:15:20.777Z] + xargs cat
[task 2024-02-13T21:15:20.777Z] + zstdmt
[task 2024-02-13T21:15:20.787Z] + echo '### Comparing number of sentences in source and artificial target files'
[task 2024-02-13T21:15:20.787Z] ### Comparing number of sentences in source and artificial target files
[task 2024-02-13T21:15:20.787Z] ++ zstdmt -dc /builds/worker/fetches/mono.ru.zst
[task 2024-02-13T21:15:20.787Z] ++ wc -l
[task 2024-02-13T21:15:20.792Z] + src_len=9947
[task 2024-02-13T21:15:20.793Z] ++ zstdmt -dc artifacts/mono.en.zst
[task 2024-02-13T21:15:20.793Z] ++ wc -l
[task 2024-02-13T21:15:20.798Z] + trg_len=9947
[task 2024-02-13T21:15:20.798Z] + '[' 9947 '!=' 9947 ']'
[fetches 2024-02-13T21:15:20.798Z] removing /builds/worker/fetches
[fetches 2024-02-13T21:15:20.799Z] finished
[taskcluster 2024-02-13 21:15:21.013Z] === Task Finished ===
[taskcluster 2024-02-13 21:15:21.210Z] Successful task run with exit code: 0 completed in 1.851 seconds
Loading