Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to run starting from a specific task (fixes #227) #377

Merged
merged 2 commits into from
Feb 14, 2024

Update poetry dependencies to pull in newer taskgraph version

37fbf27
Select commit
Loading
Failed to load commit list.
Merged

Add the ability to run starting from a specific task (fixes #227) #377

Update poetry dependencies to pull in newer taskgraph version
37fbf27
Select commit
Loading
Failed to load commit list.
firefoxci-taskcluster / collect-mono-src-ru-en succeeded Feb 13, 2024 in 1h 29m 40s

FirefoxCI (pull_request)

collect mono src ru-en

Details

View task in Taskcluster
View logs in Taskcluster


[taskcluster 2024-02-13 21:15:19.359Z] Task ID: IfddEhLmRNG1ccEFPQDAHw
[taskcluster 2024-02-13 21:15:19.359Z] Worker ID: 4777733558928565637
[taskcluster 2024-02-13 21:15:19.359Z] Worker Group: us-central1
[taskcluster 2024-02-13 21:15:19.359Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-02-13 21:15:19.359Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-02-13 21:15:19.359Z] Worker Version: 38.0.5
[taskcluster 2024-02-13 21:15:19.359Z] Public IP: 35.222.170.77
[taskcluster 2024-02-13 21:15:19.359Z] Hostname: translations-1-b-linux-large-gcp-300gb-ivrduyy3tzyw9g5fmdiifw
[taskcluster 2024-02-13 21:15:19.359Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-Gudb01f2QISFkQNad5U_NA" -> /builds/worker/checkouts

[taskcluster 2024-02-13 21:15:19.825Z] Image 'public/image.tar.zst' from task 'Gudb01f2QISFkQNad5U_NA' loaded.  Using image ID sha256:654f27369c79de5913668b0c72131803788b72c6634577cb3426bfdc8fa57c40.
[taskcluster 2024-02-13 21:15:19.872Z] === Task Starting ===
[setup 2024-02-13T21:15:20.201Z] run-task started in /builds/worker
[setup 2024-02-13T21:15:20.201Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c zstd -d --rm $MOZ_FETCHES_DIR/file* && $VCS_PATH/pipeline/translate/collect.sh fetches artifacts/mono.en.zst $MOZ_FETCHES_DIR/mono.ru.zst
[setup 2024-02-13T21:15:20.201Z] Python version: 3.10.12
[cache 2024-02-13T21:15:20.203Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-02-13T21:15:20.203Z] volume /builds/worker/checkouts is a cache
[setup 2024-02-13T21:15:20.203Z] running as worker:worker
[vcs 2024-02-13T21:15:20.203Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-02-13T21:15:20.204Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/bhearsum/firefox-translations-training', 'start-specific']
[vcs 2024-02-13T21:15:20.406Z] From https://github.com/bhearsum/firefox-translations-training
[vcs 2024-02-13T21:15:20.406Z]  * branch            start-specific -> FETCH_HEAD
[vcs 2024-02-13T21:15:20.410Z] executing ['git', 'checkout', '-f', '-B', 'start-specific', '37fbf272d7eb316897377144111a3ef057becfd4']
[vcs 2024-02-13T21:15:20.414Z] Reset branch 'start-specific'
[vcs 2024-02-13T21:15:20.414Z] executing ['git', 'submodule', 'init']
[vcs 2024-02-13T21:15:20.431Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-02-13T21:15:20.462Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-02-13T21:15:20.472Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-02-13T21:15:20.481Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-02-13T21:15:20.492Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-02-13T21:15:20.506Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-02-13T21:15:20.516Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-02-13T21:15:20.517Z] cleaning git checkout...
[vcs 2024-02-13T21:15:20.517Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-02-13T21:15:20.519Z] removing []
[vcs 2024-02-13T21:15:20.519Z] successfully cleaned git checkout!
[vcs 2024-02-13T21:15:20.520Z] TinderboxPrint:<a href='https://github.com/bhearsum/firefox-translations-training/commit/37fbf272d7eb316897377144111a3ef057becfd4' title='Built from firefox-translations-training commit 37fbf272d7eb316897377144111a3ef057becfd4'>37fbf272d7eb316897377144111a3ef057becfd4</a>
[setup 2024-02-13T21:15:20.520Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-02-13T21:15:20.520Z] fetching artifacts
[fetches 2024-02-13T21:15:20.520Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EgKyq3SpRDe1QDjywQJWXg/artifacts/public/build/mono.ru.zst to /builds/worker/fetches/mono.ru.zst
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EgKyq3SpRDe1QDjywQJWXg/artifacts/public/build/mono.ru.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOzUfFSdTGKRpg-9jHDU7Q/artifacts/public/build/file.1.out.zst to /builds/worker/fetches/file.1.out.zstattempt 1/5


Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/b0FIRywYQ7STYHVYprm58Q/artifacts/public/build/file.2.out.zst to /builds/worker/fetches/file.2.out.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOzUfFSdTGKRpg-9jHDU7Q/artifacts/public/build/file.1.out.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/b0FIRywYQ7STYHVYprm58Q/artifacts/public/build/file.2.out.zst

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/b0FIRywYQ7STYHVYprm58Q/artifacts/public/build/file.2.out.zst resolved to 38139 bytes with sha256 1735ee5b06bac6b98d689dd45f81329eccb33169427e1ad56e52fefcc59d6c15 in 0.146s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOzUfFSdTGKRpg-9jHDU7Q/artifacts/public/build/file.1.out.zst resolved to 37894 bytes with sha256 0b748fdfa3811f87a386cb81a701c6f2b0408f24f22bddba0a2ab0ae7dde25f0 in 0.160s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EgKyq3SpRDe1QDjywQJWXg/artifacts/public/build/mono.ru.zst resolved to 577499 bytes with sha256 341251ac2d5d92d71f50180d8e028a3dbdfcad4bc2d683924bb345641af3c131 in 0.173s
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.1755646430000013, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-02-13T21:15:20.769Z] finished fetching artifacts
[task 2024-02-13T21:15:20.769Z] executing ['bash', '-c', 'zstd -d --rm $MOZ_FETCHES_DIR/file* && $VCS_PATH/pipeline/translate/collect.sh fetches artifacts/mono.en.zst $MOZ_FETCHES_DIR/mono.ru.zst']
[task 2024-02-13T21:15:20.771Z] 
[task 2024-02-13T21:15:20.773Z] Decompress:  1/ 2 files. Current: .../file.1.out.zst : 0 MB...    
[task 2024-02-13T21:15:20.775Z]                                                                                
[task 2024-02-13T21:15:20.775Z] 
[task 2024-02-13T21:15:20.775Z]                                                                                
[task 2024-02-13T21:15:20.775Z] 2 files decompressed : 5348007 bytes total 
[task 2024-02-13T21:15:20.777Z] + set -euo pipefail
[task 2024-02-13T21:15:20.777Z] + chunks_dir=fetches
[task 2024-02-13T21:15:20.777Z] + output_path=artifacts/mono.en.zst
[task 2024-02-13T21:15:20.777Z] + mono_path=/builds/worker/fetches/mono.ru.zst
[task 2024-02-13T21:15:20.777Z] + COMPRESSION_CMD=zstdmt
[task 2024-02-13T21:15:20.777Z] + echo '### Collecting translations'
[task 2024-02-13T21:15:20.777Z] ### Collecting translations
[task 2024-02-13T21:15:20.777Z] + find fetches -name '*.out'
[task 2024-02-13T21:15:20.777Z] + sort -t . -k2,2n
[task 2024-02-13T21:15:20.777Z] + xargs cat
[task 2024-02-13T21:15:20.777Z] + zstdmt
[task 2024-02-13T21:15:20.787Z] + echo '### Comparing number of sentences in source and artificial target files'
[task 2024-02-13T21:15:20.787Z] ### Comparing number of sentences in source and artificial target files
[task 2024-02-13T21:15:20.787Z] ++ zstdmt -dc /builds/worker/fetches/mono.ru.zst
[task 2024-02-13T21:15:20.787Z] ++ wc -l
[task 2024-02-13T21:15:20.792Z] + src_len=9947
[task 2024-02-13T21:15:20.793Z] ++ zstdmt -dc artifacts/mono.en.zst
[task 2024-02-13T21:15:20.793Z] ++ wc -l
[task 2024-02-13T21:15:20.798Z] + trg_len=9947
[task 2024-02-13T21:15:20.798Z] + '[' 9947 '!=' 9947 ']'
[fetches 2024-02-13T21:15:20.798Z] removing /builds/worker/fetches
[fetches 2024-02-13T21:15:20.799Z] finished
[taskcluster 2024-02-13 21:15:21.013Z] === Task Finished ===
[taskcluster 2024-02-13 21:15:21.210Z] Successful task run with exit code: 0 completed in 1.851 seconds