[taskcluster 2024-05-24 19:30:36.948Z] Task ID: VtmfOOm3RGimyPbALSbXCw
[taskcluster 2024-05-24 19:30:36.948Z] Worker ID: 3482199237175629728
[taskcluster 2024-05-24 19:30:36.948Z] Worker Group: us-central1-a
[taskcluster 2024-05-24 19:30:36.948Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-05-24 19:30:36.948Z] Worker Pool: translations-1/b-linux-large-gcp
[taskcluster 2024-05-24 19:30:36.948Z] Worker Version: 38.0.5
[taskcluster 2024-05-24 19:30:36.948Z] Public IP: 34.41.161.211
[taskcluster 2024-05-24 19:30:36.948Z] Hostname: translations-1-b-linux-large-gcp-yyxfzbcuto2dmi1kg0gojq
[taskcluster 2024-05-24 19:30:36.948Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-Q_deBg9LRdWXU21Qhn7s6Q" -> /builds/worker/checkouts

[taskcluster 2024-05-24 19:30:37.402Z] Image 'public/image.tar.zst' from task 'Q_deBg9LRdWXU21Qhn7s6Q' loaded.  Using image ID sha256:1e62b245ce9b86147f4b5a4cdcf3708c9a763f8cdaa58df95c74f81427d083ab.
[taskcluster 2024-05-24 19:30:37.455Z] === Task Starting ===
[setup 2024-05-24T19:30:37.747Z] run-task started in /builds/worker
[setup 2024-05-24T19:30:37.747Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BMT_MARIAN=$MOZ_FETCHES_DIR && export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$MOZ_FETCHES_DIR/cuda-toolkit/lib64" && zstd --rm -d $MOZ_FETCHES_DIR/*.zst && $VCS_PATH/pipeline/quantize/export.sh $MOZ_FETCHES_DIR $MOZ_FETCHES_DIR/lex.s2t.pruned $MOZ_FETCHES_DIR/vocab.spm /builds/worker/artifacts
[setup 2024-05-24T19:30:37.747Z] Python version: 3.10.12
[cache 2024-05-24T19:30:37.750Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-05-24T19:30:37.750Z] volume /builds/worker/checkouts is a cache
[setup 2024-05-24T19:30:37.750Z] running as worker:worker
[vcs 2024-05-24T19:30:37.750Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-05-24T19:30:37.752Z] executing ['git', 'fetch', '--tags', '--force', 'https://github.com/mozilla/firefox-translations-training', 'revert_d2g_pool']
[vcs 2024-05-24T19:30:37.966Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-05-24T19:30:37.966Z]  * branch            revert_d2g_pool -> FETCH_HEAD
[vcs 2024-05-24T19:30:37.973Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/mozilla/firefox-translations-training', 'revert_d2g_pool']
[vcs 2024-05-24T19:30:38.223Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-05-24T19:30:38.223Z]  * branch            revert_d2g_pool -> FETCH_HEAD
[vcs 2024-05-24T19:30:38.230Z] executing ['git', 'checkout', '-f', '-B', 'revert_d2g_pool', '9db52842e17a9fa0ececcfc30809128fa1c52288']
[vcs 2024-05-24T19:30:38.235Z] Reset branch 'revert_d2g_pool'
[vcs 2024-05-24T19:30:38.236Z] executing ['git', 'submodule', 'init']
[vcs 2024-05-24T19:30:38.258Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-05-24T19:30:38.356Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-05-24T19:30:38.368Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-05-24T19:30:38.385Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-05-24T19:30:38.400Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-05-24T19:30:38.529Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-05-24T19:30:38.551Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-05-24T19:30:38.551Z] cleaning git checkout...
[vcs 2024-05-24T19:30:38.551Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-05-24T19:30:38.555Z] removing []
[vcs 2024-05-24T19:30:38.555Z] successfully cleaned git checkout!
[vcs 2024-05-24T19:30:38.556Z] TinderboxPrint:<a href='https://github.com/mozilla/firefox-translations-training/commit/9db52842e17a9fa0ececcfc30809128fa1c52288' title='Built from firefox-translations-training commit 9db52842e17a9fa0ececcfc30809128fa1c52288'>9db52842e17a9fa0ececcfc30809128fa1c52288</a>
[setup 2024-05-24T19:30:38.556Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-05-24T19:30:38.556Z] fetching artifacts
[fetches 2024-05-24T19:30:38.556Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Cml6PdlORv69gBkS1uxJ8A/artifacts/public/build/model.intgemm.alphas.bin to /builds/worker/fetches/model.intgemm.alphas.bin
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aoFZ8Q9pT2W7BNWA2I7dyw/artifacts/public/build/lex.s2t.pruned.zst to /builds/worker/fetches/lex.s2t.pruned.zst

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JNk8BMtGQCWDDrPEyNZhng/artifacts/public/build/marian.tar.zst to /builds/worker/fetches/marian.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Cml6PdlORv69gBkS1uxJ8A/artifacts/public/build/model.intgemm.alphas.bin
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aoFZ8Q9pT2W7BNWA2I7dyw/artifacts/public/build/lex.s2t.pruned.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DtQ5zn1HS4-Q7G_cWlRc0Q/artifacts/public/build/cuda-toolkit.tar.zst to /builds/worker/fetches/cuda-toolkit.tar.zst

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6FAEP8XTEKLTldrBixoVg/artifacts/public/build/vocab.spm to /builds/worker/fetches/vocab.spmDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JNk8BMtGQCWDDrPEyNZhng/artifacts/public/build/marian.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DtQ5zn1HS4-Q7G_cWlRc0Q/artifacts/public/build/cuda-toolkit.tar.zst


Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6FAEP8XTEKLTldrBixoVg/artifacts/public/build/vocab.spm
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6FAEP8XTEKLTldrBixoVg/artifacts/public/build/vocab.spm resolved to 252230 bytes with sha256 9d8ac29958816890d18f1ab76bc153d1b73b59a3e447780e1021e936802d92ac in 0.129s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aoFZ8Q9pT2W7BNWA2I7dyw/artifacts/public/build/lex.s2t.pruned.zst resolved to 36931 bytes with sha256 6e6b5496c3bd7e854a502a127cbc7ab4a99ede3957fb622bc1542478aded7f10 in 0.146s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Cml6PdlORv69gBkS1uxJ8A/artifacts/public/build/model.intgemm.alphas.bin resolved to 9081145 bytes with sha256 e17ff7156a370299293d00fe536e3eceed2485ff2eafe2c809aff19a02a32cbe in 0.265s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JNk8BMtGQCWDDrPEyNZhng/artifacts/public/build/marian.tar.zst resolved to 166255744 bytes with sha256 0e8840df993c3267582a1250c5d3b6b8e1c41db02a3ea980c0e7a82a4e302455 in 2.706s
Extracting /builds/worker/fetches/marian.tar.zst to /builds/worker/fetches
/builds/worker/fetches/marian.tar.zst extracted in 1.555s
Removing /builds/worker/fetches/marian.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DtQ5zn1HS4-Q7G_cWlRc0Q/artifacts/public/build/cuda-toolkit.tar.zst resolved to 3295138224 bytes with sha256 e3bdbf0078b8eb8c7804039d6cf6a10b673212bedf1d016a7b4a7eed3ead5f04 in 31.984s
Extracting /builds/worker/fetches/cuda-toolkit.tar.zst to /builds/worker/fetches
/builds/worker/fetches/cuda-toolkit.tar.zst extracted in 16.638s
Removing /builds/worker/fetches/cuda-toolkit.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 48.93197583899996, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-05-24T19:31:27.582Z] finished fetching artifacts
[task 2024-05-24T19:31:27.582Z] executing ['bash', '-c', 'export BMT_MARIAN=$MOZ_FETCHES_DIR && export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$MOZ_FETCHES_DIR/cuda-toolkit/lib64" && zstd --rm -d $MOZ_FETCHES_DIR/*.zst && $VCS_PATH/pipeline/quantize/export.sh $MOZ_FETCHES_DIR $MOZ_FETCHES_DIR/lex.s2t.pruned $MOZ_FETCHES_DIR/vocab.spm /builds/worker/artifacts']
[task 2024-05-24T19:31:27.585Z] 
[task 2024-05-24T19:31:27.585Z] s/lex.s2t.pruned.zst : 0 MB...     
[task 2024-05-24T19:31:27.585Z]                                                                                
[task 2024-05-24T19:31:27.585Z] /builds/worker/fetches/lex.s2t.pruned.zst: 119434 bytes 
[task 2024-05-24T19:31:27.587Z] + set -euo pipefail
[task 2024-05-24T19:31:27.587Z] + echo '###### Exporting a quantized model'
[task 2024-05-24T19:31:27.587Z] ###### Exporting a quantized model
[task 2024-05-24T19:31:27.587Z] + test -v SRC
[task 2024-05-24T19:31:27.587Z] + test -v TRG
[task 2024-05-24T19:31:27.587Z] + test -v BMT_MARIAN
[task 2024-05-24T19:31:27.587Z] + model_dir=/builds/worker/fetches
[task 2024-05-24T19:31:27.587Z] + shortlist=/builds/worker/fetches/lex.s2t.pruned
[task 2024-05-24T19:31:27.587Z] + vocab=/builds/worker/fetches/vocab.spm
[task 2024-05-24T19:31:27.587Z] + output_dir=/builds/worker/artifacts
[task 2024-05-24T19:31:27.587Z] + mkdir -p /builds/worker/artifacts
[task 2024-05-24T19:31:27.588Z] + model=/builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2024-05-24T19:31:27.588Z] + cp /builds/worker/fetches/model.intgemm.alphas.bin /builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2024-05-24T19:31:27.596Z] + pigz /builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2024-05-24T19:31:27.638Z] + shortlist_bin=/builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2024-05-24T19:31:27.638Z] + /builds/worker/fetches/marian-conv --shortlist /builds/worker/fetches/lex.s2t.pruned 50 50 0 --dump /builds/worker/artifacts/lex.50.50.ruen.s2t.bin --vocabs /builds/worker/fetches/vocab.spm /builds/worker/fetches/vocab.spm
[task 2024-05-24T19:31:27.673Z] [2024-05-24 19:31:27] [data] Loading SentencePiece vocabulary from file /builds/worker/fetches/vocab.spm
[task 2024-05-24T19:31:27.676Z] [2024-05-24 19:31:27] [data] Loading SentencePiece vocabulary from file /builds/worker/fetches/vocab.spm
[task 2024-05-24T19:31:27.678Z] [2024-05-24 19:31:27] [data] Importing text lexical shortlist as /builds/worker/fetches/lex.s2t.pruned 50 50 0
[task 2024-05-24T19:31:27.682Z] [2024-05-24 19:31:27] [data] Saving binary shortlist dump to /builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2024-05-24T19:31:27.682Z] [2024-05-24 19:31:27] Finished
[task 2024-05-24T19:31:27.689Z] + pigz /builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2024-05-24T19:31:27.692Z] + vocab_out=/builds/worker/artifacts/vocab.ruen.spm
[task 2024-05-24T19:31:27.692Z] + cp /builds/worker/fetches/vocab.spm /builds/worker/artifacts/vocab.ruen.spm
[task 2024-05-24T19:31:27.693Z] + pigz /builds/worker/artifacts/vocab.ruen.spm
[task 2024-05-24T19:31:27.706Z] + echo '### Export is completed. Results: /builds/worker/artifacts'
[task 2024-05-24T19:31:27.706Z] ### Export is completed. Results: /builds/worker/artifacts
[task 2024-05-24T19:31:27.706Z] + echo '###### Done: Exporting a quantized model'
[task 2024-05-24T19:31:27.706Z] ###### Done: Exporting a quantized model
[fetches 2024-05-24T19:31:27.706Z] removing /builds/worker/fetches
[fetches 2024-05-24T19:31:28.807Z] finished
[taskcluster 2024-05-24 19:31:29.026Z] === Task Finished ===
[taskcluster 2024-05-24 19:31:29.761Z] Successful task run with exit code: 0 completed in 52.815 seconds