[taskcluster 2023-12-20 16:30:08.725Z] Task ID: H00c-8zJQoWqnOF43LTTsA
[taskcluster 2023-12-20 16:30:08.725Z] Worker ID: 4482306338003529605
[taskcluster 2023-12-20 16:30:08.725Z] Worker Group: us-west1
[taskcluster 2023-12-20 16:30:08.725Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2023-12-20 16:30:08.725Z] Worker Pool: translations-1/b-linux-large-gcp
[taskcluster 2023-12-20 16:30:08.725Z] Worker Version: 38.0.5
[taskcluster 2023-12-20 16:30:08.725Z] Public IP: 34.168.52.15
[taskcluster 2023-12-20 16:30:08.725Z] Hostname: translations-1-b-linux-large-gcp-deghhyqqthslh7jafrxg8a
[taskcluster 2023-12-20 16:30:08.725Z] using cache "translations-level-1-checkouts-v3-58974d7dcf0417b3fe53-ARGg7JgtR3C0C2BCSaAWqA" -> /builds/worker/checkouts

[taskcluster 2023-12-20 16:30:09.101Z] Downloading artifact "public/image.tar.zst" from task ID: ARGg7JgtR3C0C2BCSaAWqA.
[taskcluster 2023-12-20 16:30:14.103Z] Download Progress: 57.89%
[taskcluster 2023-12-20 16:30:17.392Z] Downloaded artifact successfully.
[taskcluster 2023-12-20 16:30:17.392Z] Downloaded 251.882 mb
[taskcluster 2023-12-20 16:30:17.392Z] Decompressing downloaded image
[taskcluster 2023-12-20 16:30:19.054Z] Loading docker image from downloaded archive.
[taskcluster 2023-12-20 16:30:29.491Z] Image 'public/image.tar.zst' from task 'ARGg7JgtR3C0C2BCSaAWqA' loaded.  Using image ID sha256:fa53620b2ece3428f41eb15ffb863544aa69d70369d1fa2d4412339db9f7115c.
[taskcluster 2023-12-20 16:30:29.603Z] === Task Starting ===
[setup 2023-12-20T16:30:31.261Z] run-task started in /builds/worker
[setup 2023-12-20T16:30:31.261Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BMT_MARIAN=$MOZ_FETCHES_DIR && export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$MOZ_FETCHES_DIR/cuda-toolkit/lib64" && zstd --rm -d $MOZ_FETCHES_DIR/*.zst && $VCS_PATH/pipeline/quantize/export.sh $MOZ_FETCHES_DIR $MOZ_FETCHES_DIR/lex.s2t.pruned $MOZ_FETCHES_DIR/vocab.spm /builds/worker/artifacts
[setup 2023-12-20T16:30:31.261Z] Python version: 3.10.12
[cache 2023-12-20T16:30:31.263Z] cache /builds/worker/checkouts is empty; writing requirements: gid=1000 uid=1000 version=1
[volume 2023-12-20T16:30:31.263Z] volume /builds/worker/checkouts is a cache
[setup 2023-12-20T16:30:31.263Z] running as worker:worker
[vcs 2023-12-20T16:30:31.263Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2023-12-20T16:30:31.265Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/builds/worker/checkouts/vcs']
[vcs 2023-12-20T16:30:31.267Z] Cloning into '/builds/worker/checkouts/vcs'...
[vcs 2023-12-20T16:30:32.080Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/bhearsum/firefox-translations-training', 'taskgraph-7']
[vcs 2023-12-20T16:30:32.470Z] From https://github.com/bhearsum/firefox-translations-training
[vcs 2023-12-20T16:30:32.470Z]  * branch            taskgraph-7 -> FETCH_HEAD
[vcs 2023-12-20T16:30:32.473Z] executing ['git', 'checkout', '-f', '-B', 'taskgraph-7', '084cd68b442168a104ef509f6ea7bf320b139e2a']
[vcs 2023-12-20T16:30:32.513Z] Switched to a new branch 'taskgraph-7'
[vcs 2023-12-20T16:30:32.513Z] executing ['git', 'submodule', 'init']
[vcs 2023-12-20T16:30:32.535Z] Submodule '3rd_party/browsermt-marian-dev' (https://github.com/browsermt/marian-dev) registered for path '3rd_party/browsermt-marian-dev'
[vcs 2023-12-20T16:30:32.535Z] Submodule 'extract-lex' (https://github.com/marian-nmt/extract-lex) registered for path '3rd_party/extract-lex'
[vcs 2023-12-20T16:30:32.536Z] Submodule 'fast_align' (https://github.com/clab/fast_align) registered for path '3rd_party/fast_align'
[vcs 2023-12-20T16:30:32.536Z] Submodule '3rd_party/kenlm' (https://github.com/kpu/kenlm) registered for path '3rd_party/kenlm'
[vcs 2023-12-20T16:30:32.536Z] Submodule '3rd_party/marian-dev' (https://github.com/marian-nmt/marian-dev) registered for path '3rd_party/marian-dev'
[vcs 2023-12-20T16:30:32.537Z] Submodule '3rd_party/preprocess' (https://github.com/kpu/preprocess.git) registered for path '3rd_party/preprocess'
[vcs 2023-12-20T16:30:32.537Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2023-12-20T16:30:32.559Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/browsermt-marian-dev'...
[vcs 2023-12-20T16:30:33.933Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/extract-lex'...
[vcs 2023-12-20T16:30:35.332Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/fast_align'...
[vcs 2023-12-20T16:30:35.743Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/kenlm'...
[vcs 2023-12-20T16:30:36.454Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/marian-dev'...
[vcs 2023-12-20T16:30:38.114Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/preprocess'...
[vcs 2023-12-20T16:30:38.690Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2023-12-20T16:30:38.700Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2023-12-20T16:30:38.710Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2023-12-20T16:30:38.737Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2023-12-20T16:30:38.849Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2023-12-20T16:30:38.868Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2023-12-20T16:30:38.869Z] cleaning git checkout...
[vcs 2023-12-20T16:30:38.869Z] executing ['git', 'clean', '-nxdff']
[vcs 2023-12-20T16:30:38.871Z] removing []
[vcs 2023-12-20T16:30:38.872Z] successfully cleaned git checkout!
[vcs 2023-12-20T16:30:38.873Z] TinderboxPrint:<a href='https://github.com/bhearsum/firefox-translations-training/commit/084cd68b442168a104ef509f6ea7bf320b139e2a' title='Built from firefox-translations-training commit 084cd68b442168a104ef509f6ea7bf320b139e2a'>084cd68b442168a104ef509f6ea7bf320b139e2a</a>
[setup 2023-12-20T16:30:38.873Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2023-12-20T16:30:38.873Z] fetching artifacts
[fetches 2023-12-20T16:30:38.873Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6O37YcLTh-_JWmTLkRF1w/artifacts/public/build/lex.s2t.pruned.zst to /builds/worker/fetches/lex.s2t.pruned.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F8Qv8l7jSFuyty-ZCVssrg/artifacts/public/build/model.intgemm.alphas.bin to /builds/worker/fetches/model.intgemm.alphas.bin
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6O37YcLTh-_JWmTLkRF1w/artifacts/public/build/lex.s2t.pruned.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F8Qv8l7jSFuyty-ZCVssrg/artifacts/public/build/model.intgemm.alphas.binattempt 1/5

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OMJ2WdONTXGcqjwZqrmMjQ/artifacts/public/build/marian.tar.zst to /builds/worker/fetches/marian.tar.zst
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UGkRJAPkTfmkxcB8UlS-Cw/artifacts/public/build/cuda-toolkit.tar.zst to /builds/worker/fetches/cuda-toolkit.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UY1oX_H_SRiogc2_-3fL6w/artifacts/public/build/vocab.spm to /builds/worker/fetches/vocab.spmDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OMJ2WdONTXGcqjwZqrmMjQ/artifacts/public/build/marian.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UGkRJAPkTfmkxcB8UlS-Cw/artifacts/public/build/cuda-toolkit.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UY1oX_H_SRiogc2_-3fL6w/artifacts/public/build/vocab.spm
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6O37YcLTh-_JWmTLkRF1w/artifacts/public/build/lex.s2t.pruned.zst resolved to 8384 bytes with sha256 0ce260428f71e9f26b84a7b6e826f92b4fbff80c79f54d11438d047916c0e609 in 0.186s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UY1oX_H_SRiogc2_-3fL6w/artifacts/public/build/vocab.spm resolved to 252659 bytes with sha256 55399c37c1432c4d2a11bd342c3901a624b8822ca03e5ce01c92378bf38d562c in 0.216s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F8Qv8l7jSFuyty-ZCVssrg/artifacts/public/build/model.intgemm.alphas.bin resolved to 9080992 bytes with sha256 cd80a2b86c86784a7fd4ea74af7e692b2abca40de6e446a013dcee8b980f0e4b in 0.479s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OMJ2WdONTXGcqjwZqrmMjQ/artifacts/public/build/marian.tar.zst resolved to 166202633 bytes with sha256 c63aa4b3dd8218e29047854be8b2d1fd63231742ad747552ceb171a2862e5f5f in 3.058s
Extracting /builds/worker/fetches/marian.tar.zst to /builds/worker/fetches
/builds/worker/fetches/marian.tar.zst extracted in 1.493s
Removing /builds/worker/fetches/marian.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UGkRJAPkTfmkxcB8UlS-Cw/artifacts/public/build/cuda-toolkit.tar.zst resolved to 3292700193 bytes with sha256 a3fb0996aeda9edd3737d0a76d371bcf8309187462c89dae62692bcb5a9d81ce in 109.266s
Extracting /builds/worker/fetches/cuda-toolkit.tar.zst to /builds/worker/fetches
/builds/worker/fetches/cuda-toolkit.tar.zst extracted in 16.089s
Removing /builds/worker/fetches/cuda-toolkit.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 125.67143714300005, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2023-12-20T16:32:44.634Z] finished fetching artifacts
[task 2023-12-20T16:32:44.634Z] executing ['bash', '-c', 'export BMT_MARIAN=$MOZ_FETCHES_DIR && export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$MOZ_FETCHES_DIR/cuda-toolkit/lib64" && zstd --rm -d $MOZ_FETCHES_DIR/*.zst && $VCS_PATH/pipeline/quantize/export.sh $MOZ_FETCHES_DIR $MOZ_FETCHES_DIR/lex.s2t.pruned $MOZ_FETCHES_DIR/vocab.spm /builds/worker/artifacts']
[task 2023-12-20T16:32:44.637Z] 
[task 2023-12-20T16:32:44.637Z] s/lex.s2t.pruned.zst : 0 MB...     
[task 2023-12-20T16:32:44.637Z]                                                                                
[task 2023-12-20T16:32:44.637Z] /builds/worker/fetches/lex.s2t.pruned.zst: 23794 bytes 
[task 2023-12-20T16:32:44.639Z] + set -euo pipefail
[task 2023-12-20T16:32:44.639Z] + echo '###### Exporting a quantized model'
[task 2023-12-20T16:32:44.639Z] ###### Exporting a quantized model
[task 2023-12-20T16:32:44.639Z] + test -v SRC
[task 2023-12-20T16:32:44.639Z] + test -v TRG
[task 2023-12-20T16:32:44.639Z] + test -v BMT_MARIAN
[task 2023-12-20T16:32:44.639Z] + model_dir=/builds/worker/fetches
[task 2023-12-20T16:32:44.639Z] + shortlist=/builds/worker/fetches/lex.s2t.pruned
[task 2023-12-20T16:32:44.639Z] + vocab=/builds/worker/fetches/vocab.spm
[task 2023-12-20T16:32:44.639Z] + output_dir=/builds/worker/artifacts
[task 2023-12-20T16:32:44.639Z] + mkdir -p /builds/worker/artifacts
[task 2023-12-20T16:32:44.640Z] + model=/builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2023-12-20T16:32:44.641Z] + cp /builds/worker/fetches/model.intgemm.alphas.bin /builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2023-12-20T16:32:44.648Z] + pigz /builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2023-12-20T16:32:44.683Z] + shortlist_bin=/builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2023-12-20T16:32:44.683Z] + /builds/worker/fetches/marian-conv --shortlist /builds/worker/fetches/lex.s2t.pruned 50 50 0 --dump /builds/worker/artifacts/lex.50.50.ruen.s2t.bin --vocabs /builds/worker/fetches/vocab.spm /builds/worker/fetches/vocab.spm
[task 2023-12-20T16:32:44.719Z] [2023-12-20 16:32:44] [data] Loading SentencePiece vocabulary from file /builds/worker/fetches/vocab.spm
[task 2023-12-20T16:32:44.721Z] [2023-12-20 16:32:44] [data] Loading SentencePiece vocabulary from file /builds/worker/fetches/vocab.spm
[task 2023-12-20T16:32:44.723Z] [2023-12-20 16:32:44] [data] Importing text lexical shortlist as /builds/worker/fetches/lex.s2t.pruned 50 50 0
[task 2023-12-20T16:32:44.724Z] [2023-12-20 16:32:44] [data] Saving binary shortlist dump to /builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2023-12-20T16:32:44.724Z] [2023-12-20 16:32:44] Finished
[task 2023-12-20T16:32:44.731Z] + pigz /builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2023-12-20T16:32:44.733Z] + vocab_out=/builds/worker/artifacts/vocab.ruen.spm
[task 2023-12-20T16:32:44.733Z] + cp /builds/worker/fetches/vocab.spm /builds/worker/artifacts/vocab.ruen.spm
[task 2023-12-20T16:32:44.735Z] + pigz /builds/worker/artifacts/vocab.ruen.spm
[task 2023-12-20T16:32:44.748Z] + echo '### Export is completed. Results: /builds/worker/artifacts'
[task 2023-12-20T16:32:44.748Z] ### Export is completed. Results: /builds/worker/artifacts
[task 2023-12-20T16:32:44.748Z] + echo '###### Done: Exporting a quantized model'
[task 2023-12-20T16:32:44.748Z] ###### Done: Exporting a quantized model
[fetches 2023-12-20T16:32:44.748Z] removing /builds/worker/fetches
[fetches 2023-12-20T16:32:45.885Z] finished
[taskcluster 2023-12-20 16:32:46.051Z] === Task Finished ===
[taskcluster 2023-12-20 16:32:46.830Z] Successful task run with exit code: 0 completed in 158.106 seconds