[taskcluster 2024-05-24 17:46:19.112Z] Task ID: aoFZ8Q9pT2W7BNWA2I7dyw
[taskcluster 2024-05-24 17:46:19.112Z] Worker ID: 3201410613418749124
[taskcluster 2024-05-24 17:46:19.112Z] Worker Group: us-central1-b
[taskcluster 2024-05-24 17:46:19.112Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-05-24 17:46:19.112Z] Worker Pool: translations-1/b-linux-large-gcp-1tb
[taskcluster 2024-05-24 17:46:19.112Z] Worker Version: 38.0.5
[taskcluster 2024-05-24 17:46:19.112Z] Public IP: 34.171.88.111
[taskcluster 2024-05-24 17:46:19.112Z] Hostname: translations-1-b-linux-large-gcp-1tb-xpxnqxyfsc-hkzcd18bg3w
[taskcluster 2024-05-24 17:46:19.112Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-Q_deBg9LRdWXU21Qhn7s6Q" -> /builds/worker/checkouts

[taskcluster 2024-05-24 17:46:21.667Z] Downloading artifact "public/image.tar.zst" from task ID: Q_deBg9LRdWXU21Qhn7s6Q.
[taskcluster 2024-05-24 17:46:25.129Z] Downloaded artifact successfully.
[taskcluster 2024-05-24 17:46:25.130Z] Downloaded 266.072 mb
[taskcluster 2024-05-24 17:46:25.131Z] Decompressing downloaded image
[taskcluster 2024-05-24 17:46:26.600Z] Loading docker image from downloaded archive.
[taskcluster 2024-05-24 17:46:36.057Z] Image 'public/image.tar.zst' from task 'Q_deBg9LRdWXU21Qhn7s6Q' loaded.  Using image ID sha256:1e62b245ce9b86147f4b5a4cdcf3708c9a763f8cdaa58df95c74f81427d083ab.
[taskcluster 2024-05-24 17:46:36.202Z] === Task Starting ===
[setup 2024-05-24T17:46:43.049Z] run-task started in /builds/worker
[setup 2024-05-24T17:46:43.049Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c pip3 install --upgrade pip setuptools && pip3 install -r $VCS_PATH/pipeline/alignments/requirements/alignments.txt && export BIN=$MOZ_FETCHES_DIR && export MARIAN=$MOZ_FETCHES_DIR && export PATH=$PATH:$MOZ_FETCHES_DIR && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && $VCS_PATH/pipeline/alignments/generate-shortlist.sh $MOZ_FETCHES_DIR/corpus $MOZ_FETCHES_DIR/vocab.spm $TASK_WORKDIR/artifacts auto
[setup 2024-05-24T17:46:43.049Z] Python version: 3.10.12
[cache 2024-05-24T17:46:43.051Z] cache /builds/worker/checkouts is empty; writing requirements: gid=1000 uid=1000 version=1
[volume 2024-05-24T17:46:43.051Z] volume /builds/worker/checkouts is a cache
[setup 2024-05-24T17:46:43.051Z] running as worker:worker
[vcs 2024-05-24T17:46:43.051Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-05-24T17:46:43.053Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/builds/worker/checkouts/vcs']
[vcs 2024-05-24T17:46:43.054Z] Cloning into '/builds/worker/checkouts/vcs'...
[vcs 2024-05-24T17:46:43.849Z] executing ['git', 'fetch', '--tags', '--force', 'https://github.com/mozilla/firefox-translations-training', 'revert_d2g_pool']
[vcs 2024-05-24T17:46:44.048Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-05-24T17:46:44.048Z]  * branch            revert_d2g_pool -> FETCH_HEAD
[vcs 2024-05-24T17:46:44.053Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/mozilla/firefox-translations-training', 'revert_d2g_pool']
[vcs 2024-05-24T17:46:44.255Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-05-24T17:46:44.255Z]  * branch            revert_d2g_pool -> FETCH_HEAD
[vcs 2024-05-24T17:46:44.261Z] executing ['git', 'checkout', '-f', '-B', 'revert_d2g_pool', '9db52842e17a9fa0ececcfc30809128fa1c52288']
[vcs 2024-05-24T17:46:44.326Z] Switched to a new branch 'revert_d2g_pool'
[vcs 2024-05-24T17:46:44.326Z] executing ['git', 'submodule', 'init']
[vcs 2024-05-24T17:46:44.344Z] Submodule '3rd_party/browsermt-marian-dev' (https://github.com/browsermt/marian-dev) registered for path '3rd_party/browsermt-marian-dev'
[vcs 2024-05-24T17:46:44.345Z] Submodule 'extract-lex' (https://github.com/marian-nmt/extract-lex) registered for path '3rd_party/extract-lex'
[vcs 2024-05-24T17:46:44.345Z] Submodule 'fast_align' (https://github.com/clab/fast_align) registered for path '3rd_party/fast_align'
[vcs 2024-05-24T17:46:44.346Z] Submodule '3rd_party/kenlm' (https://github.com/kpu/kenlm) registered for path '3rd_party/kenlm'
[vcs 2024-05-24T17:46:44.347Z] Submodule '3rd_party/marian-dev' (https://github.com/marian-nmt/marian-dev) registered for path '3rd_party/marian-dev'
[vcs 2024-05-24T17:46:44.347Z] Submodule '3rd_party/preprocess' (https://github.com/kpu/preprocess.git) registered for path '3rd_party/preprocess'
[vcs 2024-05-24T17:46:44.348Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-05-24T17:46:44.367Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/browsermt-marian-dev'...
[vcs 2024-05-24T17:46:45.424Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/extract-lex'...
[vcs 2024-05-24T17:46:45.700Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/fast_align'...
[vcs 2024-05-24T17:46:46.001Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/kenlm'...
[vcs 2024-05-24T17:46:46.611Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/marian-dev'...
[vcs 2024-05-24T17:46:47.845Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/preprocess'...
[vcs 2024-05-24T17:46:48.321Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-05-24T17:46:48.329Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-05-24T17:46:48.338Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-05-24T17:46:48.360Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-05-24T17:46:48.457Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-05-24T17:46:48.473Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-05-24T17:46:48.473Z] cleaning git checkout...
[vcs 2024-05-24T17:46:48.473Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-05-24T17:46:48.476Z] removing []
[vcs 2024-05-24T17:46:48.476Z] successfully cleaned git checkout!
[vcs 2024-05-24T17:46:48.477Z] TinderboxPrint:<a href='https://github.com/mozilla/firefox-translations-training/commit/9db52842e17a9fa0ececcfc30809128fa1c52288' title='Built from firefox-translations-training commit 9db52842e17a9fa0ececcfc30809128fa1c52288'>9db52842e17a9fa0ececcfc30809128fa1c52288</a>
[setup 2024-05-24T17:46:48.477Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-05-24T17:46:48.477Z] fetching artifacts
[fetches 2024-05-24T17:46:48.477Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VRkqgjqmTdK1QDj_z3jmIw/artifacts/public/build/corpus.en.zst to /builds/worker/fetches/corpus.en.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VRkqgjqmTdK1QDj_z3jmIw/artifacts/public/build/corpus.en.zstattempt 1/5
attempt 1/5

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VF--UyYRTI6hby6Gr143Vw/artifacts/public/build/extract_lex.tar.zst to /builds/worker/fetches/extract_lex.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VRkqgjqmTdK1QDj_z3jmIw/artifacts/public/build/corpus.ru.zst to /builds/worker/fetches/corpus.ru.zst

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QYRWW9BSTh-34mlpXqVVsg/artifacts/public/build/fast-align.tar.zst to /builds/worker/fetches/fast-align.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VF--UyYRTI6hby6Gr143Vw/artifacts/public/build/extract_lex.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QYRWW9BSTh-34mlpXqVVsg/artifacts/public/build/fast-align.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VRkqgjqmTdK1QDj_z3jmIw/artifacts/public/build/corpus.ru.zst
attempt 1/5

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cCr_z2sITDGiow53WiHs8g/artifacts/public/build/marian.tar.zst to /builds/worker/fetches/marian.tar.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6FAEP8XTEKLTldrBixoVg/artifacts/public/build/vocab.spm to /builds/worker/fetches/vocab.spm
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cCr_z2sITDGiow53WiHs8g/artifacts/public/build/marian.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6FAEP8XTEKLTldrBixoVg/artifacts/public/build/vocab.spm
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QYRWW9BSTh-34mlpXqVVsg/artifacts/public/build/fast-align.tar.zst resolved to 722791 bytes with sha256 0cd2223cd464783e79948ea5057aa6e482f3f0bbe6660d3cd9200fab91f0e409 in 0.127s
Extracting /builds/worker/fetches/fast-align.tar.zst to /builds/worker/fetches
/builds/worker/fetches/fast-align.tar.zst extracted in 0.007s
Removing /builds/worker/fetches/fast-align.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a6FAEP8XTEKLTldrBixoVg/artifacts/public/build/vocab.spm resolved to 252230 bytes with sha256 9d8ac29958816890d18f1ab76bc153d1b73b59a3e447780e1021e936802d92ac in 0.138s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VRkqgjqmTdK1QDj_z3jmIw/artifacts/public/build/corpus.en.zst resolved to 205017 bytes with sha256 d3130c2eb0d4b280f7d2b2c040aa873c467a89ad942ef431cb918415cf0833cd in 0.146s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VRkqgjqmTdK1QDj_z3jmIw/artifacts/public/build/corpus.ru.zst resolved to 953199 bytes with sha256 420486d5271aa7194a7d0bf8322494c0a68dd8c7822c954fef57ca6e23d61302 in 0.151s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VF--UyYRTI6hby6Gr143Vw/artifacts/public/build/extract_lex.tar.zst resolved to 620577 bytes with sha256 964eea4e00f3761a1fc0cc3d50b562a2bbcfdd9b7f668c38d4e2f3ed12c0d9c1 in 0.218s
Extracting /builds/worker/fetches/extract_lex.tar.zst to /builds/worker/fetches
/builds/worker/fetches/extract_lex.tar.zst extracted in 0.006s
Removing /builds/worker/fetches/extract_lex.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cCr_z2sITDGiow53WiHs8g/artifacts/public/build/marian.tar.zst resolved to 684170918 bytes with sha256 674b3ad4462d1b7ef22e1f2a5a1e340385136821b0e5f4b48af357281a41d819 in 23.044s
Extracting /builds/worker/fetches/marian.tar.zst to /builds/worker/fetches
/builds/worker/fetches/marian.tar.zst extracted in 4.582s
Removing /builds/worker/fetches/marian.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 27.98122572999999, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-05-24T17:47:16.537Z] finished fetching artifacts
[task 2024-05-24T17:47:16.538Z] executing ['bash', '-c', 'pip3 install --upgrade pip setuptools && pip3 install -r $VCS_PATH/pipeline/alignments/requirements/alignments.txt && export BIN=$MOZ_FETCHES_DIR && export MARIAN=$MOZ_FETCHES_DIR && export PATH=$PATH:$MOZ_FETCHES_DIR && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && $VCS_PATH/pipeline/alignments/generate-shortlist.sh $MOZ_FETCHES_DIR/corpus $MOZ_FETCHES_DIR/vocab.spm $TASK_WORKDIR/artifacts auto']
[task 2024-05-24T17:47:16.852Z] WARNING: The directory '/builds/worker/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
[task 2024-05-24T17:47:16.852Z] Defaulting to user installation because normal site-packages is not writeable
[task 2024-05-24T17:47:16.873Z] Requirement already satisfied: pip in /usr/lib/python3/dist-packages (22.0.2)
[task 2024-05-24T17:47:17.036Z] Collecting pip
[task 2024-05-24T17:47:17.203Z]   Downloading pip-24.0-py3-none-any.whl (2.1 MB)
[task 2024-05-24T17:47:17.422Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 9.7 MB/s eta 0:00:00
[task 2024-05-24T17:47:17.433Z] Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
[task 2024-05-24T17:47:17.735Z] Collecting setuptools
[task 2024-05-24T17:47:17.776Z]   Downloading setuptools-70.0.0-py3-none-any.whl (863 kB)
[task 2024-05-24T17:47:17.785Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 863.4/863.4 KB 139.4 MB/s eta 0:00:00
[task 2024-05-24T17:47:17.887Z] Installing collected packages: setuptools, pip
[task 2024-05-24T17:47:19.126Z] Successfully installed pip-24.0 setuptools-70.0.0
[task 2024-05-24T17:47:19.435Z] WARNING: The directory '/builds/worker/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
[task 2024-05-24T17:47:19.436Z] Defaulting to user installation because normal site-packages is not writeable
[task 2024-05-24T17:47:19.866Z] Collecting cython==3.0.9 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 7))
[task 2024-05-24T17:47:20.042Z]   Downloading Cython-3.0.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
[task 2024-05-24T17:47:20.069Z] Collecting eflomal==1.0.0b1 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 9))
[task 2024-05-24T17:47:20.108Z]   Downloading eflomal-1.0.0b1.tar.gz (102 kB)
[task 2024-05-24T17:47:20.173Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.0/102.0 kB 1.7 MB/s eta 0:00:00
[task 2024-05-24T17:47:20.188Z]   Installing build dependencies: started
[task 2024-05-24T17:47:25.793Z]   Installing build dependencies: finished with status 'done'
[task 2024-05-24T17:47:25.794Z]   Getting requirements to build wheel: started
[task 2024-05-24T17:47:26.932Z]   Getting requirements to build wheel: finished with status 'done'
[task 2024-05-24T17:47:26.934Z]   Preparing metadata (pyproject.toml): started
[task 2024-05-24T17:47:27.433Z]   Preparing metadata (pyproject.toml): finished with status 'done'
[task 2024-05-24T17:47:27.664Z] Collecting numpy==1.26.4 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 11))
[task 2024-05-24T17:47:27.704Z]   Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
[task 2024-05-24T17:47:27.712Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 9.1 MB/s eta 0:00:00
[task 2024-05-24T17:47:27.763Z] Downloading Cython-3.0.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
[task 2024-05-24T17:47:27.933Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 21.7 MB/s eta 0:00:00
[task 2024-05-24T17:47:27.973Z] Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
[task 2024-05-24T17:47:28.179Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 87.2 MB/s eta 0:00:00
[task 2024-05-24T17:47:28.253Z] Building wheels for collected packages: eflomal
[task 2024-05-24T17:47:28.254Z]   Building wheel for eflomal (pyproject.toml): started
[task 2024-05-24T17:47:32.495Z]   Building wheel for eflomal (pyproject.toml): finished with status 'done'
[task 2024-05-24T17:47:32.497Z]   Created wheel for eflomal: filename=eflomal-1.0.0b1-cp310-cp310-linux_x86_64.whl size=320910 sha256=0073b347494cadc3e678bd0432008a9bf6f8cbb244b263050dff243eb7436668
[task 2024-05-24T17:47:32.497Z]   Stored in directory: /tmp/pip-ephem-wheel-cache-bq6c6tbt/wheels/b4/0c/55/4a2c6f00793a0d2fdaef4376b8cb3d9c6ae8df16366d6e31d1
[task 2024-05-24T17:47:32.499Z] Successfully built eflomal
[task 2024-05-24T17:47:32.591Z] Installing collected packages: numpy, cython, eflomal
[task 2024-05-24T17:47:34.746Z] Successfully installed cython-3.0.9 eflomal-1.0.0b1 numpy-1.26.4
[task 2024-05-24T17:47:35.038Z] + set -euo pipefail
[task 2024-05-24T17:47:35.038Z] + echo '###### Generating alignments and shortlist'
[task 2024-05-24T17:47:35.038Z] ###### Generating alignments and shortlist
[task 2024-05-24T17:47:35.038Z] + [[ -z /builds/worker/fetches ]]
[task 2024-05-24T17:47:35.038Z] + [[ -z /builds/worker/fetches ]]
[task 2024-05-24T17:47:35.038Z] + [[ -z ru ]]
[task 2024-05-24T17:47:35.038Z] + [[ -z en ]]
[task 2024-05-24T17:47:35.038Z] + corpus_prefix=/builds/worker/fetches/corpus
[task 2024-05-24T17:47:35.038Z] + vocab_path=/builds/worker/fetches/vocab.spm
[task 2024-05-24T17:47:35.038Z] + output_dir=/builds/worker/artifacts
[task 2024-05-24T17:47:35.038Z] + threads=auto
[task 2024-05-24T17:47:35.038Z] + COMPRESSION_CMD=zstdmt
[task 2024-05-24T17:47:35.038Z] + ARTIFACT_EXT=zst
[task 2024-05-24T17:47:35.038Z] + '[' auto = auto ']'
[task 2024-05-24T17:47:35.039Z] ++ nproc
[task 2024-05-24T17:47:35.039Z] + threads=32
[task 2024-05-24T17:47:35.040Z] ++ dirname /builds/worker/checkouts/vcs/pipeline/alignments/generate-shortlist.sh
[task 2024-05-24T17:47:35.040Z] + cd /builds/worker/checkouts/vcs/pipeline/alignments
[task 2024-05-24T17:47:35.040Z] + mkdir -p /builds/worker/artifacts
[task 2024-05-24T17:47:35.041Z] + dir=/builds/worker/artifacts/tmp
[task 2024-05-24T17:47:35.041Z] + mkdir -p /builds/worker/artifacts/tmp
[task 2024-05-24T17:47:35.042Z] + corpus_src=/builds/worker/fetches/corpus.ru.zst
[task 2024-05-24T17:47:35.042Z] + corpus_trg=/builds/worker/fetches/corpus.en.zst
[task 2024-05-24T17:47:35.042Z] + echo '### Subword segmentation with SentencePiece'
[task 2024-05-24T17:47:35.042Z] ### Subword segmentation with SentencePiece
[task 2024-05-24T17:47:35.042Z] + zstdmt -dc /builds/worker/fetches/corpus.ru.zst
[task 2024-05-24T17:47:35.042Z] + parallel --no-notice --pipe -k -j 32 --block 50M /builds/worker/fetches/spm_encode --model /builds/worker/fetches/vocab.spm
[task 2024-05-24T17:47:35.816Z] + zstdmt -dc /builds/worker/fetches/corpus.en.zst
[task 2024-05-24T17:47:35.816Z] + parallel --no-notice --pipe -k -j 32 --block 50M /builds/worker/fetches/spm_encode --model /builds/worker/fetches/vocab.spm
[task 2024-05-24T17:47:37.867Z] + python3 align.py --corpus_src=/builds/worker/artifacts/tmp/corpus.spm.ru --corpus_trg=/builds/worker/artifacts/tmp/corpus.spm.en --output_path=/builds/worker/artifacts/corpus.aln
[task 2024-05-24T17:47:37.983Z] [alignments] Running with arguments: ['align.py', '--corpus_src=/builds/worker/artifacts/tmp/corpus.spm.ru', '--corpus_trg=/builds/worker/artifacts/tmp/corpus.spm.en', '--output_path=/builds/worker/artifacts/corpus.aln']
[task 2024-05-24T17:47:37.985Z] [alignments] Starting generating alignments.
[task 2024-05-24T17:47:37.985Z] [alignments] Calculating alignments...
[task 2024-05-24T17:47:38.822Z] [eflomal] Prepared 16120 sentences for alignment
[task 2024-05-24T17:47:38.822Z] /builds/worker/.local/lib/python3.10/site-packages/eflomal/bin/eflomal -m 3 -s /tmp/tmp9619940y -t /tmp/tmpubguhjyu -n 3 -N 0.2 -1 9 -2 9 -3 39 -f /builds/worker/artifacts/tmp/aln.fwd -r /builds/worker/artifacts/tmp/aln.rev
[task 2024-05-24T17:47:39.025Z] Read texts (16120 sentences): 0.203 s
[task 2024-05-24T17:47:39.025Z] Vocabulary sizes are 502 (source), 108 (target)
[task 2024-05-24T17:47:39.034Z] Created alignment structures: 0.001 s
[task 2024-05-24T17:47:39.035Z] Created alignment structures: 0.001 s
[task 2024-05-24T17:47:39.049Z] Randomized alignment: 0.015 s
[task 2024-05-24T17:47:39.049Z] Aligning with model 1 (9 iterations)
[task 2024-05-24T17:47:39.068Z] Randomized alignment: 0.032 s
[task 2024-05-24T17:47:39.068Z] Aligning with model 1 (9 iterations)
[task 2024-05-24T17:47:50.033Z] Done: 10.983 s
[task 2024-05-24T17:47:50.033Z] Aligning with model 2 (9 iterations)
[task 2024-05-24T17:47:50.950Z] Done: 11.882 s
[task 2024-05-24T17:47:50.950Z] Aligning with model 2 (9 iterations)
[task 2024-05-24T17:48:01.465Z] Done: 11.433 s
[task 2024-05-24T17:48:01.465Z] Aligning with model 3 (39 iterations)
[task 2024-05-24T17:48:03.264Z] Done: 12.314 s
[task 2024-05-24T17:48:03.264Z] Aligning with model 3 (39 iterations)
[task 2024-05-24T17:48:57.459Z] Done: 55.993 s
[task 2024-05-24T17:49:02.492Z] Final argmax iteration: 5.033 s
[task 2024-05-24T17:49:02.492Z] Writing alignments to /builds/worker/artifacts/tmp/aln.rev for 16120 sentencess
[task 2024-05-24T17:49:03.866Z] Done: 60.602 s
[task 2024-05-24T17:49:09.203Z] Final argmax iteration: 5.337 s
[task 2024-05-24T17:49:09.203Z] Writing alignments to /builds/worker/artifacts/tmp/aln.fwd for 16120 sentencess
[task 2024-05-24T17:49:09.316Z] [alignments] Symmetrizing alignments...
[task 2024-05-24T17:49:13.131Z] [alignments] Finished generating alignments.
[task 2024-05-24T17:49:13.150Z] + echo '### Creating shortlist'
[task 2024-05-24T17:49:13.150Z] ### Creating shortlist
[task 2024-05-24T17:49:13.150Z] + /builds/worker/fetches/extract_lex /builds/worker/artifacts/tmp/corpus.spm.en /builds/worker/artifacts/tmp/corpus.spm.ru /builds/worker/artifacts/corpus.aln /builds/worker/artifacts/tmp/lex.s2t /builds/worker/artifacts/tmp/lex.t2s
[task 2024-05-24T17:49:13.151Z] Starting...
[task 2024-05-24T17:49:13.856Z] 0 10000 
[task 2024-05-24T17:49:13.856Z] Finished
[task 2024-05-24T17:49:13.857Z] + '[' -f /builds/worker/artifacts/tmp/lex.s2t ']'
[task 2024-05-24T17:49:13.857Z] + zstdmt /builds/worker/artifacts/tmp/lex.s2t
[task 2024-05-24T17:49:13.860Z] 
[task 2024-05-24T17:49:13.860Z]                                                                                
[task 2024-05-24T17:49:13.860Z] Read :  0 /  0 MB ==> 32%
[task 2024-05-24T17:49:13.860Z]                                                                                
[task 2024-05-24T17:49:13.860Z] /builds/worker/artifacts/tmp/lex.s2t : 32.06%   (126732 =>  40629 bytes, /builds/worker/artifacts/tmp/lex.s2t.zst) 
[task 2024-05-24T17:49:13.860Z] + rm /builds/worker/artifacts/tmp/corpus.spm.en
[task 2024-05-24T17:49:13.864Z] + rm /builds/worker/artifacts/tmp/corpus.spm.ru
[task 2024-05-24T17:49:13.865Z] + rm /builds/worker/artifacts/corpus.aln
[task 2024-05-24T17:49:13.866Z] + echo '### Shortlist pruning'
[task 2024-05-24T17:49:13.866Z] ### Shortlist pruning
[task 2024-05-24T17:49:13.866Z] + /builds/worker/fetches/spm_export_vocab --model=/builds/worker/fetches/vocab.spm --output=/builds/worker/artifacts/tmp/vocab.txt
[task 2024-05-24T17:49:13.870Z] + zstdmt -dc /builds/worker/artifacts/tmp/lex.s2t.zst
[task 2024-05-24T17:49:13.870Z] + grep -v NULL
[task 2024-05-24T17:49:13.870Z] + python3 prune_shortlist.py 100 /builds/worker/artifacts/tmp/vocab.txt
[task 2024-05-24T17:49:13.870Z] + zstdmt
[task 2024-05-24T17:49:13.903Z] + echo '### Deleting tmp dir'
[task 2024-05-24T17:49:13.903Z] ### Deleting tmp dir
[task 2024-05-24T17:49:13.903Z] + rm -rf /builds/worker/artifacts/tmp
[task 2024-05-24T17:49:13.906Z] + echo '###### Done: Generating alignments and shortlist'
[task 2024-05-24T17:49:13.906Z] ###### Done: Generating alignments and shortlist
[fetches 2024-05-24T17:49:13.906Z] removing /builds/worker/fetches
[fetches 2024-05-24T17:49:14.118Z] finished
[taskcluster 2024-05-24 17:49:14.391Z] === Task Finished ===
[taskcluster 2024-05-24 17:49:14.612Z] Successful task run with exit code: 0 completed in 175.501 seconds