[taskcluster:warn 2024-05-22T21:23:38.819Z] 
[taskcluster 2024-05-22T21:23:39.520Z] Uploading redirect artifact public/logs/live.log to URL https://firefoxci-websocktunnel.services.mozilla.com/us-west1-a.7006893856925146901.60099/log/G6X29pxyRg-cZpavzPA1pA with mime type "text/plain; charset=utf-8" and expiry 2024-05-29T21:53:39.500Z
[taskcluster 2024-05-22T21:23:39.972Z] [mounts] No existing writable directory cache 'translations-level-1-checkouts-v3-7afeb851dd97df8f3607-Q_deBg9LRdWXU21Qhn7s6Q' - creating caches/TgG6doeoRj2yWoBpQwyX7w
[taskcluster 2024-05-22T21:23:39.972Z] [mounts] Creating directory /task_171641295321715/cache0
[taskcluster 2024-05-22T21:23:39.981Z] [mounts] Granting task_171641295321715 full control of directory '/task_171641295321715/cache0'
[taskcluster 2024-05-22T21:23:39.982Z] [mounts] Successfully mounted writable directory cache '/task_171641295321715/cache0'
[taskcluster 2024-05-22T21:23:39.982Z] [mounts] Downloading task Q_deBg9LRdWXU21Qhn7s6Q artifact public/image.tar.zst to downloads/GuvAXPW4QgCsVl6-o3Z9VQ
[taskcluster 2024-05-22T21:23:46.338Z] [mounts] Downloaded 278996501 bytes with SHA256 20c0bfb9ede9bdea57036ae328d96caaf0223203dea299969fe41279a9b4600d from task Q_deBg9LRdWXU21Qhn7s6Q artifact public/image.tar.zst to downloads/GuvAXPW4QgCsVl6-o3Z9VQ
[taskcluster:warn 2024-05-22T21:23:46.338Z] [mounts] Download downloads/GuvAXPW4QgCsVl6-o3Z9VQ of task Q_deBg9LRdWXU21Qhn7s6Q artifact public/image.tar.zst has SHA256 20c0bfb9ede9bdea57036ae328d96caaf0223203dea299969fe41279a9b4600d but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2024-05-22T21:23:46.338Z] [mounts] Creating directory /task_171641295321715
[taskcluster 2024-05-22T21:23:46.348Z] [mounts] Decompressing zst file downloads/GuvAXPW4QgCsVl6-o3Z9VQ to '/task_171641295321715/dockerimage'
[taskcluster 2024-05-22T21:23:47.809Z] [mounts] Granting task_171641295321715 full control of file '/task_171641295321715/dockerimage'
[taskcluster 2024-05-22T21:23:47.811Z] Executing command 0: bash -cx 'timeout 604800 podman run -t --name taskcontainer --privileged -v "$(pwd)/cache0:/builds/worker/checkouts" --add-host=taskcluster:127.0.0.1 --net=host -e ARTIFACT_EXT -e COMPRESSION_CMD -e FIREFOX_TRANSLATIONS_TRAINING_BASE_REPOSITORY -e FIREFOX_TRANSLATIONS_TRAINING_HEAD_REF -e FIREFOX_TRANSLATIONS_TRAINING_HEAD_REPOSITORY -e FIREFOX_TRANSLATIONS_TRAINING_HEAD_REV -e FIREFOX_TRANSLATIONS_TRAINING_REPOSITORY_TYPE -e HG_STORE_PATH -e MOZ_AUTOMATION -e MOZ_FETCHES -e MOZ_FETCHES_DIR -e MOZ_SCM_LEVEL -e REPOSITORIES -e RUN_ID -e SCCACHE_DISABLE -e SRC -e TASKCLUSTER_CACHES -e TASKCLUSTER_PROXY_URL -e TASKCLUSTER_ROOT_URL -e TASKCLUSTER_UNTRUSTED_CACHES -e TASKCLUSTER_VOLUMES -e TASKCLUSTER_WORKER_LOCATION -e TASK_ID -e TRG -e VCS_PATH docker-archive:dockerimage /usr/local/bin/run-task --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c '\''pip3 install --upgrade pip setuptools && pip3 install -r $VCS_PATH/pipeline/alignments/requirements/alignments.txt && export BIN=$MOZ_FETCHES_DIR && export MARIAN=$MOZ_FETCHES_DIR && export PATH=$PATH:$MOZ_FETCHES_DIR && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && $VCS_PATH/pipeline/alignments/generate-shortlist.sh $MOZ_FETCHES_DIR/corpus $MOZ_FETCHES_DIR/vocab.spm $TASK_WORKDIR/artifacts auto'\''
[taskcluster 2024-05-22T21:23:47.811Z] exit_code=$?
[taskcluster 2024-05-22T21:23:47.811Z] podman cp taskcontainer:/builds/worker/artifacts artifact0
[taskcluster 2024-05-22T21:23:47.811Z] podman rm taskcontainer
[taskcluster 2024-05-22T21:23:47.811Z] exit "${exit_code}"'
++ pwd
+ timeout 604800 podman run -t --name taskcontainer --privileged -v /task_171641295321715/cache0:/builds/worker/checkouts --add-host=taskcluster:127.0.0.1 --net=host -e ARTIFACT_EXT -e COMPRESSION_CMD -e FIREFOX_TRANSLATIONS_TRAINING_BASE_REPOSITORY -e FIREFOX_TRANSLATIONS_TRAINING_HEAD_REF -e FIREFOX_TRANSLATIONS_TRAINING_HEAD_REPOSITORY -e FIREFOX_TRANSLATIONS_TRAINING_HEAD_REV -e FIREFOX_TRANSLATIONS_TRAINING_REPOSITORY_TYPE -e HG_STORE_PATH -e MOZ_AUTOMATION -e MOZ_FETCHES -e MOZ_FETCHES_DIR -e MOZ_SCM_LEVEL -e REPOSITORIES -e RUN_ID -e SCCACHE_DISABLE -e SRC -e TASKCLUSTER_CACHES -e TASKCLUSTER_PROXY_URL -e TASKCLUSTER_ROOT_URL -e TASKCLUSTER_UNTRUSTED_CACHES -e TASKCLUSTER_VOLUMES -e TASKCLUSTER_WORKER_LOCATION -e TASK_ID -e TRG -e VCS_PATH docker-archive:dockerimage /usr/local/bin/run-task --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c 'pip3 install --upgrade pip setuptools && pip3 install -r $VCS_PATH/pipeline/alignments/requirements/alignments.txt && export BIN=$MOZ_FETCHES_DIR && export MARIAN=$MOZ_FETCHES_DIR && export PATH=$PATH:$MOZ_FETCHES_DIR && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && $VCS_PATH/pipeline/alignments/generate-shortlist.sh $MOZ_FETCHES_DIR/corpus $MOZ_FETCHES_DIR/vocab.spm $TASK_WORKDIR/artifacts auto'
Getting image source signatures
Copying blob sha256:1a84dc453bda5bdb00677e6e599015475a779eb490b8b3158be45af99da083e3
Copying blob sha256:5498e8c22f6996f25ef193ee58617d5b37e2a96decf22e72de13c3b34e147591
Copying blob sha256:50b43097ddf2501adc2f8b21b8685a7508ddd563177028cae61a8ce8cd51a0bb
Copying config sha256:1e62b245ce9b86147f4b5a4cdcf3708c9a763f8cdaa58df95c74f81427d083ab
Writing manifest to image destination
Storing signatures
[setup 2024-05-22T21:23:54.904Z] run-task started in /builds/worker
[setup 2024-05-22T21:23:54.904Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c pip3 install --upgrade pip setuptools && pip3 install -r $VCS_PATH/pipeline/alignments/requirements/alignments.txt && export BIN=$MOZ_FETCHES_DIR && export MARIAN=$MOZ_FETCHES_DIR && export PATH=$PATH:$MOZ_FETCHES_DIR && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && $VCS_PATH/pipeline/alignments/generate-shortlist.sh $MOZ_FETCHES_DIR/corpus $MOZ_FETCHES_DIR/vocab.spm $TASK_WORKDIR/artifacts auto
[setup 2024-05-22T21:23:54.904Z] Python version: 3.10.12
[cache 2024-05-22T21:23:54.906Z] cache /builds/worker/checkouts is empty; writing requirements: gid=1000 uid=1000 version=1
[volume 2024-05-22T21:23:54.906Z] volume /builds/worker/checkouts is a cache
[setup 2024-05-22T21:23:54.906Z] running as worker:worker
[vcs 2024-05-22T21:23:54.906Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-05-22T21:23:54.908Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/builds/worker/checkouts/vcs']
[vcs 2024-05-22T21:23:54.910Z] Cloning into '/builds/worker/checkouts/vcs'...
[vcs 2024-05-22T21:23:55.744Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/vrigal/firefox-translations-training', 'publish-comet-metrics']
[vcs 2024-05-22T21:23:56.133Z] From https://github.com/vrigal/firefox-translations-training
[vcs 2024-05-22T21:23:56.134Z]  * branch            publish-comet-metrics -> FETCH_HEAD
[vcs 2024-05-22T21:23:56.137Z] executing ['git', 'checkout', '-f', '-B', 'publish-comet-metrics', '6d213f88caf646f94d7922167ab47ecbe3c0424d']
[vcs 2024-05-22T21:23:56.217Z] Switched to a new branch 'publish-comet-metrics'
[vcs 2024-05-22T21:23:56.218Z] executing ['git', 'submodule', 'init']
[vcs 2024-05-22T21:23:56.244Z] Submodule '3rd_party/browsermt-marian-dev' (https://github.com/browsermt/marian-dev) registered for path '3rd_party/browsermt-marian-dev'
[vcs 2024-05-22T21:23:56.244Z] Submodule 'extract-lex' (https://github.com/marian-nmt/extract-lex) registered for path '3rd_party/extract-lex'
[vcs 2024-05-22T21:23:56.245Z] Submodule 'fast_align' (https://github.com/clab/fast_align) registered for path '3rd_party/fast_align'
[vcs 2024-05-22T21:23:56.246Z] Submodule '3rd_party/kenlm' (https://github.com/kpu/kenlm) registered for path '3rd_party/kenlm'
[vcs 2024-05-22T21:23:56.246Z] Submodule '3rd_party/marian-dev' (https://github.com/marian-nmt/marian-dev) registered for path '3rd_party/marian-dev'
[vcs 2024-05-22T21:23:56.247Z] Submodule '3rd_party/preprocess' (https://github.com/kpu/preprocess.git) registered for path '3rd_party/preprocess'
[vcs 2024-05-22T21:23:56.248Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-05-22T21:23:56.274Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/browsermt-marian-dev'...
[vcs 2024-05-22T21:23:57.460Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/extract-lex'...
[vcs 2024-05-22T21:23:57.822Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/fast_align'...
[vcs 2024-05-22T21:23:58.254Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/kenlm'...
[vcs 2024-05-22T21:23:59.047Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/marian-dev'...
[vcs 2024-05-22T21:24:01.070Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/preprocess'...
[vcs 2024-05-22T21:24:02.395Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-05-22T21:24:02.414Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-05-22T21:24:02.430Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-05-22T21:24:02.464Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-05-22T21:24:02.589Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-05-22T21:24:02.616Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-05-22T21:24:02.617Z] cleaning git checkout...
[vcs 2024-05-22T21:24:02.617Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-05-22T21:24:02.621Z] removing []
[vcs 2024-05-22T21:24:02.621Z] successfully cleaned git checkout!
[vcs 2024-05-22T21:24:02.623Z] TinderboxPrint:<a href='https://github.com/vrigal/firefox-translations-training/commit/6d213f88caf646f94d7922167ab47ecbe3c0424d' title='Built from firefox-translations-training commit 6d213f88caf646f94d7922167ab47ecbe3c0424d'>6d213f88caf646f94d7922167ab47ecbe3c0424d</a>
[setup 2024-05-22T21:24:02.623Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-05-22T21:24:02.623Z] fetching artifacts
[fetches 2024-05-22T21:24:02.623Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VOTTZMd7TIKcMF2RR6MY1g/artifacts/public/build/corpus.en.zst to /builds/worker/fetches/corpus.en.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VOTTZMd7TIKcMF2RR6MY1g/artifacts/public/build/corpus.en.zstattempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VOTTZMd7TIKcMF2RR6MY1g/artifacts/public/build/corpus.ru.zst to /builds/worker/fetches/corpus.ru.zst

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VF--UyYRTI6hby6Gr143Vw/artifacts/public/build/extract_lex.tar.zst to /builds/worker/fetches/extract_lex.tar.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QYRWW9BSTh-34mlpXqVVsg/artifacts/public/build/fast-align.tar.zst to /builds/worker/fetches/fast-align.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VOTTZMd7TIKcMF2RR6MY1g/artifacts/public/build/corpus.ru.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VF--UyYRTI6hby6Gr143Vw/artifacts/public/build/extract_lex.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QYRWW9BSTh-34mlpXqVVsg/artifacts/public/build/fast-align.tar.zstattempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cCr_z2sITDGiow53WiHs8g/artifacts/public/build/marian.tar.zst to /builds/worker/fetches/marian.tar.zst

attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cCr_z2sITDGiow53WiHs8g/artifacts/public/build/marian.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/H379d38NSA6WPtWeCH9j0Q/artifacts/public/build/vocab.spm to /builds/worker/fetches/vocab.spm
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/H379d38NSA6WPtWeCH9j0Q/artifacts/public/build/vocab.spm
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VF--UyYRTI6hby6Gr143Vw/artifacts/public/build/extract_lex.tar.zst resolved to 620577 bytes with sha256 964eea4e00f3761a1fc0cc3d50b562a2bbcfdd9b7f668c38d4e2f3ed12c0d9c1 in 0.055s
Extracting /builds/worker/fetches/extract_lex.tar.zst to /builds/worker/fetches
/builds/worker/fetches/extract_lex.tar.zst extracted in 0.008s
Removing /builds/worker/fetches/extract_lex.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/H379d38NSA6WPtWeCH9j0Q/artifacts/public/build/vocab.spm resolved to 252230 bytes with sha256 9d8ac29958816890d18f1ab76bc153d1b73b59a3e447780e1021e936802d92ac in 0.121s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QYRWW9BSTh-34mlpXqVVsg/artifacts/public/build/fast-align.tar.zst resolved to 722791 bytes with sha256 0cd2223cd464783e79948ea5057aa6e482f3f0bbe6660d3cd9200fab91f0e409 in 0.134s
Extracting /builds/worker/fetches/fast-align.tar.zst to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VOTTZMd7TIKcMF2RR6MY1g/artifacts/public/build/corpus.en.zst resolved to 101495 bytes with sha256 5cad3eff3175a77db92245276a110255c7070c9a331c9aaae0c061a44aaee10a in 0.143s
/builds/worker/fetches/fast-align.tar.zst extracted in 0.009s
Removing /builds/worker/fetches/fast-align.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VOTTZMd7TIKcMF2RR6MY1g/artifacts/public/build/corpus.ru.zst resolved to 979526 bytes with sha256 0acd766279fcead83f6a8a583960fd080d968c6aecf28aaf0d486c83ff58b5a9 in 0.196s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cCr_z2sITDGiow53WiHs8g/artifacts/public/build/marian.tar.zst resolved to 684170918 bytes with sha256 674b3ad4462d1b7ef22e1f2a5a1e340385136821b0e5f4b48af357281a41d819 in 87.905s
Extracting /builds/worker/fetches/marian.tar.zst to /builds/worker/fetches
/builds/worker/fetches/marian.tar.zst extracted in 5.657s
Removing /builds/worker/fetches/marian.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 93.64644394999999, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-05-22T21:25:36.359Z] finished fetching artifacts
[task 2024-05-22T21:25:36.360Z] executing ['bash', '-c', 'pip3 install --upgrade pip setuptools && pip3 install -r $VCS_PATH/pipeline/alignments/requirements/alignments.txt && export BIN=$MOZ_FETCHES_DIR && export MARIAN=$MOZ_FETCHES_DIR && export PATH=$PATH:$MOZ_FETCHES_DIR && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && $VCS_PATH/pipeline/alignments/generate-shortlist.sh $MOZ_FETCHES_DIR/corpus $MOZ_FETCHES_DIR/vocab.spm $TASK_WORKDIR/artifacts auto']
[task 2024-05-22T21:25:36.699Z] WARNING: The directory '/builds/worker/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
[task 2024-05-22T21:25:36.700Z] Defaulting to user installation because normal site-packages is not writeable
[task 2024-05-22T21:25:36.723Z] Requirement already satisfied: pip in /usr/lib/python3/dist-packages (22.0.2)
[task 2024-05-22T21:25:36.867Z] Collecting pip
[task 2024-05-22T21:25:36.934Z]   Downloading pip-24.0-py3-none-any.whl (2.1 MB)
[task 2024-05-22T21:25:37.087Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 14.2 MB/s eta 0:00:00
[task 2024-05-22T21:25:37.098Z] Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
[task 2024-05-22T21:25:37.385Z] Collecting setuptools
[task 2024-05-22T21:25:37.393Z]   Downloading setuptools-70.0.0-py3-none-any.whl (863 kB)
[task 2024-05-22T21:25:37.427Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 863.4/863.4 KB 27.8 MB/s eta 0:00:00
[task 2024-05-22T21:25:37.524Z] Installing collected packages: setuptools, pip
[task 2024-05-22T21:25:38.897Z] Successfully installed pip-24.0 setuptools-70.0.0
[task 2024-05-22T21:25:39.233Z] WARNING: The directory '/builds/worker/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
[task 2024-05-22T21:25:39.234Z] Defaulting to user installation because normal site-packages is not writeable
[task 2024-05-22T21:25:39.633Z] Collecting cython==3.0.9 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 7))
[task 2024-05-22T21:25:39.702Z]   Downloading Cython-3.0.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
[task 2024-05-22T21:25:39.721Z] Collecting eflomal==1.0.0b1 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 9))
[task 2024-05-22T21:25:39.731Z]   Downloading eflomal-1.0.0b1.tar.gz (102 kB)
[task 2024-05-22T21:25:39.753Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.0/102.0 kB 5.9 MB/s eta 0:00:00
[task 2024-05-22T21:25:39.767Z]   Installing build dependencies: started
[task 2024-05-22T21:25:44.854Z]   Installing build dependencies: finished with status 'done'
[task 2024-05-22T21:25:44.855Z]   Getting requirements to build wheel: started
[task 2024-05-22T21:25:46.104Z]   Getting requirements to build wheel: finished with status 'done'
[task 2024-05-22T21:25:46.106Z]   Preparing metadata (pyproject.toml): started
[task 2024-05-22T21:25:46.659Z]   Preparing metadata (pyproject.toml): finished with status 'done'
[task 2024-05-22T21:25:46.862Z] Collecting numpy==1.26.4 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 11))
[task 2024-05-22T21:25:46.870Z]   Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
[task 2024-05-22T21:25:46.876Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 11.7 MB/s eta 0:00:00
[task 2024-05-22T21:25:46.898Z] Downloading Cython-3.0.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
[task 2024-05-22T21:25:47.072Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 21.0 MB/s eta 0:00:00
[task 2024-05-22T21:25:47.080Z] Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
[task 2024-05-22T21:25:47.289Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 126.2 MB/s eta 0:00:00
[task 2024-05-22T21:25:47.358Z] Building wheels for collected packages: eflomal
[task 2024-05-22T21:25:47.360Z]   Building wheel for eflomal (pyproject.toml): started
[task 2024-05-22T21:25:52.189Z]   Building wheel for eflomal (pyproject.toml): finished with status 'done'
[task 2024-05-22T21:25:52.191Z]   Created wheel for eflomal: filename=eflomal-1.0.0b1-cp310-cp310-linux_x86_64.whl size=320909 sha256=49bb521595cee4f20562d247e3cab5e8c5bf343fed49786e7501425b7e7ed3fa
[task 2024-05-22T21:25:52.191Z]   Stored in directory: /tmp/pip-ephem-wheel-cache-guw0_bmn/wheels/b4/0c/55/4a2c6f00793a0d2fdaef4376b8cb3d9c6ae8df16366d6e31d1
[task 2024-05-22T21:25:52.193Z] Successfully built eflomal
[task 2024-05-22T21:25:52.285Z] Installing collected packages: numpy, cython, eflomal
[task 2024-05-22T21:25:54.604Z] Successfully installed cython-3.0.9 eflomal-1.0.0b1 numpy-1.26.4
[task 2024-05-22T21:25:54.908Z] + set -euo pipefail
[task 2024-05-22T21:25:54.908Z] + echo '###### Generating alignments and shortlist'
[task 2024-05-22T21:25:54.908Z] ###### Generating alignments and shortlist
[task 2024-05-22T21:25:54.908Z] + [[ -z /builds/worker/fetches ]]
[task 2024-05-22T21:25:54.908Z] + [[ -z /builds/worker/fetches ]]
[task 2024-05-22T21:25:54.908Z] + [[ -z ru ]]
[task 2024-05-22T21:25:54.908Z] + [[ -z en ]]
[task 2024-05-22T21:25:54.908Z] + corpus_prefix=/builds/worker/fetches/corpus
[task 2024-05-22T21:25:54.909Z] + vocab_path=/builds/worker/fetches/vocab.spm
[task 2024-05-22T21:25:54.909Z] + output_dir=/builds/worker/artifacts
[task 2024-05-22T21:25:54.909Z] + threads=auto
[task 2024-05-22T21:25:54.909Z] + COMPRESSION_CMD=zstdmt
[task 2024-05-22T21:25:54.909Z] + ARTIFACT_EXT=zst
[task 2024-05-22T21:25:54.909Z] + '[' auto = auto ']'
[task 2024-05-22T21:25:54.909Z] ++ nproc
[task 2024-05-22T21:25:54.910Z] + threads=32
[task 2024-05-22T21:25:54.910Z] ++ dirname /builds/worker/checkouts/vcs/pipeline/alignments/generate-shortlist.sh
[task 2024-05-22T21:25:54.911Z] + cd /builds/worker/checkouts/vcs/pipeline/alignments
[task 2024-05-22T21:25:54.911Z] + mkdir -p /builds/worker/artifacts
[task 2024-05-22T21:25:54.912Z] + dir=/builds/worker/artifacts/tmp
[task 2024-05-22T21:25:54.912Z] + mkdir -p /builds/worker/artifacts/tmp
[task 2024-05-22T21:25:54.913Z] + corpus_src=/builds/worker/fetches/corpus.ru.zst
[task 2024-05-22T21:25:54.913Z] + corpus_trg=/builds/worker/fetches/corpus.en.zst
[task 2024-05-22T21:25:54.913Z] + echo '### Subword segmentation with SentencePiece'
[task 2024-05-22T21:25:54.913Z] ### Subword segmentation with SentencePiece
[task 2024-05-22T21:25:54.914Z] + zstdmt -dc /builds/worker/fetches/corpus.ru.zst
[task 2024-05-22T21:25:54.914Z] + parallel --no-notice --pipe -k -j 32 --block 50M /builds/worker/fetches/spm_encode --model /builds/worker/fetches/vocab.spm
[task 2024-05-22T21:25:55.832Z] + zstdmt -dc /builds/worker/fetches/corpus.en.zst
[task 2024-05-22T21:25:55.832Z] + parallel --no-notice --pipe -k -j 32 --block 50M /builds/worker/fetches/spm_encode --model /builds/worker/fetches/vocab.spm
[task 2024-05-22T21:25:56.340Z] + python3 align.py --corpus_src=/builds/worker/artifacts/tmp/corpus.spm.ru --corpus_trg=/builds/worker/artifacts/tmp/corpus.spm.en --output_path=/builds/worker/artifacts/corpus.aln
[task 2024-05-22T21:25:56.505Z] [alignments] Running with arguments: ['align.py', '--corpus_src=/builds/worker/artifacts/tmp/corpus.spm.ru', '--corpus_trg=/builds/worker/artifacts/tmp/corpus.spm.en', '--output_path=/builds/worker/artifacts/corpus.aln']
[task 2024-05-22T21:25:56.507Z] [alignments] Starting generating alignments.
[task 2024-05-22T21:25:56.507Z] [alignments] Calculating alignments...
[task 2024-05-22T21:25:56.919Z] [eflomal] Prepared 16120 sentences for alignment
[task 2024-05-22T21:25:56.919Z] /builds/worker/.local/lib/python3.10/site-packages/eflomal/bin/eflomal -m 3 -s /tmp/tmpntf946sy -t /tmp/tmp1spz1lkp -n 3 -N 0.2 -1 9 -2 9 -3 39 -f /builds/worker/artifacts/tmp/aln.fwd -r /builds/worker/artifacts/tmp/aln.rev
[task 2024-05-22T21:25:57.044Z] Read texts (16120 sentences): 0.124 s
[task 2024-05-22T21:25:57.044Z] Vocabulary sizes are 501 (source), 98 (target)
[task 2024-05-22T21:25:57.046Z] Created alignment structures: 0.001 s
[task 2024-05-22T21:25:57.047Z] Created alignment structures: 0.001 s
[task 2024-05-22T21:25:57.056Z] Randomized alignment: 0.011 s
[task 2024-05-22T21:25:57.057Z] Aligning with model 1 (9 iterations)
[task 2024-05-22T21:25:57.063Z] Randomized alignment: 0.016 s
[task 2024-05-22T21:25:57.063Z] Aligning with model 1 (9 iterations)
[task 2024-05-22T21:25:59.694Z] Done: 2.638 s
[task 2024-05-22T21:25:59.694Z] Aligning with model 2 (9 iterations)
[task 2024-05-22T21:25:59.824Z] Done: 2.761 s
[task 2024-05-22T21:25:59.824Z] Aligning with model 2 (9 iterations)
[task 2024-05-22T21:26:02.642Z] Done: 2.948 s
[task 2024-05-22T21:26:02.642Z] Aligning with model 3 (39 iterations)
[task 2024-05-22T21:26:02.850Z] Done: 3.026 s
[task 2024-05-22T21:26:02.850Z] Aligning with model 3 (39 iterations)
[task 2024-05-22T21:26:18.102Z] Done: 15.252 s
[task 2024-05-22T21:26:18.155Z] Done: 15.513 s
[task 2024-05-22T21:26:19.703Z] Final argmax iteration: 1.602 s
[task 2024-05-22T21:26:19.703Z] Writing alignments to /builds/worker/artifacts/tmp/aln.rev for 16120 sentencess
[task 2024-05-22T21:26:19.744Z] Final argmax iteration: 1.589 s
[task 2024-05-22T21:26:19.744Z] Writing alignments to /builds/worker/artifacts/tmp/aln.fwd for 16120 sentencess
[task 2024-05-22T21:26:19.782Z] [alignments] Symmetrizing alignments...
[task 2024-05-22T21:26:20.854Z] [alignments] Finished generating alignments.
[task 2024-05-22T21:26:20.875Z] + echo '### Creating shortlist'
[task 2024-05-22T21:26:20.875Z] ### Creating shortlist
[task 2024-05-22T21:26:20.875Z] + /builds/worker/fetches/extract_lex /builds/worker/artifacts/tmp/corpus.spm.en /builds/worker/artifacts/tmp/corpus.spm.ru /builds/worker/artifacts/corpus.aln /builds/worker/artifacts/tmp/lex.s2t /builds/worker/artifacts/tmp/lex.t2s
[task 2024-05-22T21:26:20.877Z] Starting...
[task 2024-05-22T21:26:21.266Z] 0 10000 
[task 2024-05-22T21:26:21.266Z] Finished
[task 2024-05-22T21:26:21.267Z] + '[' -f /builds/worker/artifacts/tmp/lex.s2t ']'
[task 2024-05-22T21:26:21.267Z] + zstdmt /builds/worker/artifacts/tmp/lex.s2t
[task 2024-05-22T21:26:21.270Z] 
[task 2024-05-22T21:26:21.270Z]                                                                                
[task 2024-05-22T21:26:21.270Z] Read :  0 /  0 MB ==> 31%
[task 2024-05-22T21:26:21.270Z]                                                                                
[task 2024-05-22T21:26:21.270Z] /builds/worker/artifacts/tmp/lex.s2t : 31.20%   (101148 =>  31560 bytes, /builds/worker/artifacts/tmp/lex.s2t.zst) 
[task 2024-05-22T21:26:21.270Z] + rm /builds/worker/artifacts/tmp/corpus.spm.en
[task 2024-05-22T21:26:21.271Z] + rm /builds/worker/artifacts/tmp/corpus.spm.ru
[task 2024-05-22T21:26:21.273Z] + rm /builds/worker/artifacts/corpus.aln
[task 2024-05-22T21:26:21.274Z] + echo '### Shortlist pruning'
[task 2024-05-22T21:26:21.274Z] ### Shortlist pruning
[task 2024-05-22T21:26:21.274Z] + /builds/worker/fetches/spm_export_vocab --model=/builds/worker/fetches/vocab.spm --output=/builds/worker/artifacts/tmp/vocab.txt
[task 2024-05-22T21:26:21.279Z] + zstdmt -dc /builds/worker/artifacts/tmp/lex.s2t.zst
[task 2024-05-22T21:26:21.280Z] + grep -v NULL
[task 2024-05-22T21:26:21.280Z] + python3 prune_shortlist.py 100 /builds/worker/artifacts/tmp/vocab.txt
[task 2024-05-22T21:26:21.280Z] + zstdmt
[task 2024-05-22T21:26:21.310Z] + echo '### Deleting tmp dir'
[task 2024-05-22T21:26:21.310Z] ### Deleting tmp dir
[task 2024-05-22T21:26:21.310Z] + rm -rf /builds/worker/artifacts/tmp
[task 2024-05-22T21:26:21.312Z] + echo '###### Done: Generating alignments and shortlist'
[task 2024-05-22T21:26:21.312Z] ###### Done: Generating alignments and shortlist
[fetches 2024-05-22T21:26:21.312Z] removing /builds/worker/fetches
[fetches 2024-05-22T21:26:21.649Z] finished
+ exit_code=0
+ podman cp taskcontainer:/builds/worker/artifacts artifact0
+ podman rm taskcontainer
d20f7c0a9c75c97669497600ce3c9e6eaacb27db8a3a3b1edf3058e5f251e3b0
+ exit 0
[taskcluster 2024-05-22T21:26:23.299Z]    Exit Code: 0
[taskcluster 2024-05-22T21:26:23.299Z]    User Time: 9.589798s
[taskcluster 2024-05-22T21:26:23.299Z]  Kernel Time: 4.957041s
[taskcluster 2024-05-22T21:26:23.299Z]    Wall Time: 2m35.487595407s
[taskcluster 2024-05-22T21:26:23.299Z]       Result: SUCCEEDED
[taskcluster 2024-05-22T21:26:23.299Z] === Task Finished ===
[taskcluster 2024-05-22T21:26:23.299Z] Task Duration: 2m35.487996579s
[taskcluster 2024-05-22T21:26:23.341Z] Uploading artifact public/build/lex.s2t.pruned.zst from file /task_171641295321715/artifact0/lex.s2t.pruned.zst with content encoding "identity", mime type "application/zstd" and expiry 2024-08-20T20:54:42.753Z
[taskcluster 2024-05-22T21:26:23.535Z] [mounts] Preserving cache: Moving "/task_171641295321715/cache0" to "caches/TgG6doeoRj2yWoBpQwyX7w"
[taskcluster 2024-05-22T21:26:23.566Z] [mounts] Denying task_171641295321715 access to 'caches/TgG6doeoRj2yWoBpQwyX7w'
[taskcluster 2024-05-22T21:26:23.626Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-08-20T20:54:42.753Z