[taskcluster 2024-05-24 17:02:38.488Z] Task ID: CNkzq1f4TQW7VSM1Md0CAg
[taskcluster 2024-05-24 17:02:38.489Z] Worker ID: 459303397715966052
[taskcluster 2024-05-24 17:02:38.489Z] Worker Group: us-central1-f
[taskcluster 2024-05-24 17:02:38.489Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-05-24 17:02:38.489Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-05-24 17:02:38.489Z] Worker Version: 38.0.5
[taskcluster 2024-05-24 17:02:38.489Z] Public IP: 104.197.135.232
[taskcluster 2024-05-24 17:02:38.489Z] Hostname: translations-1-b-linux-large-gcp-300gb-br0qyfjtr7qh1sdjrxxvfq
[taskcluster 2024-05-24 17:02:38.489Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-Q_deBg9LRdWXU21Qhn7s6Q" -> /builds/worker/checkouts

[taskcluster 2024-05-24 17:02:39.089Z] Image 'public/image.tar.zst' from task 'Q_deBg9LRdWXU21Qhn7s6Q' loaded.  Using image ID sha256:1e62b245ce9b86147f4b5a4cdcf3708c9a763f8cdaa58df95c74f81427d083ab.
[taskcluster 2024-05-24 17:02:39.137Z] === Task Starting ===
[setup 2024-05-24T17:02:39.468Z] run-task started in /builds/worker
[setup 2024-05-24T17:02:39.468Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/clean/merge-corpus.sh artifacts/corpus $MOZ_FETCHES_DIR/*.zst
[setup 2024-05-24T17:02:39.468Z] Python version: 3.10.12
[cache 2024-05-24T17:02:39.470Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-05-24T17:02:39.470Z] volume /builds/worker/checkouts is a cache
[setup 2024-05-24T17:02:39.470Z] running as worker:worker
[vcs 2024-05-24T17:02:39.470Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-05-24T17:02:39.472Z] executing ['git', 'fetch', '--tags', '--force', 'https://github.com/mozilla/firefox-translations-training', 'revert_d2g_pool']
[vcs 2024-05-24T17:02:39.681Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-05-24T17:02:39.681Z]  * branch            revert_d2g_pool -> FETCH_HEAD
[vcs 2024-05-24T17:02:39.687Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/mozilla/firefox-translations-training', 'revert_d2g_pool']
[vcs 2024-05-24T17:02:39.884Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-05-24T17:02:39.884Z]  * branch            revert_d2g_pool -> FETCH_HEAD
[vcs 2024-05-24T17:02:39.890Z] executing ['git', 'checkout', '-f', '-B', 'revert_d2g_pool', '9db52842e17a9fa0ececcfc30809128fa1c52288']
[vcs 2024-05-24T17:02:39.895Z] Switched to a new branch 'revert_d2g_pool'
[vcs 2024-05-24T17:02:39.896Z] executing ['git', 'submodule', 'init']
[vcs 2024-05-24T17:02:39.916Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-05-24T17:02:39.948Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-05-24T17:02:39.958Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-05-24T17:02:39.968Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-05-24T17:02:39.978Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-05-24T17:02:39.993Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-05-24T17:02:40.003Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-05-24T17:02:40.003Z] cleaning git checkout...
[vcs 2024-05-24T17:02:40.003Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-05-24T17:02:40.006Z] removing []
[vcs 2024-05-24T17:02:40.006Z] successfully cleaned git checkout!
[vcs 2024-05-24T17:02:40.008Z] TinderboxPrint:<a href='https://github.com/mozilla/firefox-translations-training/commit/9db52842e17a9fa0ececcfc30809128fa1c52288' title='Built from firefox-translations-training commit 9db52842e17a9fa0ececcfc30809128fa1c52288'>9db52842e17a9fa0ececcfc30809128fa1c52288</a>
[setup 2024-05-24T17:02:40.008Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-05-24T17:02:40.008Z] fetching artifacts
[fetches 2024-05-24T17:02:40.008Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cEFbQlaUQxmnC9hffjUAaA/artifacts/public/build/ELRC-3075-wikipedia_health_v1.en.zst to /builds/worker/fetches/ELRC-3075-wikipedia_health_v1.en.zst
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cEFbQlaUQxmnC9hffjUAaA/artifacts/public/build/ELRC-3075-wikipedia_health_v1.en.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cEFbQlaUQxmnC9hffjUAaA/artifacts/public/build/ELRC-3075-wikipedia_health_v1.ru.zst to /builds/worker/fetches/ELRC-3075-wikipedia_health_v1.ru.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cEFbQlaUQxmnC9hffjUAaA/artifacts/public/build/ELRC-3075-wikipedia_health_v1.ru.zstattempt 1/5

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dllI0BLIT4WW6OOIFPP1ow/artifacts/public/build/ada83_v1.en.zst to /builds/worker/fetches/ada83_v1.en.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dllI0BLIT4WW6OOIFPP1ow/artifacts/public/build/ada83_v1.ru.zst to /builds/worker/fetches/ada83_v1.ru.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dt7X26lwS_Gyv8ni2I08PQ/artifacts/public/build/gcp_pytest-dataset_a0017e.en.zst to /builds/worker/fetches/gcp_pytest-dataset_a0017e.en.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dllI0BLIT4WW6OOIFPP1ow/artifacts/public/build/ada83_v1.ru.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dt7X26lwS_Gyv8ni2I08PQ/artifacts/public/build/gcp_pytest-dataset_a0017e.en.zstattempt 1/5
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dt7X26lwS_Gyv8ni2I08PQ/artifacts/public/build/gcp_pytest-dataset_a0017e.ru.zst to /builds/worker/fetches/gcp_pytest-dataset_a0017e.ru.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dllI0BLIT4WW6OOIFPP1ow/artifacts/public/build/ada83_v1.en.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst to /builds/worker/fetches/dedupe.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dt7X26lwS_Gyv8ni2I08PQ/artifacts/public/build/gcp_pytest-dataset_a0017e.ru.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dt7X26lwS_Gyv8ni2I08PQ/artifacts/public/build/gcp_pytest-dataset_a0017e.ru.zst resolved to 755 bytes with sha256 37b8ecdecfaaefbbcc0555a44d8f1201c09d9e0c58e5a1ee8fec159ddee93375 in 0.111s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dt7X26lwS_Gyv8ni2I08PQ/artifacts/public/build/gcp_pytest-dataset_a0017e.en.zst resolved to 539 bytes with sha256 1584216350b567c80f2e92b665881d19b4466596d4c848af460c1c16bfd193ab in 0.117s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dllI0BLIT4WW6OOIFPP1ow/artifacts/public/build/ada83_v1.ru.zst resolved to 164865 bytes with sha256 741811858877b11c10de47ff79ac2278cfa4272793769563a66c3e1786b94e22 in 0.151s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cEFbQlaUQxmnC9hffjUAaA/artifacts/public/build/ELRC-3075-wikipedia_health_v1.ru.zst resolved to 218815 bytes with sha256 a81b233f3e4eb53b599b1f4afab8cd1debcaaf09fedad5af44c9d27ea2dbe7f0 in 0.158s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dllI0BLIT4WW6OOIFPP1ow/artifacts/public/build/ada83_v1.en.zst resolved to 110134 bytes with sha256 b5aa13577484746f222b9a43f4117147420505eb606986d4f3b6aaa64132a920 in 0.162s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cEFbQlaUQxmnC9hffjUAaA/artifacts/public/build/ELRC-3075-wikipedia_health_v1.en.zst resolved to 154187 bytes with sha256 9a5c3cde479c654c86e8635d79003eb1e6e36fa71d88ddcfbbc4d7f3541f6082 in 0.177s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst resolved to 133246 bytes with sha256 1e567b7eabfbbac81c949ba165181bf3b4b5a15d4fcd114a0fc1d7ec82cfac07 in 0.237s
Extracting /builds/worker/fetches/dedupe.tar.zst to /builds/worker/fetches
/builds/worker/fetches/dedupe.tar.zst extracted in 0.002s
Removing /builds/worker/fetches/dedupe.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.2461546210000165, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-05-24T17:02:40.331Z] finished fetching artifacts
[task 2024-05-24T17:02:40.331Z] executing ['bash', '-c', 'export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/clean/merge-corpus.sh artifacts/corpus $MOZ_FETCHES_DIR/*.zst']
[task 2024-05-24T17:02:40.333Z] + set -euo pipefail
[task 2024-05-24T17:02:40.333Z] + echo '###### Merging parallel datasets'
[task 2024-05-24T17:02:40.333Z] ###### Merging parallel datasets
[task 2024-05-24T17:02:40.333Z] + test -v SRC
[task 2024-05-24T17:02:40.333Z] + test -v TRG
[task 2024-05-24T17:02:40.333Z] + test -v BIN
[task 2024-05-24T17:02:40.333Z] + output_prefix=artifacts/corpus
[task 2024-05-24T17:02:40.333Z] + inputs=("${@:2}")
[task 2024-05-24T17:02:40.333Z] + COMPRESSION_CMD=zstdmt
[task 2024-05-24T17:02:40.333Z] + ARTIFACT_EXT=zst
[task 2024-05-24T17:02:40.333Z] + tmp=artifacts/corpus/merge
[task 2024-05-24T17:02:40.333Z] + mkdir -p artifacts/corpus/merge
[task 2024-05-24T17:02:40.335Z] + echo '### Merging'
[task 2024-05-24T17:02:40.335Z] ### Merging
[task 2024-05-24T17:02:40.335Z] + [[ /builds/worker/fetches/ada83_v1.en.zst == *.zst ]]
[task 2024-05-24T17:02:40.335Z] ++ tr ' ' '\n'
[task 2024-05-24T17:02:40.335Z] ++ echo /builds/worker/fetches/ada83_v1.en.zst /builds/worker/fetches/ada83_v1.ru.zst /builds/worker/fetches/ELRC-3075-wikipedia_health_v1.en.zst /builds/worker/fetches/ELRC-3075-wikipedia_health_v1.ru.zst /builds/worker/fetches/gcp_pytest-dataset_a0017e.en.zst /builds/worker/fetches/gcp_pytest-dataset_a0017e.ru.zst
[task 2024-05-24T17:02:40.335Z] ++ grep ru.zst
[task 2024-05-24T17:02:40.335Z] ++ tr '\n' ' '
[task 2024-05-24T17:02:40.337Z] + cat /builds/worker/fetches/ada83_v1.ru.zst /builds/worker/fetches/ELRC-3075-wikipedia_health_v1.ru.zst /builds/worker/fetches/gcp_pytest-dataset_a0017e.ru.zst
[task 2024-05-24T17:02:40.338Z] ++ echo /builds/worker/fetches/ada83_v1.en.zst /builds/worker/fetches/ada83_v1.ru.zst /builds/worker/fetches/ELRC-3075-wikipedia_health_v1.en.zst /builds/worker/fetches/ELRC-3075-wikipedia_health_v1.ru.zst /builds/worker/fetches/gcp_pytest-dataset_a0017e.en.zst /builds/worker/fetches/gcp_pytest-dataset_a0017e.ru.zst
[task 2024-05-24T17:02:40.338Z] ++ tr ' ' '\n'
[task 2024-05-24T17:02:40.338Z] ++ grep en.zst
[task 2024-05-24T17:02:40.338Z] ++ tr '\n' ' '
[task 2024-05-24T17:02:40.339Z] + cat /builds/worker/fetches/ada83_v1.en.zst /builds/worker/fetches/ELRC-3075-wikipedia_health_v1.en.zst /builds/worker/fetches/gcp_pytest-dataset_a0017e.en.zst
[task 2024-05-24T17:02:40.340Z] + echo '### Deduplication'
[task 2024-05-24T17:02:40.340Z] ### Deduplication
[task 2024-05-24T17:02:40.341Z] + /builds/worker/fetches/dedupe
[task 2024-05-24T17:02:40.341Z] + zstdmt
[task 2024-05-24T17:02:40.341Z] ++ zstdmt -dc artifacts/corpus/merge/corpus.ru.dup.zst
[task 2024-05-24T17:02:40.341Z] + paste /dev/fd/63 /dev/fd/62
[task 2024-05-24T17:02:40.341Z] ++ zstdmt -dc artifacts/corpus/merge/corpus.en.dup.zst
[task 2024-05-24T17:02:40.342Z] File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
[task 2024-05-24T17:02:40.361Z] Kept 7028 / 7181 = 0.978694
[task 2024-05-24T17:02:40.378Z] + zstdmt -dc artifacts/corpus/merge.ruen.zst
[task 2024-05-24T17:02:40.378Z] + cut -f1
[task 2024-05-24T17:02:40.378Z] + zstdmt
[task 2024-05-24T17:02:40.404Z] + zstdmt -dc artifacts/corpus/merge.ruen.zst
[task 2024-05-24T17:02:40.404Z] + cut -f2
[task 2024-05-24T17:02:40.404Z] + zstdmt
[task 2024-05-24T17:02:40.421Z] + rm -rf artifacts/corpus/merge
[task 2024-05-24T17:02:40.422Z] + echo '###### Done: Merging parallel datasets'
[task 2024-05-24T17:02:40.422Z] ###### Done: Merging parallel datasets
[fetches 2024-05-24T17:02:40.422Z] removing /builds/worker/fetches
[fetches 2024-05-24T17:02:40.423Z] finished
[taskcluster 2024-05-24 17:02:40.637Z] === Task Finished ===
[taskcluster 2024-05-24 17:02:41.312Z] Successful task run with exit code: 0 completed in 2.824 seconds