[taskcluster 2024-05-24 17:43:57.950Z] Task ID: a2LeLawzTledMJLMV7PMjA
[taskcluster 2024-05-24 17:43:57.950Z] Worker ID: 3089034284472561602
[taskcluster 2024-05-24 17:43:57.950Z] Worker Group: us-west1-b
[taskcluster 2024-05-24 17:43:57.950Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-05-24 17:43:57.950Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-05-24 17:43:57.950Z] Worker Version: 38.0.5
[taskcluster 2024-05-24 17:43:57.950Z] Public IP: 104.196.249.28
[taskcluster 2024-05-24 17:43:57.950Z] Hostname: translations-1-b-linux-large-gcp-300gb-ahsjjfcesluu2dtipc5g9g
[taskcluster 2024-05-24 17:43:57.950Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-Q_deBg9LRdWXU21Qhn7s6Q" -> /builds/worker/checkouts

[taskcluster 2024-05-24 17:43:58.358Z] Image 'public/image.tar.zst' from task 'Q_deBg9LRdWXU21Qhn7s6Q' loaded.  Using image ID sha256:1e62b245ce9b86147f4b5a4cdcf3708c9a763f8cdaa58df95c74f81427d083ab.
[taskcluster 2024-05-24 17:43:58.372Z] === Task Starting ===
[setup 2024-05-24T17:43:58.652Z] run-task started in /builds/worker
[setup 2024-05-24T17:43:58.652Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/translate/merge-corpus.sh $MOZ_FETCHES_DIR/corpus.ru.zst $MOZ_FETCHES_DIR/mono.ru.zst $MOZ_FETCHES_DIR/corpus.en.zst $MOZ_FETCHES_DIR/mono.en.zst /builds/worker/artifacts/corpus.ru.zst /builds/worker/artifacts/corpus.en.zst
[setup 2024-05-24T17:43:58.652Z] Python version: 3.10.12
[cache 2024-05-24T17:43:58.654Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-05-24T17:43:58.654Z] volume /builds/worker/checkouts is a cache
[setup 2024-05-24T17:43:58.654Z] running as worker:worker
[vcs 2024-05-24T17:43:58.654Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-05-24T17:43:58.656Z] executing ['git', 'fetch', '--tags', '--force', 'https://github.com/mozilla/firefox-translations-training', 'revert_d2g_pool']
[vcs 2024-05-24T17:43:58.920Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-05-24T17:43:58.920Z]  * branch            revert_d2g_pool -> FETCH_HEAD
[vcs 2024-05-24T17:43:58.925Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/mozilla/firefox-translations-training', 'revert_d2g_pool']
[vcs 2024-05-24T17:43:59.185Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-05-24T17:43:59.185Z]  * branch            revert_d2g_pool -> FETCH_HEAD
[vcs 2024-05-24T17:43:59.190Z] executing ['git', 'checkout', '-f', '-B', 'revert_d2g_pool', '9db52842e17a9fa0ececcfc30809128fa1c52288']
[vcs 2024-05-24T17:43:59.194Z] Reset branch 'revert_d2g_pool'
[vcs 2024-05-24T17:43:59.195Z] executing ['git', 'submodule', 'init']
[vcs 2024-05-24T17:43:59.212Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-05-24T17:43:59.244Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-05-24T17:43:59.253Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-05-24T17:43:59.263Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-05-24T17:43:59.273Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-05-24T17:43:59.288Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-05-24T17:43:59.298Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-05-24T17:43:59.298Z] cleaning git checkout...
[vcs 2024-05-24T17:43:59.298Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-05-24T17:43:59.301Z] removing []
[vcs 2024-05-24T17:43:59.301Z] successfully cleaned git checkout!
[vcs 2024-05-24T17:43:59.302Z] TinderboxPrint:<a href='https://github.com/mozilla/firefox-translations-training/commit/9db52842e17a9fa0ececcfc30809128fa1c52288' title='Built from firefox-translations-training commit 9db52842e17a9fa0ececcfc30809128fa1c52288'>9db52842e17a9fa0ececcfc30809128fa1c52288</a>
[setup 2024-05-24T17:43:59.302Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-05-24T17:43:59.302Z] fetching artifacts
[fetches 2024-05-24T17:43:59.302Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/S2YTFNyBSTGMFD4176hUEw/artifacts/public/build/corpus.en.zst to /builds/worker/fetches/corpus.en.zst
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/S2YTFNyBSTGMFD4176hUEw/artifacts/public/build/corpus.en.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZOYgr18YS4G5GbmUCRIFAQ/artifacts/public/build/mono.en.zst to /builds/worker/fetches/mono.en.zst
attempt 1/5attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Jry7MPYQTze2fbplT4Nbow/artifacts/public/build/mono.ru.zst to /builds/worker/fetches/mono.ru.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CNkzq1f4TQW7VSM1Md0CAg/artifacts/public/build/corpus.ru.zst to /builds/worker/fetches/corpus.ru.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZOYgr18YS4G5GbmUCRIFAQ/artifacts/public/build/mono.en.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst to /builds/worker/fetches/dedupe.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Jry7MPYQTze2fbplT4Nbow/artifacts/public/build/mono.ru.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CNkzq1f4TQW7VSM1Md0CAg/artifacts/public/build/corpus.ru.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/S2YTFNyBSTGMFD4176hUEw/artifacts/public/build/corpus.en.zst resolved to 114886 bytes with sha256 ea255ba3de3163e8af7dd7d5aaf0fd3cb6d33d61188aaac350064637c5cb6878 in 0.126s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst resolved to 133246 bytes with sha256 1e567b7eabfbbac81c949ba165181bf3b4b5a15d4fcd114a0fc1d7ec82cfac07 in 0.217s
Extracting /builds/worker/fetches/dedupe.tar.zst to /builds/worker/fetches
/builds/worker/fetches/dedupe.tar.zst extracted in 0.002s
Removing /builds/worker/fetches/dedupe.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZOYgr18YS4G5GbmUCRIFAQ/artifacts/public/build/mono.en.zst resolved to 107804 bytes with sha256 18486d841cb75a4fb3729ad2975beb0a8ce5e42fffa61fd1c85b7cc04eddc787 in 0.249s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Jry7MPYQTze2fbplT4Nbow/artifacts/public/build/mono.ru.zst resolved to 583453 bytes with sha256 0f61cf2bd692316e65d372716d73c918dbb764c540bcc4a300098124ab773282 in 0.324s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CNkzq1f4TQW7VSM1Md0CAg/artifacts/public/build/corpus.ru.zst resolved to 383206 bytes with sha256 80402ba043c57ee1f77ab1e1635c4533cdf18075177ed1c7b3543be79b24f452 in 1.352s
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 1.3571096260000104, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-05-24T17:44:00.736Z] finished fetching artifacts
[task 2024-05-24T17:44:00.736Z] executing ['bash', '-c', 'export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/translate/merge-corpus.sh $MOZ_FETCHES_DIR/corpus.ru.zst $MOZ_FETCHES_DIR/mono.ru.zst $MOZ_FETCHES_DIR/corpus.en.zst $MOZ_FETCHES_DIR/mono.en.zst /builds/worker/artifacts/corpus.ru.zst /builds/worker/artifacts/corpus.en.zst']
[task 2024-05-24T17:44:00.738Z] + set -euo pipefail
[task 2024-05-24T17:44:00.738Z] + test -v BIN
[task 2024-05-24T17:44:00.738Z] + echo '###### Merging datasets'
[task 2024-05-24T17:44:00.738Z] ###### Merging datasets
[task 2024-05-24T17:44:00.738Z] + src1=/builds/worker/fetches/corpus.ru.zst
[task 2024-05-24T17:44:00.738Z] + src2=/builds/worker/fetches/mono.ru.zst
[task 2024-05-24T17:44:00.738Z] + trg1=/builds/worker/fetches/corpus.en.zst
[task 2024-05-24T17:44:00.738Z] + trg2=/builds/worker/fetches/mono.en.zst
[task 2024-05-24T17:44:00.738Z] + res_src=/builds/worker/artifacts/corpus.ru.zst
[task 2024-05-24T17:44:00.738Z] + res_trg=/builds/worker/artifacts/corpus.en.zst
[task 2024-05-24T17:44:00.738Z] + COMPRESSION_CMD=zstdmt
[task 2024-05-24T17:44:00.738Z] + ARTIFACT_EXT=zst
[task 2024-05-24T17:44:00.739Z] ++ dirname /builds/worker/artifacts/corpus.ru.zst
[task 2024-05-24T17:44:00.739Z] + tmp_dir=/builds/worker/artifacts/tmp
[task 2024-05-24T17:44:00.739Z] + mkdir -p /builds/worker/artifacts/tmp
[task 2024-05-24T17:44:00.741Z] + zstdmt
[task 2024-05-24T17:44:00.741Z] ++ zstdmt -dc /builds/worker/fetches/corpus.ru.zst
[task 2024-05-24T17:44:00.741Z] + cat /dev/fd/63 /dev/fd/62
[task 2024-05-24T17:44:00.741Z] ++ zstdmt -dc /builds/worker/fetches/mono.ru.zst
[task 2024-05-24T17:44:00.772Z] + zstdmt
[task 2024-05-24T17:44:00.773Z] ++ zstdmt -dc /builds/worker/fetches/corpus.en.zst
[task 2024-05-24T17:44:00.773Z] + cat /dev/fd/63 /dev/fd/62
[task 2024-05-24T17:44:00.773Z] ++ zstdmt -dc /builds/worker/fetches/mono.en.zst
[task 2024-05-24T17:44:00.796Z] + echo '#### Deduplicating'
[task 2024-05-24T17:44:00.796Z] #### Deduplicating
[task 2024-05-24T17:44:00.796Z] + /builds/worker/fetches/dedupe
[task 2024-05-24T17:44:00.796Z] + shuf --random-source=/dev/fd/63
[task 2024-05-24T17:44:00.796Z] ++ zstdmt -dc /builds/worker/artifacts/tmp/original.src.zst
[task 2024-05-24T17:44:00.796Z] + paste /dev/fd/63 /dev/fd/62
[task 2024-05-24T17:44:00.796Z] + zstdmt
[task 2024-05-24T17:44:00.796Z] ++ get_seeded_random 42
[task 2024-05-24T17:44:00.796Z] ++ seed=42
[task 2024-05-24T17:44:00.796Z] ++ openssl enc -aes-256-ctr -pass pass:42 -nosalt
[task 2024-05-24T17:44:00.796Z] ++ zstdmt -dc /builds/worker/artifacts/tmp/original.trg.zst
[task 2024-05-24T17:44:00.798Z] File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
[task 2024-05-24T17:44:01.030Z] Kept 16967 / 16977 = 0.999411
[task 2024-05-24T17:44:01.043Z] + zstdmt -dc /builds/worker/artifacts/tmp/all.zst
[task 2024-05-24T17:44:01.043Z] + cut -f1
[task 2024-05-24T17:44:01.043Z] + zstdmt
[task 2024-05-24T17:44:01.162Z] + zstdmt -dc /builds/worker/artifacts/tmp/all.zst
[task 2024-05-24T17:44:01.162Z] + cut -f2
[task 2024-05-24T17:44:01.163Z] + zstdmt
[task 2024-05-24T17:44:01.345Z] ++ zstdmt -dc /builds/worker/artifacts/corpus.ru.zst
[task 2024-05-24T17:44:01.345Z] ++ wc -l
[task 2024-05-24T17:44:01.354Z] + src_len=16967
[task 2024-05-24T17:44:01.354Z] ++ zstdmt -dc /builds/worker/artifacts/corpus.en.zst
[task 2024-05-24T17:44:01.355Z] ++ wc -l
[task 2024-05-24T17:44:01.370Z] + trg_len=16967
[task 2024-05-24T17:44:01.370Z] + '[' 16967 '!=' 16967 ']'
[task 2024-05-24T17:44:01.370Z] + rm -rf /builds/worker/artifacts/tmp
[task 2024-05-24T17:44:01.371Z] + echo '###### Done: Merging datasets'
[task 2024-05-24T17:44:01.371Z] ###### Done: Merging datasets
[fetches 2024-05-24T17:44:01.371Z] removing /builds/worker/fetches
[fetches 2024-05-24T17:44:01.372Z] finished
[taskcluster 2024-05-24 17:44:01.597Z] === Task Finished ===
[taskcluster 2024-05-24 17:44:01.969Z] Successful task run with exit code: 0 completed in 4.019 seconds