[taskcluster 2023-12-20 15:44:46.034Z] Task ID: asY3dw0rQ1WDzE5PPyxnXg
[taskcluster 2023-12-20 15:44:46.034Z] Worker ID: 9098964554170153616
[taskcluster 2023-12-20 15:44:46.034Z] Worker Group: us-central1
[taskcluster 2023-12-20 15:44:46.034Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2023-12-20 15:44:46.034Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2023-12-20 15:44:46.034Z] Worker Version: 38.0.5
[taskcluster 2023-12-20 15:44:46.034Z] Public IP: 34.132.154.70
[taskcluster 2023-12-20 15:44:46.034Z] Hostname: translations-1-b-linux-large-gcp-300gb-aykln-kprnqufopxtsrakw
[taskcluster 2023-12-20 15:44:46.034Z] using cache "translations-level-1-checkouts-v3-58974d7dcf0417b3fe53-ARGg7JgtR3C0C2BCSaAWqA" -> /builds/worker/checkouts

[taskcluster 2023-12-20 15:44:46.635Z] Image 'public/image.tar.zst' from task 'ARGg7JgtR3C0C2BCSaAWqA' loaded.  Using image ID sha256:fa53620b2ece3428f41eb15ffb863544aa69d70369d1fa2d4412339db9f7115c.
[taskcluster 2023-12-20 15:44:46.683Z] === Task Starting ===
[setup 2023-12-20T15:44:46.996Z] run-task started in /builds/worker
[setup 2023-12-20T15:44:46.996Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/translate/merge-corpus.sh $MOZ_FETCHES_DIR/corpus.ru.zst $MOZ_FETCHES_DIR/mono.ru.zst $MOZ_FETCHES_DIR/corpus.en.zst $MOZ_FETCHES_DIR/mono.en.zst /builds/worker/artifacts/corpus.ru.zst /builds/worker/artifacts/corpus.en.zst
[setup 2023-12-20T15:44:46.996Z] Python version: 3.10.12
[cache 2023-12-20T15:44:46.998Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2023-12-20T15:44:46.998Z] volume /builds/worker/checkouts is a cache
[setup 2023-12-20T15:44:46.998Z] running as worker:worker
[vcs 2023-12-20T15:44:46.998Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2023-12-20T15:44:46.999Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/bhearsum/firefox-translations-training', 'taskgraph-7']
[vcs 2023-12-20T15:44:47.202Z] From https://github.com/bhearsum/firefox-translations-training
[vcs 2023-12-20T15:44:47.202Z]  * branch            taskgraph-7 -> FETCH_HEAD
[vcs 2023-12-20T15:44:47.207Z] executing ['git', 'checkout', '-f', '-B', 'taskgraph-7', '084cd68b442168a104ef509f6ea7bf320b139e2a']
[vcs 2023-12-20T15:44:47.241Z] Reset branch 'taskgraph-7'
[vcs 2023-12-20T15:44:47.241Z] executing ['git', 'submodule', 'init']
[vcs 2023-12-20T15:44:47.259Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2023-12-20T15:44:47.342Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2023-12-20T15:44:47.353Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2023-12-20T15:44:47.363Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2023-12-20T15:44:47.387Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2023-12-20T15:44:47.501Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2023-12-20T15:44:47.519Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2023-12-20T15:44:47.519Z] cleaning git checkout...
[vcs 2023-12-20T15:44:47.519Z] executing ['git', 'clean', '-nxdff']
[vcs 2023-12-20T15:44:47.521Z] removing []
[vcs 2023-12-20T15:44:47.522Z] successfully cleaned git checkout!
[vcs 2023-12-20T15:44:47.523Z] TinderboxPrint:<a href='https://github.com/bhearsum/firefox-translations-training/commit/084cd68b442168a104ef509f6ea7bf320b139e2a' title='Built from firefox-translations-training commit 084cd68b442168a104ef509f6ea7bf320b139e2a'>084cd68b442168a104ef509f6ea7bf320b139e2a</a>
[setup 2023-12-20T15:44:47.523Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2023-12-20T15:44:47.523Z] fetching artifacts
[fetches 2023-12-20T15:44:47.523Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NeSTScKBTvi4prQHfLt7Pw/artifacts/public/build/corpus.en.zst to /builds/worker/fetches/corpus.en.zstattempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IKdhXN_UQcWlK99AzBoQ4A/artifacts/public/build/mono.en.zst to /builds/worker/fetches/mono.en.zst

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NeSTScKBTvi4prQHfLt7Pw/artifacts/public/build/corpus.en.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LmuTtggrQmGLG-wnrmBALw/artifacts/public/build/corpus.ru.zst to /builds/worker/fetches/corpus.ru.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IKdhXN_UQcWlK99AzBoQ4A/artifacts/public/build/mono.en.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LmuTtggrQmGLG-wnrmBALw/artifacts/public/build/corpus.ru.zstattempt 1/5

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fsh5EyQpSuyeGtJN9h_xgw/artifacts/public/build/mono.ru.zst to /builds/worker/fetches/mono.ru.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst to /builds/worker/fetches/dedupe.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fsh5EyQpSuyeGtJN9h_xgw/artifacts/public/build/mono.ru.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LmuTtggrQmGLG-wnrmBALw/artifacts/public/build/corpus.ru.zst resolved to 379559 bytes with sha256 dcfb5b5d3ecf6fa76671c33bcfd69b0b793352405e9e92e51b639d9bd5ee7815 in 0.173s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IKdhXN_UQcWlK99AzBoQ4A/artifacts/public/build/mono.en.zst resolved to 78764 bytes with sha256 6f534bb1e71014d959bb890b6dc8407b2d8832a6b3a05ff3a57e9bc5ba1d6c90 in 0.180s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NeSTScKBTvi4prQHfLt7Pw/artifacts/public/build/corpus.en.zst resolved to 91365 bytes with sha256 57316917f047c090cb3679af5233c422aa54c0b0ff575a9c7505224159be2d4b in 0.195s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fsh5EyQpSuyeGtJN9h_xgw/artifacts/public/build/mono.ru.zst resolved to 579339 bytes with sha256 2db63d6cdf5a15265b06cabd937bea6b3eef7ae669c16efa7207c823d75c4e55 in 0.251s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst resolved to 133246 bytes with sha256 1e567b7eabfbbac81c949ba165181bf3b4b5a15d4fcd114a0fc1d7ec82cfac07 in 0.297s
Extracting /builds/worker/fetches/dedupe.tar.zst to /builds/worker/fetches
/builds/worker/fetches/dedupe.tar.zst extracted in 0.002s
Removing /builds/worker/fetches/dedupe.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.303738261000035, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2023-12-20T15:44:47.902Z] finished fetching artifacts
[task 2023-12-20T15:44:47.902Z] executing ['bash', '-c', 'export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/translate/merge-corpus.sh $MOZ_FETCHES_DIR/corpus.ru.zst $MOZ_FETCHES_DIR/mono.ru.zst $MOZ_FETCHES_DIR/corpus.en.zst $MOZ_FETCHES_DIR/mono.en.zst /builds/worker/artifacts/corpus.ru.zst /builds/worker/artifacts/corpus.en.zst']
[task 2023-12-20T15:44:47.904Z] + set -euo pipefail
[task 2023-12-20T15:44:47.904Z] + test -v BIN
[task 2023-12-20T15:44:47.904Z] + echo '###### Merging datasets'
[task 2023-12-20T15:44:47.904Z] ###### Merging datasets
[task 2023-12-20T15:44:47.904Z] + src1=/builds/worker/fetches/corpus.ru.zst
[task 2023-12-20T15:44:47.904Z] + src2=/builds/worker/fetches/mono.ru.zst
[task 2023-12-20T15:44:47.904Z] + trg1=/builds/worker/fetches/corpus.en.zst
[task 2023-12-20T15:44:47.904Z] + trg2=/builds/worker/fetches/mono.en.zst
[task 2023-12-20T15:44:47.904Z] + res_src=/builds/worker/artifacts/corpus.ru.zst
[task 2023-12-20T15:44:47.904Z] + res_trg=/builds/worker/artifacts/corpus.en.zst
[task 2023-12-20T15:44:47.904Z] + COMPRESSION_CMD=zstdmt
[task 2023-12-20T15:44:47.904Z] + ARTIFACT_EXT=zst
[task 2023-12-20T15:44:47.905Z] ++ dirname /builds/worker/artifacts/corpus.ru.zst
[task 2023-12-20T15:44:47.905Z] + tmp_dir=/builds/worker/artifacts/tmp
[task 2023-12-20T15:44:47.905Z] + mkdir -p /builds/worker/artifacts/tmp
[task 2023-12-20T15:44:47.907Z] + zstdmt
[task 2023-12-20T15:44:47.907Z] ++ zstdmt -dc /builds/worker/fetches/corpus.ru.zst
[task 2023-12-20T15:44:47.907Z] + cat /dev/fd/63 /dev/fd/62
[task 2023-12-20T15:44:47.907Z] ++ zstdmt -dc /builds/worker/fetches/mono.ru.zst
[task 2023-12-20T15:44:47.938Z] + zstdmt
[task 2023-12-20T15:44:47.938Z] ++ zstdmt -dc /builds/worker/fetches/corpus.en.zst
[task 2023-12-20T15:44:47.938Z] + cat /dev/fd/63 /dev/fd/62
[task 2023-12-20T15:44:47.938Z] ++ zstdmt -dc /builds/worker/fetches/mono.en.zst
[task 2023-12-20T15:44:47.959Z] + echo '#### Deduplicating'
[task 2023-12-20T15:44:47.959Z] #### Deduplicating
[task 2023-12-20T15:44:47.960Z] + /builds/worker/fetches/dedupe
[task 2023-12-20T15:44:47.960Z] + shuf --random-source=/dev/fd/63
[task 2023-12-20T15:44:47.960Z] + paste /dev/fd/63 /dev/fd/62
[task 2023-12-20T15:44:47.960Z] + zstdmt
[task 2023-12-20T15:44:47.960Z] ++ zstdmt -dc /builds/worker/artifacts/tmp/original.src.zst
[task 2023-12-20T15:44:47.960Z] ++ get_seeded_random 42
[task 2023-12-20T15:44:47.960Z] ++ zstdmt -dc /builds/worker/artifacts/tmp/original.trg.zst
[task 2023-12-20T15:44:47.960Z] ++ seed=42
[task 2023-12-20T15:44:47.960Z] ++ openssl enc -aes-256-ctr -pass pass:42 -nosalt
[task 2023-12-20T15:44:47.961Z] File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
[task 2023-12-20T15:44:48.141Z] Kept 16842 / 16851 = 0.999466
[task 2023-12-20T15:44:48.152Z] + zstdmt -dc /builds/worker/artifacts/tmp/all.zst
[task 2023-12-20T15:44:48.152Z] + cut -f1
[task 2023-12-20T15:44:48.152Z] + zstdmt
[task 2023-12-20T15:44:48.254Z] + zstdmt -dc /builds/worker/artifacts/tmp/all.zst
[task 2023-12-20T15:44:48.254Z] + cut -f2
[task 2023-12-20T15:44:48.254Z] + zstdmt
[task 2023-12-20T15:44:48.393Z] ++ zstdmt -dc /builds/worker/artifacts/corpus.ru.zst
[task 2023-12-20T15:44:48.393Z] ++ wc -l
[task 2023-12-20T15:44:48.401Z] + src_len=16842
[task 2023-12-20T15:44:48.401Z] ++ zstdmt -dc /builds/worker/artifacts/corpus.en.zst
[task 2023-12-20T15:44:48.402Z] ++ wc -l
[task 2023-12-20T15:44:48.413Z] + trg_len=16842
[task 2023-12-20T15:44:48.414Z] + '[' 16842 '!=' 16842 ']'
[task 2023-12-20T15:44:48.414Z] + rm -rf /builds/worker/artifacts/tmp
[task 2023-12-20T15:44:48.415Z] + echo '###### Done: Merging datasets'
[task 2023-12-20T15:44:48.415Z] ###### Done: Merging datasets
[fetches 2023-12-20T15:44:48.415Z] removing /builds/worker/fetches
[fetches 2023-12-20T15:44:48.415Z] finished
[taskcluster 2023-12-20 15:44:48.654Z] === Task Finished ===
[taskcluster 2023-12-20 15:44:49.121Z] Successful task run with exit code: 0 completed in 3.088 seconds