[taskcluster 2024-02-28 02:52:33.822Z] Task ID: PjALbiBNQP2iXS_AzLlvGg
[taskcluster 2024-02-28 02:52:33.822Z] Worker ID: 664258403898696148
[taskcluster 2024-02-28 02:52:33.822Z] Worker Group: us-west1
[taskcluster 2024-02-28 02:52:33.822Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-02-28 02:52:33.822Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-02-28 02:52:33.822Z] Worker Version: 38.0.5
[taskcluster 2024-02-28 02:52:33.822Z] Public IP: 35.227.148.246
[taskcluster 2024-02-28 02:52:33.822Z] Hostname: translations-1-b-linux-large-gcp-300gb-sug-adt9rluwi4lp0kkaog
[taskcluster 2024-02-28 02:52:33.822Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-e_fnyhlhT6CGxAxd6QIFSA" -> /builds/worker/checkouts

[taskcluster 2024-02-28 02:52:34.257Z] Image 'public/image.tar.zst' from task 'e_fnyhlhT6CGxAxd6QIFSA' loaded.  Using image ID sha256:4a564314aaa610230a33ae7b01128154df8057c24ea44b0f87454c1d2ac59a3d.
[taskcluster 2024-02-28 02:52:34.268Z] === Task Starting ===
[setup 2024-02-28T02:52:34.580Z] run-task started in /builds/worker
[setup 2024-02-28T02:52:34.580Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/translate/merge-corpus.sh $MOZ_FETCHES_DIR/corpus.ru.zst $MOZ_FETCHES_DIR/mono.ru.zst $MOZ_FETCHES_DIR/corpus.en.zst $MOZ_FETCHES_DIR/mono.en.zst /builds/worker/artifacts/corpus.ru.zst /builds/worker/artifacts/corpus.en.zst
[setup 2024-02-28T02:52:34.580Z] Python version: 3.10.12
[cache 2024-02-28T02:52:34.582Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-02-28T02:52:34.582Z] volume /builds/worker/checkouts is a cache
[setup 2024-02-28T02:52:34.582Z] running as worker:worker
[vcs 2024-02-28T02:52:34.583Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-02-28T02:52:34.584Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/bhearsum/firefox-translations-training', 'term']
[vcs 2024-02-28T02:52:34.851Z] From https://github.com/bhearsum/firefox-translations-training
[vcs 2024-02-28T02:52:34.851Z]  * branch            term       -> FETCH_HEAD
[vcs 2024-02-28T02:52:34.856Z] executing ['git', 'checkout', '-f', '-B', 'term', '38340942e4bdac1ff2749e0647f6fc7f10d602db']
[vcs 2024-02-28T02:52:34.910Z] Reset branch 'term'
[vcs 2024-02-28T02:52:34.911Z] executing ['git', 'submodule', 'init']
[vcs 2024-02-28T02:52:34.929Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-02-28T02:52:35.013Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-02-28T02:52:35.023Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-02-28T02:52:35.034Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-02-28T02:52:35.058Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-02-28T02:52:35.172Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-02-28T02:52:35.191Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-02-28T02:52:35.191Z] cleaning git checkout...
[vcs 2024-02-28T02:52:35.191Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-02-28T02:52:35.193Z] removing []
[vcs 2024-02-28T02:52:35.193Z] successfully cleaned git checkout!
[vcs 2024-02-28T02:52:35.195Z] TinderboxPrint:<a href='https://github.com/bhearsum/firefox-translations-training/commit/38340942e4bdac1ff2749e0647f6fc7f10d602db' title='Built from firefox-translations-training commit 38340942e4bdac1ff2749e0647f6fc7f10d602db'>38340942e4bdac1ff2749e0647f6fc7f10d602db</a>
[setup 2024-02-28T02:52:35.195Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-02-28T02:52:35.195Z] fetching artifacts
[fetches 2024-02-28T02:52:35.195Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AsKi5ARHQWuvSJRdpbqjMg/artifacts/public/build/corpus.en.zst to /builds/worker/fetches/corpus.en.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I3vnNnuNRWKaw_m-WLqCrg/artifacts/public/build/mono.en.zst to /builds/worker/fetches/mono.en.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I3vnNnuNRWKaw_m-WLqCrg/artifacts/public/build/mono.en.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AsKi5ARHQWuvSJRdpbqjMg/artifacts/public/build/corpus.en.zst

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IoVWhulTSZitG0cbzcVFlQ/artifacts/public/build/mono.ru.zst to /builds/worker/fetches/mono.ru.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IoVWhulTSZitG0cbzcVFlQ/artifacts/public/build/mono.ru.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst to /builds/worker/fetches/dedupe.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RhekS-dRTlyIyxdFBYy3bA/artifacts/public/build/corpus.ru.zst to /builds/worker/fetches/corpus.ru.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RhekS-dRTlyIyxdFBYy3bA/artifacts/public/build/corpus.ru.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst resolved to 133246 bytes with sha256 1e567b7eabfbbac81c949ba165181bf3b4b5a15d4fcd114a0fc1d7ec82cfac07 in 0.079s
Extracting /builds/worker/fetches/dedupe.tar.zst to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I3vnNnuNRWKaw_m-WLqCrg/artifacts/public/build/mono.en.zst resolved to 65672 bytes with sha256 7623827a4e753ef25a9c00f56ed2022d15d407306ff419f20dffccbb3eb078b2 in 0.083s
/builds/worker/fetches/dedupe.tar.zst extracted in 0.002s
Removing /builds/worker/fetches/dedupe.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AsKi5ARHQWuvSJRdpbqjMg/artifacts/public/build/corpus.en.zst resolved to 77301 bytes with sha256 b42270723c954ac7152395fcd3e670e8d58f94bdb2254758383fcc5fd9001c7b in 0.104s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IoVWhulTSZitG0cbzcVFlQ/artifacts/public/build/mono.ru.zst resolved to 581349 bytes with sha256 11b186135b74e934f71fe1ddbd5fc691b62d63d01dd2be8e921117e8835c02f3 in 0.113s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RhekS-dRTlyIyxdFBYy3bA/artifacts/public/build/corpus.ru.zst resolved to 382063 bytes with sha256 441664c9142c429483b99094ad680ea120e6040667f0090f1023192b1de8c54b in 0.228s
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.23310788099999513, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-02-28T02:52:35.507Z] finished fetching artifacts
[task 2024-02-28T02:52:35.507Z] executing ['bash', '-c', 'export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/translate/merge-corpus.sh $MOZ_FETCHES_DIR/corpus.ru.zst $MOZ_FETCHES_DIR/mono.ru.zst $MOZ_FETCHES_DIR/corpus.en.zst $MOZ_FETCHES_DIR/mono.en.zst /builds/worker/artifacts/corpus.ru.zst /builds/worker/artifacts/corpus.en.zst']
[task 2024-02-28T02:52:35.509Z] + set -euo pipefail
[task 2024-02-28T02:52:35.509Z] + test -v BIN
[task 2024-02-28T02:52:35.509Z] + echo '###### Merging datasets'
[task 2024-02-28T02:52:35.509Z] ###### Merging datasets
[task 2024-02-28T02:52:35.509Z] + src1=/builds/worker/fetches/corpus.ru.zst
[task 2024-02-28T02:52:35.509Z] + src2=/builds/worker/fetches/mono.ru.zst
[task 2024-02-28T02:52:35.509Z] + trg1=/builds/worker/fetches/corpus.en.zst
[task 2024-02-28T02:52:35.509Z] + trg2=/builds/worker/fetches/mono.en.zst
[task 2024-02-28T02:52:35.509Z] + res_src=/builds/worker/artifacts/corpus.ru.zst
[task 2024-02-28T02:52:35.509Z] + res_trg=/builds/worker/artifacts/corpus.en.zst
[task 2024-02-28T02:52:35.509Z] + COMPRESSION_CMD=zstdmt
[task 2024-02-28T02:52:35.509Z] + ARTIFACT_EXT=zst
[task 2024-02-28T02:52:35.509Z] ++ dirname /builds/worker/artifacts/corpus.ru.zst
[task 2024-02-28T02:52:35.510Z] + tmp_dir=/builds/worker/artifacts/tmp
[task 2024-02-28T02:52:35.510Z] + mkdir -p /builds/worker/artifacts/tmp
[task 2024-02-28T02:52:35.511Z] + zstdmt
[task 2024-02-28T02:52:35.512Z] + cat /dev/fd/63 /dev/fd/62
[task 2024-02-28T02:52:35.512Z] ++ zstdmt -dc /builds/worker/fetches/corpus.ru.zst
[task 2024-02-28T02:52:35.512Z] ++ zstdmt -dc /builds/worker/fetches/mono.ru.zst
[task 2024-02-28T02:52:35.545Z] + zstdmt
[task 2024-02-28T02:52:35.545Z] + cat /dev/fd/63 /dev/fd/62
[task 2024-02-28T02:52:35.545Z] ++ zstdmt -dc /builds/worker/fetches/corpus.en.zst
[task 2024-02-28T02:52:35.545Z] ++ zstdmt -dc /builds/worker/fetches/mono.en.zst
[task 2024-02-28T02:52:35.559Z] + echo '#### Deduplicating'
[task 2024-02-28T02:52:35.559Z] #### Deduplicating
[task 2024-02-28T02:52:35.560Z] + /builds/worker/fetches/dedupe
[task 2024-02-28T02:52:35.560Z] + zstdmt
[task 2024-02-28T02:52:35.560Z] + paste /dev/fd/63 /dev/fd/62
[task 2024-02-28T02:52:35.560Z] ++ zstdmt -dc /builds/worker/artifacts/tmp/original.src.zst
[task 2024-02-28T02:52:35.560Z] + shuf --random-source=/dev/fd/63
[task 2024-02-28T02:52:35.560Z] ++ get_seeded_random 42
[task 2024-02-28T02:52:35.560Z] ++ zstdmt -dc /builds/worker/artifacts/tmp/original.trg.zst
[task 2024-02-28T02:52:35.560Z] ++ seed=42
[task 2024-02-28T02:52:35.560Z] ++ openssl enc -aes-256-ctr -pass pass:42 -nosalt
[task 2024-02-28T02:52:35.561Z] File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
[task 2024-02-28T02:52:35.648Z] Kept 16987 / 16994 = 0.999588
[task 2024-02-28T02:52:35.676Z] + zstdmt -dc /builds/worker/artifacts/tmp/all.zst
[task 2024-02-28T02:52:35.676Z] + cut -f1
[task 2024-02-28T02:52:35.676Z] + zstdmt
[task 2024-02-28T02:52:35.748Z] + zstdmt -dc /builds/worker/artifacts/tmp/all.zst
[task 2024-02-28T02:52:35.748Z] + cut -f2
[task 2024-02-28T02:52:35.748Z] + zstdmt
[task 2024-02-28T02:52:35.805Z] ++ zstdmt -dc /builds/worker/artifacts/corpus.ru.zst
[task 2024-02-28T02:52:35.805Z] ++ wc -l
[task 2024-02-28T02:52:35.813Z] + src_len=16987
[task 2024-02-28T02:52:35.814Z] ++ zstdmt -dc /builds/worker/artifacts/corpus.en.zst
[task 2024-02-28T02:52:35.814Z] ++ wc -l
[task 2024-02-28T02:52:35.820Z] + trg_len=16987
[task 2024-02-28T02:52:35.820Z] + '[' 16987 '!=' 16987 ']'
[task 2024-02-28T02:52:35.820Z] + rm -rf /builds/worker/artifacts/tmp
[task 2024-02-28T02:52:35.821Z] + echo '###### Done: Merging datasets'
[task 2024-02-28T02:52:35.822Z] ###### Done: Merging datasets
[fetches 2024-02-28T02:52:35.822Z] removing /builds/worker/fetches
[fetches 2024-02-28T02:52:35.822Z] finished
[taskcluster 2024-02-28 02:52:36.041Z] === Task Finished ===
[taskcluster 2024-02-28 02:52:36.449Z] Successful task run with exit code: 0 completed in 2.628 seconds