[taskcluster 2023-12-15 19:03:18.839Z] Task ID: KAFn2cQEQgSu_gb9x2LXpg
[taskcluster 2023-12-15 19:03:18.839Z] Worker ID: 7166567156057971816
[taskcluster 2023-12-15 19:03:18.839Z] Worker Group: us-west1
[taskcluster 2023-12-15 19:03:18.839Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2023-12-15 19:03:18.839Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2023-12-15 19:03:18.839Z] Worker Version: 38.0.5
[taskcluster 2023-12-15 19:03:18.839Z] Public IP: 34.83.126.14
[taskcluster 2023-12-15 19:03:18.839Z] Hostname: translations-1-b-linux-large-gcp-300gb-wgd0lpkstkym7a7za5gjcq
[taskcluster 2023-12-15 19:03:18.839Z] using cache "translations-level-1-checkouts-hg58-v3-2922149087d95338df26" -> /builds/worker/checkouts

[taskcluster 2023-12-15 19:03:19.278Z] Image 'public/image.tar.zst' from task 'coL2A8X7TtKPvOvddcYtiw' loaded.  Using image ID sha256:a6912c5313466b965d7bb74ac22f26d3fe1e8789f9d2bbd3c516122194ef152e.
[taskcluster 2023-12-15 19:03:19.289Z] === Task Starting ===
[setup 2023-12-15T19:03:19.607Z] run-task started in /builds/worker
[setup 2023-12-15T19:03:19.607Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/clean/merge-mono.sh /builds/worker/artifacts/mono.$LOCALE.zst 10000 $MOZ_FETCHES_DIR/*.zst
[setup 2023-12-15T19:03:19.607Z] Python version: 3.10.12
[cache 2023-12-15T19:03:19.609Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2023-12-15T19:03:19.609Z] volume /builds/worker/checkouts is a cache
[setup 2023-12-15T19:03:19.609Z] running as worker:worker
[vcs 2023-12-15T19:03:19.609Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/bhearsum/firefox-translations-training', 'followup']
[vcs 2023-12-15T19:03:19.917Z] From https://github.com/bhearsum/firefox-translations-training
[vcs 2023-12-15T19:03:19.917Z]  * branch            followup   -> FETCH_HEAD
[vcs 2023-12-15T19:03:19.921Z] executing ['git', 'checkout', '-f', '-B', 'followup', '4aa774c2a9f31b93627bcac82e0c235839cb7c0e']
[vcs 2023-12-15T19:03:19.925Z] Reset branch 'followup'
[vcs 2023-12-15T19:03:19.925Z] executing ['git', 'submodule', 'init']
[vcs 2023-12-15T19:03:19.942Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2023-12-15T19:03:19.974Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2023-12-15T19:03:19.983Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2023-12-15T19:03:19.993Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2023-12-15T19:03:20.004Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2023-12-15T19:03:20.018Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2023-12-15T19:03:20.028Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2023-12-15T19:03:20.029Z] cleaning git checkout...
[vcs 2023-12-15T19:03:20.029Z] executing ['git', 'clean', '-nxdff']
[vcs 2023-12-15T19:03:20.031Z] removing []
[vcs 2023-12-15T19:03:20.031Z] successfully cleaned git checkout!
[vcs 2023-12-15T19:03:20.032Z] TinderboxPrint:<a href='https://github.com/bhearsum/firefox-translations-training/commit/4aa774c2a9f31b93627bcac82e0c235839cb7c0e' title='Built from firefox-translations-training commit 4aa774c2a9f31b93627bcac82e0c235839cb7c0e'>4aa774c2a9f31b93627bcac82e0c235839cb7c0e</a>
[setup 2023-12-15T19:03:20.032Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2023-12-15T19:03:20.032Z] fetching artifacts
[fetches 2023-12-15T19:03:20.032Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xlf1U6RHSXCei3fDqbeb8Q/artifacts/public/build/news_2007.en.zst to /builds/worker/fetches/news_2007.en.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst to /builds/worker/fetches/dedupe.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xlf1U6RHSXCei3fDqbeb8Q/artifacts/public/build/news_2007.en.zst

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xlf1U6RHSXCei3fDqbeb8Q/artifacts/public/build/news_2007.en.zst resolved to 510318 bytes with sha256 b6ec9ca0e24646170e8bd4ce9e82745fe376003b685e43916b58a29a1f4e01bd in 0.098s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AgPlxUayScSDmqgddgY9pw/artifacts/public/build/dedupe.tar.zst resolved to 133246 bytes with sha256 1e567b7eabfbbac81c949ba165181bf3b4b5a15d4fcd114a0fc1d7ec82cfac07 in 0.110s
Extracting /builds/worker/fetches/dedupe.tar.zst to /builds/worker/fetches
/builds/worker/fetches/dedupe.tar.zst extracted in 0.002s
Removing /builds/worker/fetches/dedupe.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.1143343370000025, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2023-12-15T19:03:20.224Z] finished fetching artifacts
[task 2023-12-15T19:03:20.224Z] executing ['bash', '-c', 'export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/clean/merge-mono.sh /builds/worker/artifacts/mono.$LOCALE.zst 10000 $MOZ_FETCHES_DIR/*.zst']
[task 2023-12-15T19:03:20.226Z] + set -euo pipefail
[task 2023-12-15T19:03:20.226Z] + test -v BIN
[task 2023-12-15T19:03:20.226Z] + echo '###### Merging monolingual datasets'
[task 2023-12-15T19:03:20.226Z] ###### Merging monolingual datasets
[task 2023-12-15T19:03:20.226Z] + output=/builds/worker/artifacts/mono.en.zst
[task 2023-12-15T19:03:20.226Z] + max_sent=10000
[task 2023-12-15T19:03:20.226Z] + datasets=("${@:3}")
[task 2023-12-15T19:03:20.226Z] + COMPRESSION_CMD=zstdmt
[task 2023-12-15T19:03:20.227Z] ++ dirname /builds/worker/artifacts/mono.en.zst
[task 2023-12-15T19:03:20.227Z] + dir=/builds/worker/artifacts
[task 2023-12-15T19:03:20.227Z] + mkdir -p /builds/worker/artifacts
[task 2023-12-15T19:03:20.228Z] + zstdmt -dc /builds/worker/fetches/news_2007.en.zst
[task 2023-12-15T19:03:20.229Z] + /builds/worker/fetches/dedupe
[task 2023-12-15T19:03:20.229Z] + shuf -n 10000
[task 2023-12-15T19:03:20.229Z] + zstdmt
[task 2023-12-15T19:03:20.230Z] File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
[task 2023-12-15T19:03:20.235Z] Kept 9932 / 9932 = 1
[task 2023-12-15T19:03:20.250Z] + echo '###### Done: Merging monolingual datasets'
[task 2023-12-15T19:03:20.250Z] ###### Done: Merging monolingual datasets
[fetches 2023-12-15T19:03:20.250Z] removing /builds/worker/fetches
[fetches 2023-12-15T19:03:20.250Z] finished
[taskcluster 2023-12-15 19:03:20.438Z] === Task Finished ===
[taskcluster 2023-12-15 19:03:20.629Z] Successful task run with exit code: 0 completed in 1.79 seconds