[taskcluster 2024-02-13 19:59:05.242Z] Task ID: Ai7Mm3H6Q3SRbQ3z8CcPFw
[taskcluster 2024-02-13 19:59:05.242Z] Worker ID: 2476314636612695959
[taskcluster 2024-02-13 19:59:05.242Z] Worker Group: us-central1
[taskcluster 2024-02-13 19:59:05.242Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-02-13 19:59:05.242Z] Worker Pool: translations-1/b-linux-large-gcp
[taskcluster 2024-02-13 19:59:05.242Z] Worker Version: 38.0.5
[taskcluster 2024-02-13 19:59:05.242Z] Public IP: 34.134.168.179
[taskcluster 2024-02-13 19:59:05.242Z] Hostname: translations-1-b-linux-large-gcp-v6aey3cpqpuvpvmoqlotog
[taskcluster 2024-02-13 19:59:05.242Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-Gudb01f2QISFkQNad5U_NA" -> /builds/worker/checkouts

[taskcluster 2024-02-13 19:59:07.772Z] Downloading artifact "public/image.tar.zst" from task ID: Gudb01f2QISFkQNad5U_NA.
[taskcluster 2024-02-13 19:59:11.897Z] Downloaded artifact successfully.
[taskcluster 2024-02-13 19:59:11.898Z] Downloaded 264.808 mb
[taskcluster 2024-02-13 19:59:11.898Z] Decompressing downloaded image
[taskcluster 2024-02-13 19:59:13.746Z] Loading docker image from downloaded archive.
[taskcluster 2024-02-13 19:59:23.454Z] Image 'public/image.tar.zst' from task 'Gudb01f2QISFkQNad5U_NA' loaded.  Using image ID sha256:654f27369c79de5913668b0c72131803788b72c6634577cb3426bfdc8fa57c40.
[taskcluster 2024-02-13 19:59:23.618Z] === Task Starting ===
[setup 2024-02-13T19:59:25.407Z] run-task started in /builds/worker
[setup 2024-02-13T19:59:25.407Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/clean/merge-corpus.sh artifacts/devset $MOZ_FETCHES_DIR/*.zst
[setup 2024-02-13T19:59:25.407Z] Python version: 3.10.12
[cache 2024-02-13T19:59:25.409Z] cache /builds/worker/checkouts is empty; writing requirements: gid=1000 uid=1000 version=1
[volume 2024-02-13T19:59:25.409Z] volume /builds/worker/checkouts is a cache
[setup 2024-02-13T19:59:25.409Z] running as worker:worker
[vcs 2024-02-13T19:59:25.409Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-02-13T19:59:25.411Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/builds/worker/checkouts/vcs']
[vcs 2024-02-13T19:59:25.413Z] Cloning into '/builds/worker/checkouts/vcs'...
[vcs 2024-02-13T19:59:26.129Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/bhearsum/firefox-translations-training', 'start-specific']
[vcs 2024-02-13T19:59:26.319Z] From https://github.com/bhearsum/firefox-translations-training
[vcs 2024-02-13T19:59:26.320Z]  * branch            start-specific -> FETCH_HEAD
[vcs 2024-02-13T19:59:26.326Z] executing ['git', 'checkout', '-f', '-B', 'start-specific', '37fbf272d7eb316897377144111a3ef057becfd4']
[vcs 2024-02-13T19:59:26.386Z] Switched to a new branch 'start-specific'
[vcs 2024-02-13T19:59:26.387Z] executing ['git', 'submodule', 'init']
[vcs 2024-02-13T19:59:26.408Z] Submodule '3rd_party/browsermt-marian-dev' (https://github.com/browsermt/marian-dev) registered for path '3rd_party/browsermt-marian-dev'
[vcs 2024-02-13T19:59:26.408Z] Submodule 'extract-lex' (https://github.com/marian-nmt/extract-lex) registered for path '3rd_party/extract-lex'
[vcs 2024-02-13T19:59:26.409Z] Submodule 'fast_align' (https://github.com/clab/fast_align) registered for path '3rd_party/fast_align'
[vcs 2024-02-13T19:59:26.409Z] Submodule '3rd_party/kenlm' (https://github.com/kpu/kenlm) registered for path '3rd_party/kenlm'
[vcs 2024-02-13T19:59:26.409Z] Submodule '3rd_party/marian-dev' (https://github.com/marian-nmt/marian-dev) registered for path '3rd_party/marian-dev'
[vcs 2024-02-13T19:59:26.410Z] Submodule '3rd_party/preprocess' (https://github.com/kpu/preprocess.git) registered for path '3rd_party/preprocess'
[vcs 2024-02-13T19:59:26.410Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-02-13T19:59:26.432Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/browsermt-marian-dev'...
[vcs 2024-02-13T19:59:27.609Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/extract-lex'...
[vcs 2024-02-13T19:59:27.912Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/fast_align'...
[vcs 2024-02-13T19:59:28.274Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/kenlm'...
[vcs 2024-02-13T19:59:28.915Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/marian-dev'...
[vcs 2024-02-13T19:59:30.301Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/preprocess'...
[vcs 2024-02-13T19:59:30.855Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-02-13T19:59:30.865Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-02-13T19:59:30.876Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-02-13T19:59:30.902Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-02-13T19:59:31.016Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-02-13T19:59:31.037Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-02-13T19:59:31.037Z] cleaning git checkout...
[vcs 2024-02-13T19:59:31.037Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-02-13T19:59:31.040Z] removing []
[vcs 2024-02-13T19:59:31.040Z] successfully cleaned git checkout!
[vcs 2024-02-13T19:59:31.042Z] TinderboxPrint:<a href='https://github.com/bhearsum/firefox-translations-training/commit/37fbf272d7eb316897377144111a3ef057becfd4' title='Built from firefox-translations-training commit 37fbf272d7eb316897377144111a3ef057becfd4'>37fbf272d7eb316897377144111a3ef057becfd4</a>
[setup 2024-02-13T19:59:31.042Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-02-13T19:59:31.042Z] fetching artifacts
[fetches 2024-02-13T19:59:31.042Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5attempt 1/5

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NgfWAG6bRF6K-Ug3qCOGdg/artifacts/public/build/dev.en.zst to /builds/worker/fetches/dev.en.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NgfWAG6bRF6K-Ug3qCOGdg/artifacts/public/build/dev.ru.zst to /builds/worker/fetches/dev.ru.zst

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PZ0aWkVaQ7i15XftLSNtfA/artifacts/public/build/aug-mix_wmt19.en.zst to /builds/worker/fetches/aug-mix_wmt19.en.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NgfWAG6bRF6K-Ug3qCOGdg/artifacts/public/build/dev.ru.zst
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PZ0aWkVaQ7i15XftLSNtfA/artifacts/public/build/aug-mix_wmt19.ru.zst to /builds/worker/fetches/aug-mix_wmt19.ru.zst
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NgfWAG6bRF6K-Ug3qCOGdg/artifacts/public/build/dev.en.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PZ0aWkVaQ7i15XftLSNtfA/artifacts/public/build/aug-mix_wmt19.en.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PZ0aWkVaQ7i15XftLSNtfA/artifacts/public/build/aug-mix_wmt19.ru.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/REKgFXLrT6CIU704yzczpA/artifacts/public/build/dedupe.tar.zst to /builds/worker/fetches/dedupe.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/REKgFXLrT6CIU704yzczpA/artifacts/public/build/dedupe.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PZ0aWkVaQ7i15XftLSNtfA/artifacts/public/build/aug-mix_wmt19.en.zst resolved to 95245 bytes with sha256 00997381beb3c8921b385547375493f8aa80fd3b394cca8153c801fa170c7202 in 0.105s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/REKgFXLrT6CIU704yzczpA/artifacts/public/build/dedupe.tar.zst resolved to 133246 bytes with sha256 7b7924f13c53e79ffcba7737004067f754e4eb2c01dfe4594f7cf856832c0050 in 0.105s
Extracting /builds/worker/fetches/dedupe.tar.zst to /builds/worker/fetches
/builds/worker/fetches/dedupe.tar.zst extracted in 0.003s
Removing /builds/worker/fetches/dedupe.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PZ0aWkVaQ7i15XftLSNtfA/artifacts/public/build/aug-mix_wmt19.ru.zst resolved to 120486 bytes with sha256 85df5784bfc02ae1898af8d1b5cf726fd0da0c21cc091f87aef6a3a237d176da in 0.112s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NgfWAG6bRF6K-Ug3qCOGdg/artifacts/public/build/dev.ru.zst resolved to 79378 bytes with sha256 210bb9c1d6fb5b6670a45331cf6eafda30c58531f79df4454707897254a1f73d in 0.157s
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NgfWAG6bRF6K-Ug3qCOGdg/artifacts/public/build/dev.en.zst resolved to 52931 bytes with sha256 34bb856629c69358a86d9f68c584bfb8cb2c6175c86dd7216f3ce2198eb4f36a in 0.195s
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.19950106899999298, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-02-13T19:59:31.331Z] finished fetching artifacts
[task 2024-02-13T19:59:31.331Z] executing ['bash', '-c', 'export BIN=$MOZ_FETCHES_DIR && $VCS_PATH/pipeline/clean/merge-corpus.sh artifacts/devset $MOZ_FETCHES_DIR/*.zst']
[task 2024-02-13T19:59:31.334Z] + set -euo pipefail
[task 2024-02-13T19:59:31.334Z] + echo '###### Merging parallel datasets'
[task 2024-02-13T19:59:31.334Z] ###### Merging parallel datasets
[task 2024-02-13T19:59:31.334Z] + test -v SRC
[task 2024-02-13T19:59:31.334Z] + test -v TRG
[task 2024-02-13T19:59:31.334Z] + test -v BIN
[task 2024-02-13T19:59:31.334Z] + output_prefix=artifacts/devset
[task 2024-02-13T19:59:31.334Z] + inputs=("${@:2}")
[task 2024-02-13T19:59:31.334Z] + COMPRESSION_CMD=zstdmt
[task 2024-02-13T19:59:31.334Z] + ARTIFACT_EXT=zst
[task 2024-02-13T19:59:31.334Z] + tmp=artifacts/devset/merge
[task 2024-02-13T19:59:31.334Z] + mkdir -p artifacts/devset/merge
[task 2024-02-13T19:59:31.335Z] + echo '### Merging'
[task 2024-02-13T19:59:31.335Z] ### Merging
[task 2024-02-13T19:59:31.335Z] + [[ /builds/worker/fetches/aug-mix_wmt19.en.zst == *.zst ]]
[task 2024-02-13T19:59:31.336Z] ++ echo /builds/worker/fetches/aug-mix_wmt19.en.zst /builds/worker/fetches/aug-mix_wmt19.ru.zst /builds/worker/fetches/dev.en.zst /builds/worker/fetches/dev.ru.zst
[task 2024-02-13T19:59:31.336Z] ++ tr ' ' '\n'
[task 2024-02-13T19:59:31.336Z] ++ grep ru.zst
[task 2024-02-13T19:59:31.336Z] ++ tr '\n' ' '
[task 2024-02-13T19:59:31.337Z] + cat /builds/worker/fetches/aug-mix_wmt19.ru.zst /builds/worker/fetches/dev.ru.zst
[task 2024-02-13T19:59:31.339Z] ++ echo /builds/worker/fetches/aug-mix_wmt19.en.zst /builds/worker/fetches/aug-mix_wmt19.ru.zst /builds/worker/fetches/dev.en.zst /builds/worker/fetches/dev.ru.zst
[task 2024-02-13T19:59:31.339Z] ++ tr ' ' '\n'
[task 2024-02-13T19:59:31.339Z] ++ tr '\n' ' '
[task 2024-02-13T19:59:31.339Z] ++ grep en.zst
[task 2024-02-13T19:59:31.340Z] + cat /builds/worker/fetches/aug-mix_wmt19.en.zst /builds/worker/fetches/dev.en.zst
[task 2024-02-13T19:59:31.341Z] + echo '### Deduplication'
[task 2024-02-13T19:59:31.341Z] ### Deduplication
[task 2024-02-13T19:59:31.342Z] + /builds/worker/fetches/dedupe
[task 2024-02-13T19:59:31.342Z] + zstdmt
[task 2024-02-13T19:59:31.342Z] ++ zstdmt -dc artifacts/devset/merge/corpus.ru.dup.zst
[task 2024-02-13T19:59:31.342Z] + paste /dev/fd/63 /dev/fd/62
[task 2024-02-13T19:59:31.342Z] ++ zstdmt -dc artifacts/devset/merge/corpus.en.dup.zst
[task 2024-02-13T19:59:31.344Z] File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
[task 2024-02-13T19:59:31.351Z] Kept 2997 / 2997 = 1
[task 2024-02-13T19:59:31.362Z] + zstdmt -dc artifacts/devset/merge.ruen.zst
[task 2024-02-13T19:59:31.362Z] + cut -f1
[task 2024-02-13T19:59:31.362Z] + zstdmt
[task 2024-02-13T19:59:31.375Z] + zstdmt -dc artifacts/devset/merge.ruen.zst
[task 2024-02-13T19:59:31.375Z] + cut -f2
[task 2024-02-13T19:59:31.375Z] + zstdmt
[task 2024-02-13T19:59:31.386Z] + rm -rf artifacts/devset/merge
[task 2024-02-13T19:59:31.387Z] + echo '###### Done: Merging parallel datasets'
[task 2024-02-13T19:59:31.387Z] ###### Done: Merging parallel datasets
[fetches 2024-02-13T19:59:31.387Z] removing /builds/worker/fetches
[fetches 2024-02-13T19:59:31.387Z] finished
[taskcluster 2024-02-13 19:59:31.738Z] === Task Finished ===
[taskcluster 2024-02-13 19:59:32.343Z] Successful task run with exit code: 0 completed in 27.102 seconds