Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clean corpus unable to fork sometimes on generic-worker/d2g #640

Closed
Tracked by #490 ...
bhearsum opened this issue May 27, 2024 · 1 comment
Closed
Tracked by #490 ...

clean corpus unable to fork sometimes on generic-worker/d2g #640

bhearsum opened this issue May 27, 2024 · 1 comment
Labels
taskcluster Issues related to the Taskcluster implementation of the training pipeline

Comments

@bhearsum
Copy link
Collaborator

For example:

[task 2024-05-24T00:19:51.583Z] ### Cleaning /builds/worker/fetches/NLLB_v1 with filter /builds/worker/artifacts/NLLB_v1.en-uk.filters.json
[task 2024-05-24T00:19:51.584Z] + opuscleaner-clean --parallel 32 --batch-size=50000 --input=- /builds/worker/artifacts/NLLB_v1.en-uk.filters.json en uk
[task 2024-05-24T00:19:51.584Z] + cut -f2
[task 2024-05-24T00:19:51.584Z] ++ zstdmt -dc /builds/worker/fetches/NLLB_v1.en.zst
[task 2024-05-24T00:19:51.584Z] + paste /dev/fd/63 /dev/fd/62
[task 2024-05-24T00:19:51.584Z] + zstdmt
[task 2024-05-24T00:19:51.584Z] + tee /dev/fd/63
[task 2024-05-24T00:19:51.584Z] ++ zstdmt -dc /builds/worker/fetches/NLLB_v1.uk.zst
[task 2024-05-24T00:19:51.585Z] ++ cut -f1
[task 2024-05-24T00:19:51.585Z] ++ zstdmt
[task 2024-05-24T00:19:56.663Z] [24/4:deescape-special-chars] Error: [Errno 11] Resource temporarily unavailable
[task 2024-05-24T00:19:56.663Z] [24/1:normalize_whitespace] Error: can't start new thread
[task 2024-05-24T00:19:56.664Z] [24/2:normalize_whitespace] Error: can't start new thread
[task 2024-05-24T00:19:56.670Z] [24/0:remove_empty_lines] Traceback (most recent call last):
[task 2024-05-24T00:19:56.670Z] [24/0:remove_empty_lines]   File "/builds/worker/.local/lib/python3.10/site-packages/opuscleaner/filters/./remove_empty_lines.py", line 16, in <module>
[task 2024-05-24T00:19:56.672Z] [24/0:remove_empty_lines]     main()
[task 2024-05-24T00:19:56.672Z] [24/0:remove_empty_lines]   File "/builds/worker/.local/lib/python3.10/site-packages/opuscleaner/filters/./remove_empty_lines.py", line 13, in main
[task 2024-05-24T00:19:56.672Z] [24/0:remove_empty_lines]     sys.stdout.write(line)
[task 2024-05-24T00:19:56.672Z] [24/0:remove_empty_lines] BrokenPipeError: [Errno 32] Broken pipe
[task 2024-05-24T00:19:56.787Z] [25/3:deescape-special-chars] Error: can't start new thread
[task 2024-05-24T00:19:56.787Z] [25/6:max_length] /bin/sh: 1: Cannot fork

(From https://firefox-ci-tc.services.mozilla.com/tasks/eYPAybjqQG27x-Y3SxespA)

Seems like it might be some sort of resource allocation issue with podman? Unclear to me so far.

@bhearsum
Copy link
Collaborator Author

bhearsum commented Jul 4, 2024

Same root cause as #630.

@bhearsum bhearsum closed this as completed Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
taskcluster Issues related to the Taskcluster implementation of the training pipeline
Projects
None yet
Development

No branches or pull requests

2 participants