Pushing causes "Too many open files" (EMFILE)

[Cynical-Optimist]

See original issue on GitLab
In GitLab by [Gitlab user @valentindavid] on Aug 11, 2020, 14:37

Summary

On a builder for Freedesktop SDK we often have failures happening during push.

Note, most of the time there is nothing to push, the ref is already on all the servers. And the error happens.

The pushers number is default (should be 4). The fetchers number is reduced to 5.

Example of logs:

• https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/680262924
• https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/679162774

This is using BuildStream 1.4.3.

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Possible fixes

Other relevant information

• BuildStream version affected: /milestone %BuildStream_v1.x

---

[Cynical-Optimist]

In GitLab by [Gitlab user @valentindavid] on Sep 28, 2020, 10:42

mentioned in merge request freedesktop-sdk/infrastructure/freedesktop-sdk-docker-images!143

[Cynical-Optimist]

In GitLab by [Gitlab user @nanonyme] on Oct 7, 2020, 18:28

[Gitlab user @juergbi] explained to me on IRC the way bst1 uses FUSE results in large amounts of fd's used. The 1024 files open max that freedesktop-sdk has for x86_64/i686 is simply too little. bst2 may end up using less but regardless, this seems like a user error based on investigation.

[Cynical-Optimist]

In GitLab by [Gitlab user @nanonyme] on Oct 8, 2020, 22:21

It would probably be nice if bst1 issued warning if nofile hard limit is too low.