Worker: Use generator functions to reduce memory load

[jan-janssen]

No description provided.

[pmrv]

We should find a way to check what is actually causing the memory increase. I cannot image that keeping the lists around that store the job ids actually cause a significant memory penalty unless your are submitted upwards of 1e9 jobs. My feeling is rather the we have a memory leak somewhere when running jobs and should look into this, maybe something like this old issue.

[jan-janssen]

So my issue is that the worker runs into a memory issue when running with 510 jobs. I am not exactly sure if this is the main process of the worker or the sum of sub process. I looked into different spawn methods but that did not help. I also tried to only execute the executable in the subprocess #722 .

[jan-janssen]

File "D:\a\pyiron_base\pyiron_base\pyiron_base\job\worker.py", line 12, in <module>
[275](https://github.com/pyiron/pyiron_base/runs/5538617780?check_suite_focus=true#step:5:275)
    import resource
[276](https://github.com/pyiron/pyiron_base/runs/5538617780?check_suite_focus=true#step:5:276)
ModuleNotFoundError: No module named 'resource'

[jan-janssen]

Unfortunately the resource package is unix specific https://docs.python.org/3/library/resource.html

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.