Mpes process files from memory

[rettigl]

Processing directly from the files for mpes is rather slow if reading from a filesystem with poor seeking performance (e.g. hdd, network drives). To overcome this, we load all files completely into memory for each binning step, and then access them from there.

[coveralls]

Pull Request Test Coverage Report for Build 12371852256

Details

• 17 of 19 (89.47%) changed or added relevant lines in 1 file are covered.
• 3 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.03%) to 91.759%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/loader/mpes/loader.py	17	19	89.47%

Files with Coverage Reduction	New Missed Lines	%
sed/binning/numba_bin.py	3	87.74%

Totals
Change from base Build 12349863088:	-0.03%
Covered Lines:	6525
Relevant Lines:	7111

---

💛 - Coveralls

[rettigl]

@zain-sohail, this is mpes-specific, maybe you can give it a brief look and commend/approve?

[zain-sohail]

would this still work with smaller memory setups?

[rettigl]

would this still work with smaller memory setups?

We are talking about additional ~100 mbyte memory per compute core, compared to several GByte for the actual binning volume, so typically only an increase of 10% or less in memory consumption. You can always compensate with using less cores if memory is an issue.