I/O / data iteration / file sizes

[tdixon97]

We should make sure to handle the buffering / IO / memory usage of reboost. For example in our current (legacy) L-200 simulations stack often some processes use > 10 GB memory meaning few processes can be run, since some simulations (2vbb) are very large.

• currently for the hit tier (for hpge) I read the data in chunks and this works fine
• however for events / TCM it is not possible and the full hit tier file is read in memory (similar to evt tier in data)
• we could consider adding some limit on the maximum hit file size? Or an option to generate multiple files?

Maybe we should review the overall strategy @gipert / @ManuelHu ?

[ManuelHu]

on the optical part: the actual files containing stp data or my optmap "evt" files (just summed hitcounts) are iterated and should not take too much memory.

optical maps on the other hand can get huge (we are talking about > 50 GB for all individual channels together). So this is certainly a problem.

• For optical map creation this is solved by using multiprocessing and shared memory (multiprocessing for optical map building #7)
• For the consumption of the optical maps, no solution exists at the moment