Distribution of detector blocks across MPI processes

[anand-avinash]

I am opening this discussion to discuss about the distribution of detector blocks across the MPI processes. Andrea Sabatucci (@AndreaSaba94) and I are working on implementing the detector cross-talk in the framework. We were wondering about distributing the detector blocks in Observation class in a way that avoids MPI communication for cross-talk simulation. The ideal way to achieve it would be to group the detectors into blocks according to the squid chain they are connected to (each squid chain has its own cross-talk matrix) and then distribute the detector blocks (and their time blocks) across the MPI processes. This will make the set of TODs needed for cross-talk within a squid chain available on the same MPI process. Such a distribution scheme will also be useful for implementing the correlated noise among detectors, which I am planning to work on.

In the current version of the litebird_sim, the Observation class accepts a list of detectors and distributes them into n_blocks_det number of blocks evenly through the distribute_evenly() function. This doesn't allows the detectors to be grouped by any of their properties. So I am proposing to add functionalities in the Observation class that creates the detector blocks according to some of the detector properties. This will make the n_blocks_det to be set by the detector properties that we would like to group the detectors with. However, such an implementation would require re-writing most of the methods of Observation class and it may have effects on functions/classes elsewhere. Of course, all of these can be avoided if we simply use n_blocks_det = 1 while simulating cross-talk and correlated noise, but this requirement is restrictive.

So I am looking for your suggestions and feedback on how to move forward on this.

[paganol]

Hi @anand-avinash, I think your proposal is reasonable and something that should be done sooner or later. We had a similar discussion last year with @ziotom78 and @giuspugl, but then nothing came out, because we had other priorities at that time. Regarding the potential impact on other modules, I wouldn't be worried about it, we will deal with it when any problems arise.
Thanks for bringing this up and let me know if you need any help.