litebird · mreineck · Oct 30, 2024 · Oct 30, 2024 · Oct 30, 2024 · Oct 31, 2024
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -87,6 +87,7 @@ jobs:
             for proc in 1 5 9 ; do
               echo "Running MPI test ($MPI) with $proc processes"
               PYTHONPATH=. mpiexec -n $proc python3 ./test/test_mpi.py
+              PYTHONPATH=. mpiexec -n $proc python3 ./test/test_detector_blocks.py
             done
 
       - name: Tests OpenMPI
@@ -95,4 +96,5 @@ jobs:
             for proc in 1 2 ; do
               echo "Running MPI test ($MPI) with $proc processes"
               PYTHONPATH=. mpiexec -n $proc python3 ./test/test_mpi.py
+              PYTHONPATH=. mpiexec -n $proc python3 ./test/test_detector_blocks.py
             done
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,7 +1,7 @@
 repos:
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version.
-  rev: v0.1.7
+  rev: v0.1.15
   hooks:
     # Run the linter.
     - id: ruff

diff --git a/Beam synthesis example.ipynb b/Beam synthesis example.ipynb
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,4 +1,19 @@
 # HEAD
+
+-   **Breaking change**: Change to the pointing API [#358](https://github.com/litebird/litebird_sim/pull/358), in detail:
+
+    1. DetectorInfo has three new attributes: pol_angle_rad (polarization angle), pol_efficiency (polarization efficiency)and mueller_hwp (mueller matrix of the HWP).
+
+    2. get_pointings() return only the orientation ψ of the detector, the polarization angle is a separate variable stored in the `Observation` class. The same class also handles the mueller_hwp for each detector, and it has a new bool variable `has_hwp` that is set to true if an HWP object is passed to `prepare_pointings()`.
+
+    3. The mock vPTEP IMo has been updated accordingly.
+
+    4. The `HWP` class has a new field called mueller, that contains the mueller matrix of the HWP.
+
+    5. The function `scan_map()` now handles three possible algebras: (i) no HWP, (ii) ideal HWP, (iii) generic optical chain.
+
+-   Implementation of distributing detectors across the MPI processes by grouping them according to given attributes [#334](https://github.com/litebird/litebird_sim/pull/334)
+
 -   **Breaking change**: `PointingSys()` now requires `Observation` as an argument. And several functions to add pointing systematics are merged into `left_multiply_syst_quats()`.
 
 -   **Breaking change**: Redefinition (ωt instead of 2ωt) of the hwp angle returned by `get_pointings()`. Change in the pointing returned by `Observation.get_pointings()`, now behaving as it was before this [commit](https://github.com/litebird/litebird_sim/pull/319/commits/b3bc3bb2049c152cc183d6cfc68f4598f5b93ec0). Documentation updated accordingly. [#340](https://github.com/litebird/litebird_sim/pull/340)

diff --git a/README.md b/README.md
@@ -128,7 +128,7 @@ Clicking on a dot reveals the `DetectorInfo` for that detector in real time, hig
 Additionally, if you agree during the conversation to generate a detector file,
 a list of starred detectors will be saved into a text file at the designated location after you closed the plot.
 
-![plot_fp_usage](https://private-user-images.githubusercontent.com/83496454/388083441-93876c66-8b61-40c0-b23f-79a6998e5e33.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzIzNjAzOTAsIm5iZiI6MTczMjM2MDA5MCwicGF0aCI6Ii84MzQ5NjQ1NC8zODgwODM0NDEtOTM4NzZjNjYtOGI2MS00MGMwLWIyM2YtNzlhNjk5OGU1ZTMzLmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTIzVDExMDgxMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJkMmM3NTYzMzc4MTU2NGU3ODE2YWMzZTFlZDVjZmEzODJhNWMwMjhhMDY4MzUxMWU3OGYzZmZiYWFiZDcwNDQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.zG2QVKA9VeL5j79jTjiPNQ9kH8uyxVSeCzd3GBj3QVk)
+![FP Usage](./images/plot_fp_usage.gif)
 
 ## Roadmap
 

diff --git a/default_imo/schema.json.gz b/default_imo/schema.json.gz
diff --git a/docs/source/beam_convolution.rst b/docs/source/beam_convolution.rst
@@ -0,0 +1,31 @@
+.. _beamconvolution:
+
+Convolve Alms with a Beam to fill a TOD
+=======================================
+
+Minimal documentation should contain:
+- quick descrition of the math (reference to papers)
+- conventions 
+- details on the interface and examples 
+- description of the SphericalHarmonics class and how to store alms
+- example with high level interface
+
+
+The framework provides :func:`.add_convolved_sky`, a routine which 
+takes sky and beam alms perform harmonic space convolution and fills
+the detector timestreams. It implements both the algebra for...  
+
+Methods of the Simulation class
+-------------------------------
+
+The class :class:`.Simulation` provides the function
+:func:`.Simulation.convolve_sky`, which takes sky alms and perform the
+convolution 
+
+API reference
+-------------
+
+.. automodule:: litebird_sim.beam_convolution
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/source/images/detector_block_communicator.png b/docs/source/images/detector_block_communicator.png
diff --git a/docs/source/images/detector_groups_case1.png b/docs/source/images/detector_groups_case1.png
diff --git a/docs/source/images/detector_groups_case2.png b/docs/source/images/detector_groups_case2.png
diff --git a/docs/source/images/detector_groups_case3.png b/docs/source/images/detector_groups_case3.png
diff --git a/docs/source/images/grid_communicator.png b/docs/source/images/grid_communicator.png
diff --git a/docs/source/images/time_block_communicator.png b/docs/source/images/time_block_communicator.png
diff --git a/docs/source/map_scanning.rst b/docs/source/map_scanning.rst
@@ -5,7 +5,39 @@ Scanning a map to fill a TOD
 
 The framework provides :func:`.scan_map`, a routine which scans an
 input map accordingly to the scanning strategy and fills the detector
-timestreams. You can fill with signal an existing TOD by using the
+timestreams. It implements three possible algebras:
+
+- No HWP:
+
+.. math:: 
+   d_t = T + \gamma \left(Q\cos(2\theta + 2\psi_t)+U\sin(2\theta + 2\psi_t)\right),
+   :label: noHWP
+
+where :math:`\theta` is the polarization angle of the detecotor, :math:`\psi_t` 
+is the orientation of the detector at the time :math:`t`, and :math:`\gamma`
+is the polarization efficiency.
+
+- Ideal HWP:
+
+.. math::
+   d_t = T + \gamma \left(Q\cos(4\alpha_t - 2\theta + 2\psi_t)+U\sin(4\alpha_t - 2\theta + 2\psi_t)\right),
+   :label: idealHWP
+
+where :math:`\alpha_t` is the HWP angle at the time :math:`t`.
+
+- Generic HWP:
+
+.. math::
+   d_t = (1,0,0,0) \times M_{\rm pol} \times R(\theta) \times R^T(\alpha_t) \times M_{\rm HWP} \times R(\alpha_t) \times R(\psi_t) \times \vec{S}
+   :label: genericHWP
+
+where 
+    * :math:`M_{\rm pol}` is mueller matrix of the polarimeter;
+    * :math:`M_{\rm HWP}` is mueller matrix of the HWP;
+    * :math:`R` is rotation matrix;
+    * :math:`\vec{S}` is the Stokes vector.
+
+You can fill with signal an existing TOD by using the
 function :func:`.scan_map_in_observations`, as the following example
 shows:
 
@@ -91,6 +123,7 @@ shows:
     -0.00601
     -0.00676
 
+The code automatically selects the fastest algebra based on the provided HWP.
 
 The input maps to scan can be either included in a dictionary with the name of
 the channel or the name of the dectector as keyword (the routines described in
@@ -103,13 +136,17 @@ coherent, so either a single Observation and a single numpy array, or same
 lenght list of Observations and numpy arrays.
 If the input map is ecliptic coordinates set `input_map_in_galactic` to `False`.
 The effect of a possible HWP is included in the pointing information, see
-:ref:`scanning-strategy`.
+:ref:`scanning-strategy`, the polarization angle of the detectors is taken from 
+the corresponding attributes included in the observations. The same applies to 
+the polarization efficiency. 
 
 The routine provides an on-the-fly interpolation of the input maps. This option
 is available through the argument `interpolation` which specifies the type of TOD
 interpolation ("" for no interpolation, "linear" for linear interpolation).
 Default: no interpolation.
 
+The low level function, :func:`.scan_map`, allows a more refined handling of the
+inputs. 
 
 Methods of the Simulation class
 -------------------------------

diff --git a/docs/source/mpi.rst b/docs/source/mpi.rst
@@ -133,6 +133,31 @@ variable :data:`.MPI_ENABLED`::
 To ensure that your code uses MPI in the proper way, you should always
 use :data:`.MPI_COMM_WORLD` instead of importing ``mpi4py`` directly.
 
+The simulation framework also provides a global object
+:data:`.MPI_COMM_GRID`. It has two attributes:
+
+- ``COMM_OBS_GRID``: This is an MPI communicator that contains all the
+  MPI processes with the global rank less than ``n_blocks_time * n_blocks_det``.
+  It provides a safety net to the operations and MPI communications
+  that are needed to be performed only on the partition of :data:`.MPI_COMM_WORLD`
+  that contain non-zero number of pointings and TODs. By default,
+  ``COMM_OBS_GRID`` points to the global MPI communicator :data:`.MPI_COMM_WORLD`.
+  It is updated once :class:`.Observation` are defined. For example,
+  consider the case when a user runs the simulation with 10 MPI
+  processes but due some specific ``det_blocks_attributes`` argument
+  in :class:`.Observation` class, the number of detector and time
+  blocks are determined to be 2 and 4 respectively. Then the
+  simulation framework will store the pointings and TODs only on
+  :math:`2\times4=8` MPI processes and the last two ranks of :data:`.MPI_COMM_WORLD`
+  will be left unused. Once this happens, ``COMM_OBS_GRID`` on first 8
+  ranks (rank 0 to 7) will point to the local sub-communicator
+  containing the processes with global rank 0 to 7. On the unused
+  ranks, it will simply point to the NULL communicator.
+- ``COMM_NULL``: If :data:`.MPI_ENABLED` is ``True``, this object
+  points to a NULL MPI communicator (``mpi4py.MPI.COMM_NULL``).
+  Otherwise it is set to ``None``. The user should compare
+  ``COMM_OBS_GRID`` with ``COMM_NULL`` on every MPI process in order
+  to avoid running a piece of code on unused MPI processes.
 
 Enabling/disabling MPI
 ----------------------