Add some routines to get background subtracted images

[jeanconn]

Description

Add some convenience routines to get dark-current background subtracted ACA images.

Interface impacts

Testing

This depends on sot/mica#301 and sot/mica#311 .

Unit tests

• Mac

(ska3-flight-latest) flame:chandra_aca jean$ git rev-parse HEAD
6e20ccf80970ac9dad70a01e4008a5381d108d0c
(ska3-flight-latest) flame:chandra_aca jean$ python -c "import mica; print(mica.__version__)"
4.37.1.dev2+g8c56ac2
(ska3-flight-latest) flame:chandra_aca jean$ pytest
======================================================================= test session starts =======================================================================
platform darwin -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0
PyQt5 5.15.9 -- Qt runtime 5.15.8 -- Qt compiled 5.15.8
rootdir: /Users/jean/git
configfile: pytest.ini
plugins: astropy-0.11.0, qt-4.4.0, cov-5.0.0, timeout-2.2.0, remotedata-0.4.1, anyio-4.3.0, filter-subpackage-0.2.0, doctestplus-1.2.1, astropy-header-0.2.2, hypothesis-6.112.0, arraydiff-0.6.1, mock-3.14.0
collected 231 items                                                                                                                                               

chandra_aca/tests/test_aca_image.py ...................                                                                                                     [  8%]
chandra_aca/tests/test_all.py ........................                                                                                                      [ 18%]
chandra_aca/tests/test_attitude.py .............................................................                                                            [ 45%]
chandra_aca/tests/test_dark_model.py ............                                                                                                           [ 50%]
chandra_aca/tests/test_dark_subtract.py ........                                                                                                            [ 53%]
chandra_aca/tests/test_drift.py ..........................                                                                                                  [ 64%]
chandra_aca/tests/test_maude_decom.py .........................                                                                                             [ 75%]
chandra_aca/tests/test_planets.py ...............                                                                                                           [ 82%]
chandra_aca/tests/test_psf.py ...                                                                                                                           [ 83%]
chandra_aca/tests/test_residuals.py ss...                                                                                                                   [ 85%]
chandra_aca/tests/test_star_probs.py .................................                                                                                      [100%]

======================================================================== warnings summary =========================================================================
chandra_aca/chandra_aca/tests/test_residuals.py::test_obc
  /Users/jean/miniforge3/envs/ska3-flight-latest/lib/python3.11/site-packages/django/utils/encoding.py:266: DeprecationWarning: 'locale.getdefaultlocale' is deprecated and slated for removal in Python 3.15. Use setlocale(), getencoding() and getlocale() instead.
    encoding = locale.getdefaultlocale()[1] or 'ascii'

chandra_aca/chandra_aca/tests/test_residuals.py::test_obc
  /Users/jean/miniforge3/envs/ska3-flight-latest/lib/python3.11/site-packages/django/http/request.py:1: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
    import cgi

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================================== 229 passed, 2 skipped, 2 warnings in 65.07s (0:01:05

Independent check of unit tests by [REVIEWER NAME]

• [PLATFORM]:

Functional tests

No functional testing.

[jeanconn]

@javierggt and @taldcroft this could use feedback on organization and names of methods etc.

[jeanconn]

Actually it looks like I had some more changes in my local version, so I'll re-request review when those are up.

[taldcroft]

It's one of those days where I felt like spending too much time making some little code nicer. The hope is to take advantage of the power of CxoTime and u.Quantity, and solve a common problem of chunking a time interval.

def get_chunk_times(start: CxoTime, stop: CxoTime, dt_max: u.Quantity):
    """Get uniform time chunks for a given time range.

    Output times are spaced uniformly spaced by up to ``dt_max`` and cover the time
    range from ``start`` to ``stop``.

    Parameters
    ----------
    start : CxoTime
        Start time of the time range.
    stop : CxoTime
        Stop time of the time range.
    dt_max : u.Quantity (timelike)
        Maximum time interval for each chunk.

    Returns
    -------
    CxoTime
        CxoTime with time bins for each chunk.
    """
    # Calculate chunks to cover time range, handling edge case of start == stop
    n_chunk = max(np.ceil(float((stop - start) / dt_max)), 1)
    dt = (stop - start) / n_chunk
    times = start + np.arange(n_chunk + 1) * dt
    return times


@retry.retry(exceptions=requests.exceptions.RequestException, delay=5, tries=3)
@lru_cache(maxsize=2)
def get_maude_images(start: CxoTimeLike, stop: CxoTimeLike, fetch_limit=100 * u.hr):
    """
    Get ACA images from MAUDE for a given time range.

    Parameters
    ----------
    start : CxoTimeLike
        Start time of the time range.
    stop : CxoTimeLike
        Stop time of the time range.
    fetch_limit : u.Quantity (timelike)
        Maximum number of hours to fetch from MAUDE in one go.

    Returns
    -------
    dict
        dict with a astropy.table of ACA image data for each slot.
    """
    start = CxoTime(start)
    stop = CxoTime(stop)

    if stop - start > fetch_limit:
        raise ValueError("Time range too large for maude fetch, increase 'fetch_limit'")

    chunks = []
    # Break maude fetches into max 3 hour chunks required by maude_decom fetch
    times = get_chunk_times(start, stop, 3 * u.hr)
    for mstart, mstop in zip(times[:-1], times[1:]):
        imgs = maude_decom.get_aca_images(mstart, mstop)
        chunks.append(imgs)
    chunks = vstack(chunks)

    slot_data = {}
    for slot in range(8):
        slot_data[slot] = chunks[chunks["IMGNUM"] == slot]

    return slot_data

For example:

In [8]: get_chunk_times(CxoTime("2024:001"), CxoTime("2024:002"), 3.5 * u.hr)
Out[8]: 
<CxoTime object: scale='utc' format='date' value=['2024:001:00:00:00.000' '2024:001:03:25:42.857' '2024:001:06:51:25.714'
 '2024:001:10:17:08.571' '2024:001:13:42:51.429' '2024:001:17:08:34.286'
 '2024:001:20:34:17.143' '2024:002:00:00:00.000']>

[taldcroft]

@jeanconn - big picture comment about API. What about sticking with the maude_decom.get_aca_images standard of a single Table of images and related columns. Then include a helper function to convert that into a dict of Table keyed by slot. I think this might be more general instead of enforcing a new data structure (and one that I don't commonly use).

For existing code I have that grabs ACA images from MAUDE, I'd like an easy way to use this new code to do the dark subtraction on my existing table of images. So that might mean passing in a table of images and having the function return a new table which is matched to the input and contains relevant outputs, most importantly the dark-subtracted image.

[taldcroft]

It also occurs to me that we might want to upstream the 3-hr chunking and time limit here to maude_decom.

[jeanconn]

It looks like your concept of "get_chunk_times" is pretty general - where should that function live?

[jeanconn]

With regard to " What about sticking with the maude_decom.get_aca_images standard of a single Table of images and related columns." - sure - when I wanted the data by slot I figured a compromise between the maude_decom.get_aca_images standard (has one table but a column for slot) and the mica.archive.aca_l0.get_slot_data standard (underlying data is in files by slot so the fetch works by slot) made sense, but if you prefer to use a table with slot as a column that's fine.

[jeanconn]

Regarding "I'd like an easy way to use this new code to do the dark subtraction on my existing table of times." - is that a table of times or a table of aca images? I'm a little confused about which piece you want to plug in and where - do you want to point me to which code you had in mind?

[taldcroft]

I meant "table of images from maude_decom.get_aca_images".

The point of a more atomic API (accepting a single table of images) is that if your application is working with a dict of tables, it is trivial to do this in your app:

for slot, image_table in image_table_dict.items():
    images_dark_sub = ... # table of dark subtracted images for that slot
    # Do something with it

Whereas if I have a single table with multiple slots included then that can also work with the same functions. If the functions required a dict of table then that wouldn't work.

[taldcroft]

It looks like your concept of "get_chunk_times" is pretty general - where should that function live?

Hmm, your guess is as good as mine. ska_helpers.utils is there as a last resort to dump anything.

[jeanconn]

Right, but wouldn't it make just as much sense to replace the call to maude_decom.get_aca_images with a call to whatever we call this?

[jeanconn]

Regarding "get_chunk_times" I was thinking cxotime first though I suppose it isn't specific to cxotimes.

[taldcroft]

OK, this is very close! Basically all easy/uncontroversial things to fix.

[jeanconn]

Thanks for the Nth review of this one Tom!