SATURATION flag for partially saturated pixels should be propagated to the _rate DQ

[jdavies-st]

This is a follow-on from the discussion in spacetelescope/stcal#125 and #7363

I think the CalWG needs to revisit the decision to not propagate SATURATION flags to the _rate DQ array, and I agree with Eddie's comments in that above issue that that information is definitely needed at the _rate level and afterwards.

It is really important in the _rate product to know if a pixel has been partially saturated at some point in a group. The most obvious reason is that saturated pixels leave persistence in subsequent integrations and exposures. For the NIR detectors, fully saturated pixels from a snowball or bright star can last ~2500 seconds or so in subsequent integrations and exposures. So in order to either model this or even mask it, we first need to know that the pixel was saturated. Currently that info is suppressed in the _rate file.

And this affects all NIR data thanks to snowballs, which always leave some partially saturated pixels in the image, which leave persistence in subsequent integrations, exposures and dithers, a persistence that masquerades as faint high-z galaxies, and contaminates source catalogs unless masked.

Saturated pixels also affect their neighbors, leaking charge into them. Again, if one wants to at a minimum mask this, one needs to know which pixels became saturated at some point in the ramp.

Neither of these cases above require one to look at the group individual groups. Working with the _rate files is fine for such masking.

Before the change in spacetelescope/stcal#125, where

• partial saturation of some groups produced a slope with SATURATED flag set in the dq array
• all groups saturated produced a NaN with SATURATED and DO_NOT_USE flags set in the dq array

was ideal, as it give the most amount of information in the _rate file that can be represented from the group data. We should go back to this.

I would also support a PARTIAL_SATURATION flag instead of SATURATED, as that gives the same needed info, but it seems currently that all DQ bits are currently in use.

As it is now, if I want to do exposure-to-exposure persistence flagging on the _rate files, I need to go back to the level 1 pipeline, specifically write out the jump step product, then use saturation flags in the last group there to figure out persistence in the _rate files. Not ideal, as I'm not actually needing to rerun level 1 pipeline to produce a different _rate product, I'm only doing so to get the information that should be propagated anyway (and was before the change in spacetelescope/stcal#125).

So can we (JWST users, CalWG) revisit this?

A proposal:

• SATURATED should always be set for a pixel if it was saturated at any point in the integration or exposure, full stop, regardless of how the ramps are fit.
• DO_NOT_USE being set will depend on how the ramp is fit. If your ramp fitting settings surpress using single group and all subsequent groups were saturated, then it will get set to DO_NOT_USE. If it uses the first group, then you will get a slope. So DO_NOT_USE is ramp fitting algorithm specific.
• If I want to find out which pixels are saturated in their first group, I have to look at the first group of each integration, and I should not assume anything in the _rate file can tell me this. If I care about this, I'm already looking at the GROUPDQ and do not need to rely on the _rate DQ.

Finally, the official documentation describes that bit as "Pixel saturated during exposure":

https://github.com/spacetelescope/stdatamodels/blob/44a5691d56e001f5f1a9c54167481fb73be14813/src/stdatamodels/jwst/datamodels/dqflags.py#L28C11-L28C71

https://jwst-pipeline.readthedocs.io/en/latest/jwst/references_general/references_general.html#data-quality-flags

Currently, that is misleading - most pixels that did saturate during an exposure are not flagged as such in the _rate DQ array.