Fix assert_expected and EphysPostDLC task to find data in revisions

[juhuntenburg]

When running a task for which some of the input data is revised, the correct revision of a dataset is downloaded, but the assert_expected function then does not recognize that data if it is in a revision folder. Example, running a zadorlab task on angelakilab, i.e. input data needs to be downloaded:

In [16]: session_path=Path(/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-30/001/)

In [17]: task = EphysPostDLC(session_path, one=one, location='AWS', cameras=['left', 'right', 'body'])

In [18]: task.setUp()
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/_ibl_leftCamera.dlc.pqt: 100%|█████████████████████████████████████████████████████████████████████████| 44.0M/44.0M [00:01<00:00, 41.4MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/_ibl_rightCamera.dlc.pqt: 100%|████████████████████████████████████████████████████████████████████████| 95.4M/95.4M [00:01<00:00, 88.3MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/_ibl_bodyCamera.dlc.pqt: 100%|█████████████████████████████████████████████████████████████████████████| 2.66M/2.66M [00:00<00:00, 8.48MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/#2023-04-20#/_ibl_leftCamera.times.npy: 100%|██████████████████████████████████████████████████████████| 2.23M/2.23M [00:00<00:00, 6.88MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/#2023-04-20#/_ibl_rightCamera.times.npy: 100%|█████████████████████████████████████████████████████████| 5.56M/5.56M [00:00<00:00, 13.6MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/_ibl_bodyCamera.times.npy: 100%|███████████████████████████████████████████████████████████████████████| 1.12M/1.12M [00:00<00:00, 6.98MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/raw_video_data/_iblrig_leftCamera.raw.mp4: 100%|████████████████████████████████████████████████████████████| 3.44G/3.44G [00:34<00:00, 100MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/raw_video_data/_iblrig_rightCamera.raw.mp4: 100%|██████████████████████████████████████████████████████████| 3.35G/3.35G [00:33<00:00, 98.8MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/raw_video_data/_iblrig_bodyCamera.raw.mp4: 100%|█████████████████████████████████████████████████████████████| 473M/473M [00:05<00:00, 92.0MB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/leftROIMotionEnergy.position.npy: 100%|████████████████████████████████████████████████████████████████████| 160/160 [00:00<00:00, 1.74kB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/rightROIMotionEnergy.position.npy: 100%|███████████████████████████████████████████████████████████████████| 160/160 [00:00<00:00, 2.01kB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/bodyROIMotionEnergy.position.npy: 100%|████████████████████████████████████████████████████████████████████| 160/160 [00:00<00:00, 1.88kB/s]
/mnt/s0/Data/FlatIron/zadorlab/Subjects/CSH_ZAD_025/2020-07-26/001/alf/_ibl_trials.table.pqt: 100%|████████████████████████████████████████████████████████████████████████████| 52.1k/52.1k [00:00<00:00, 622kB/s]
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-18-040a33e3ae47> in <module>
----> 1 task.setUp()

~/Documents/PYTHON/envs/iblenv/lib/python3.8/site-packages/ibllib/pipes/tasks.py in setUp(self, **kwargs)
    327             self.data_handler.setUp()
    328             self.get_signatures(**kwargs)
--> 329             self.assert_expected_inputs()
    330             return True
    331 

~/Documents/PYTHON/envs/iblenv/lib/python3.8/site-packages/ibllib/pipes/tasks.py in assert_expected_inputs(self, raise_error)
    374 
    375         if not everything_is_fine and raise_error:
--> 376             raise FileNotFoundError('Missing inputs to run task')
    377 
    378         return everything_is_fine, files

FileNotFoundError: Missing inputs to run task

2024-03-26 09:46:46.727 INFO     [tasks.py:372] Checking input files
2024-03-26 09:46:46.730 ERROR    [tasks.py:388] Signature file expected ('_ibl_leftCamera.times.npy', 'alf', True) not found
2024-03-26 09:46:46.730 ERROR    [tasks.py:388] Signature file expected ('_ibl_rightCamera.times.npy', 'alf', True) not found

[juhuntenburg]

Additionally, the EphysPostDLC task itself loads the data in a way that it won't find the data in the revision folder

https://github.com/int-brain-lab/ibllib/blob/develop/ibllib/pipes/video_tasks.py#L509
https://github.com/int-brain-lab/ibllib/blob/develop/ibllib/pipes/video_tasks.py#L524

This needs to be fixed and we should check if this is the case for any other tasks

[mayofaulkner]

Fixed in ibllib release 2.34.0 #756