fix gap detector to update status #427
Conversation
| let mut default_gap_detector = DefaultGapDetector::new(starting_version); | ||
| let mut parquet_gap_detector = ParquetFileGapDetector::new(starting_version); |
There was a problem hiding this comment.
Also I missed this in the Parquet processor PR, but why do we need both of them here? Each processor only needs one gap detector right?
There was a problem hiding this comment.
yeah we just need one, maybe I can add a trait in processor to check isParquet, then just create a parquet gap detector. is it okay to address along with parquet feedbacks from dport?
There was a problem hiding this comment.
You could have create_gap_detector_status_tracker_loop take in a GapDetector and the processor choose which one to use.
There was a problem hiding this comment.
Or just create 2 separate loops.
There was a problem hiding this comment.
yeah either one will be better creating both gap detectors. I think we can go with the former one, since we already have the enum
processor status wasn't being updated.
###Test Plan
tested locally