Deduplicate Already Processed Pairs by Searching S3 #179
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jhkennedy
reviewed
Nov 4, 2024
jhkennedy
approved these changes
Nov 7, 2024
There was a problem hiding this comment.
Okay this is looking good! I picked a reference scene processed yesterday:
LC09_L1GT_063110_20241105_20241105_02_T2
https://hyp3-its-live.asf.alaska.edu/jobs/77728a21-a200-4064-a9d9-b8fd22598096
And ran it through its-live-monitoring (output below). I'm seeing it successfully de-duplicate from S3!
$ python its_live_monitoring/src/main.py -v LC09_L1GT_063110_20241105_20241105_02_T2
2024-11-06 22:49:28,765 - DEBUG - its_live_monitoring/src/main.py -v LC09_L1GT_063110_20241105_20241105_02_T2
2024-11-06 22:49:28,881 - INFO - LC09_L1GT_063110_20241105_20241105_02_T2 qualifies for processing
2024-11-06 22:49:29,061 - DEBUG - LC08_L1GT_063110_20241028_20241104_02_T2 qualifies for processing
2024-11-06 22:49:29,061 - DEBUG - LC09_L1GT_063110_20241020_20241020_02_T2 qualifies for processing
2024-11-06 22:49:29,061 - DEBUG - LC08_L1GT_063110_20241012_20241021_02_T2 qualifies for processing
2024-11-06 22:49:29,062 - DEBUG - LC09_L1GT_063110_20241004_20241004_02_T2 qualifies for processing
2024-11-06 22:49:29,062 - DEBUG - LC08_L1GT_063110_20240926_20241005_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,062 - DEBUG - LC09_L1GT_063110_20240918_20240919_02_T2 qualifies for processing
2024-11-06 22:49:29,062 - DEBUG - LC08_L1GT_063110_20240910_20240914_02_T2 qualifies for processing
2024-11-06 22:49:29,063 - DEBUG - LC08_L1GT_063110_20240403_20240411_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,063 - DEBUG - LC09_L1GT_063110_20240326_20240326_02_T2 qualifies for processing
2024-11-06 22:49:29,063 - DEBUG - LC08_L1GT_063110_20240318_20240402_02_T2 qualifies for processing
2024-11-06 22:49:29,239 - DEBUG - LC09_L1GT_063110_20240310_20240310_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,239 - DEBUG - LC08_L1GT_063110_20240302_20240314_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,240 - DEBUG - LC09_L1GT_063110_20240223_20240223_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,240 - DEBUG - LC08_L1GT_063110_20240215_20240223_02_T2 qualifies for processing
2024-11-06 22:49:29,240 - DEBUG - LC09_L1GT_063110_20240207_20240207_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,240 - DEBUG - LC08_L1GT_063110_20240130_20240207_02_T2 qualifies for processing
2024-11-06 22:49:29,241 - DEBUG - LC09_L1GT_063110_20240122_20240122_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,241 - DEBUG - LC08_L1GT_063110_20240114_20240124_02_T2 qualifies for processing
2024-11-06 22:49:29,241 - DEBUG - LC09_L1GT_063110_20240106_20240106_02_T2 qualifies for processing
2024-11-06 22:49:29,241 - DEBUG - LC08_L1GT_063110_20231229_20240108_02_T2 qualifies for processing
2024-11-06 22:49:29,407 - DEBUG - LC09_L1GT_063110_20231221_20231223_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,408 - DEBUG - LC08_L1GT_063110_20231213_20240102_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,408 - DEBUG - LC09_L1GT_063110_20231205_20231205_02_T2 qualifies for processing
2024-11-06 22:49:29,408 - DEBUG - LC08_L1GT_063110_20231127_20231208_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,408 - DEBUG - LC09_L1GT_063110_20231119_20231119_02_T2 qualifies for processing
2024-11-06 22:49:29,409 - DEBUG - LC08_L1GT_063110_20231111_20231117_02_T2 qualifies for processing
2024-11-06 22:49:29,409 - DEBUG - LC09_L1GT_063110_20231103_20231103_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,409 - DEBUG - LC08_L1GT_063110_20231026_20231101_02_T2 qualifies for processing
2024-11-06 22:49:29,409 - DEBUG - LC09_L1GT_063110_20231018_20231018_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,409 - DEBUG - LC08_L1GT_063110_20231010_20231017_02_T2 qualifies for processing
2024-11-06 22:49:29,548 - DEBUG - LC09_L1GT_063110_20231002_20231002_02_T2 qualifies for processing
2024-11-06 22:49:29,548 - DEBUG - LC08_L1GT_063110_20230924_20231002_02_T2 disqualifies for processing because it has too much cloud cover
2024-11-06 22:49:29,548 - DEBUG - LC09_L1GT_063110_20230916_20230916_02_T2 qualifies for processing
2024-11-06 22:49:29,548 - DEBUG - LC08_L1GT_063110_20230908_20230912_02_T2 qualifies for processing
2024-11-06 22:49:29,659 - DEBUG - Found 21 secondary scenes for LC09_L1GT_063110_20241105_20241105_02_T2
2024-11-06 22:49:29,686 - INFO - Found 21 pairs for LC09_L1GT_063110_20241105_20241105_02_T2
2024-11-06 22:49:29,694 - DEBUG - reference secondary
1 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20241020_20241020_02_T2
3 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20241004_20241004_02_T2
4 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20240918_20240919_02_T2
6 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20240326_20240326_02_T2
11 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20240106_20240106_02_T2
13 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20231205_20231205_02_T2
14 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20231119_20231119_02_T2
18 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20231002_20231002_02_T2
19 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20230916_20230916_02_T2
0 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20241028_20241104_02_T2
2 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20241012_20241021_02_T2
5 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240910_20240914_02_T2
7 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240318_20240402_02_T2
8 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240215_20240223_02_T2
9 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240130_20240207_02_T2
10 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240114_20240124_02_T2
12 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20231229_20240108_02_T2
15 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20231111_20231117_02_T2
16 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20231026_20231101_02_T2
17 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20231010_20231017_02_T2
20 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20230908_20230912_02_T2
2024-11-06 22:49:30,146 - INFO - Deduplicated HyP3 running/pending pairs; 21 remaining
2024-11-06 22:49:30,151 - DEBUG - reference secondary
1 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20241020_20241020_02_T2
3 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20241004_20241004_02_T2
4 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20240918_20240919_02_T2
6 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20240326_20240326_02_T2
11 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20240106_20240106_02_T2
13 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20231205_20231205_02_T2
14 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20231119_20231119_02_T2
18 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20231002_20231002_02_T2
19 LC09_L1GT_063110_20241105_20241105_02_T2 LC09_L1GT_063110_20230916_20230916_02_T2
0 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20241028_20241104_02_T2
2 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20241012_20241021_02_T2
5 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240910_20240914_02_T2
7 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240318_20240402_02_T2
8 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240215_20240223_02_T2
9 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240130_20240207_02_T2
10 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20240114_20240124_02_T2
12 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20231229_20240108_02_T2
15 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20231111_20231117_02_T2
16 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20231026_20231101_02_T2
17 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20231010_20231017_02_T2
20 LC09_L1GT_063110_20241105_20241105_02_T2 LC08_L1GT_063110_20230908_20230912_02_T2
2024-11-06 22:49:33,930 - INFO - Deduplicated already published pairs; 0 remaining
2024-11-06 22:49:33,934 - DEBUG - Empty DataFrame
Columns: [reference, secondary]
Index: []
2024-11-06 22:49:33,936 - INFO - 0 HyP3 Jobs: 0 succeeded, 0 failed, 0 running, 0 pending.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.