Frame ID and TopsApp Processing #93
Labels
bug
Something isn't working
Comments
|
My only other guess is to remove the integer truncation we do here. The DEM might control the |
|
Another option is to decrease the frame size per @mgovorcin's suggestion. |
I second this idea. I remember topo step taking too long on my tests with virtual processing as well. I would stay make the yellow frame smaller enough so it doesn't touch the overlapping areas. |
|
It appears all the experts seem to be converging on decreasing the frame size. Thank you, @ehavazli. Here are some relevant insights from @piyushrpt:
|
|
We should do some rough counting to ensure frame size is suitable. ESA Frames (images) come in packages of 9 bursts (I am grouping bursts across all 3 swaths as is done in the relative burst map). If our fixed frames are 10 bursts in size, we will get 3 images when the fixed frame covers all of an intersecting ESA image frame because an ESA image frame will have at least 1 burst overlap between its neighbors*. I would imagine similar edge cases for 9 burst frames because of issues identified below. I would say 8 burst size would prevent more than 2 SLCs from being identified at the outset. *There are additional issues by our grouping of bursts:
|
|
Also, there are multiple regions of interest if I remember. One specified in the sensor - crops things right up front and only processes the bursts that intersect the ROI. Read up the docs on this. All of this is well documented in the UNAVCO classes. I haven't used this in more than 2 years. |
|
Thanks, Piyush! I use this for reference frequently: https://github.com/isce-framework/isce2-docs/blob/master/Notebooks/UNAVCO_2020/TOPS/topsApp.ipynb |
|
I would also check |
|
Im still a bit puzzled on the long runtime. From what i seen on the ISCE log file that @cmarshak showed me. The region of interest seems to work. We are extracting like 10-13 burst per subswath which is pretty much what we expect knowing we increased the overlap by 2 burst so about 4 extra than a regular SLC. Perhaps the resources were capped, or multiple jobs were in competition? What was the number of threads/ memory availability at the time of the test? What should not matter:
|
|
Check OMP_NUM_THREADS - the number of threads used should also be in the screen log |
|
Thank you again, Piyush, for all your help and insight to this. @dbekaert - I proceeded to make the frames 8 bursts with 1 overlap and didn't really look back. As you noted, there are a number of issues that I may failed to include in the bug report:
That said, I had 3 SLCs within my region of interest with 13 bursts (as I recall). Should not have been significantly more than the roughly 10-11 burst products we have seen in GUNWs. The isce.log is an excellent place as David mentioned to confirm |
|
I am going to close this issue because we are using 8 frames to ensure at most 2 SLCs in our interferogram. It might be wise to investigate this further should this come back up, but this will serve as a helpful reference for such issues. |
|
@cmarshak Would still be instructive for your fame-code to know if there are any runtime issues. Once you have settled on the frames could you run it without competing resources, ensuring we set the number of treads correctly available etc. As you mentioned i would not expect much more than 10-15% increase in processing time as its only slightly larger. We should do a back of the envelope calculation to ensure the unwrapping with snaphu has enough memory available (It was one of the reasons to drop to 90m and frame sized product at the time) |
|
So the topo issue was resolved in the latest runs. Again, I may have opened the issue ticket prematurely. I think there is a lot of valuable information in the exercise. Can continue the frame discussion here (included some of the info and more there): #94 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
@dbekaert @sssangha @ehavazli @mgovorcin @gracebato - please let me know your thoughts.
The frame processing is taking an unreasonable amount of time particularly for
topostep (see these topsapp steps). I want to make sure that I am doing things correctly with ISCE2 (or if this frame approach is possible as we currently have scoped it out 🙃).I am specifying a region of interest that should be significantly smaller than the 3 slcs fed in. But my guess is that topo is still being performed for the 3 SLCs in the reference/secondary, which defeats the purpose of specifying the region of interest (see the image at the end of this issue ticket).
Here is the time for each of the above steps on our server leffe:
TopsApp Steps: 0%| | 0/24 [00:00<?, ?it/s]
TopsApp Steps: 4%|▍ | 1/24 [00:00<00:11, 2.00it/s]
TopsApp Steps: 8%|▊ | 2/24 [01:45<22:38, 61.77s/it]
TopsApp Steps: 12%|█▎ | 3/24 [01:48<12:15, 35.03s/it]
TopsApp Steps: 17%|█▋ | 4/24 [01:49<07:09, 21.49s/it]
TopsApp Steps: 21%|██ | 5/24 [16:43:58<114:30:38, 21696.76s/it]
TopsApp Steps: 25%|██▌ | 6/24 [16:44:02<71:36:20, 14321.11s/it]
TopsApp Steps: 29%|██▉ | 7/24 [16:44:02<45:31:10, 9639.42s/it]
TopsApp Steps: 33%|███▎ | 8/24 [16:44:03<29:12:14, 6570.90s/it]
TopsApp Steps: 38%|███▊ | 9/24 [16:44:03<18:49:14, 4516.94s/it]
TopsApp Steps: 42%|████▏ | 10/24 [16:44:04<12:08:37, 3122.67s/it]
TopsApp Steps: 46%|████▌ | 11/24 [16:44:05<7:49:32, 2167.14s/it]
TopsApp Steps: 50%|█████ | 12/24 [16:44:05<5:01:36, 1508.07s/it]
TopsApp Steps: 54%|█████▍ | 13/24 [17:20:23<5:13:41, 1711.06s/it]
TopsApp Steps: 58%|█████▊ | 14/24 [17:38:27<4:13:36, 1521.60s/it]
TopsApp Steps: 62%|██████▎ | 15/24 [17:38:30<2:39:34, 1063.82s/it]
TopsApp Steps: 67%|██████▋ | 16/24 [17:57:10<2:24:06, 1080.78s/it]
TopsApp Steps: 71%|███████ | 17/24 [17:59:14<1:32:31, 793.09s/it]
TopsApp Steps: 75%|███████▌ | 18/24 [17:59:35<56:05, 560.92s/it]
TopsApp Steps: 79%|███████▉ | 19/24 [18:08:23<45:55, 551.15s/it]
TopsApp Steps: 83%|████████▎ | 20/24 [18:08:24<25:43, 385.86s/it]
TopsApp Steps: 88%|████████▊ | 21/24 [18:23:38<27:12, 544.33s/it]
TopsApp Steps: 92%|█████████▏| 22/24 [18:23:38<12:42, 381.19s/it]
TopsApp Steps: 96%|█████████▌| 23/24 [18:23:39<04:27, 267.07s/it]
TopsApp Steps: 100%|██████████| 24/24 [18:23:40<00:00, 187.14s/it]
TopsApp Steps: 100%|██████████| 24/24 [18:23:40<00:00, 2759.19s/it]
I am running two jobs like this simultaneously on leffe (and both have the result of taking too long on
topo).To Reproduce
See these examples: https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp#using-fixed-frames-experimental
Here is the xml:
Below is the image of the SLCs (blue) and the frame in yellow.
The text was updated successfully, but these errors were encountered: