new alosStack, PRF change of ALOS-1, update of TOPS ionospheric correction, new ionospheric filter #175

CunrenLiang · 2020-08-04T00:41:03Z

new alosStack application
handling PRF change for ALOS-1 raw data
adding options for discarding burst properties in TOPS ionospheric phase estimation
updated ionospheric phase filtering for ALOS-2
a few other minor changes including fixed a bug in ionospheric computation in topsApp.py, fixed problems in getting range sampling frequency, range bandwidth, chirp slope and pulse duration for TerraSAR-X, fixed bug in computing cost flow in snaphu, and others.

hfattahi

Thank you @CunrenLiang and sorry for very late review. It Looks good to me. There is one function that I don't see any where used. It will be great to remove it if it's not needed.

hfattahi · 2020-10-04T13:50:43Z

components/isceobj/Alos2Proc/runIonFilt.py

+    return f2d/np.sum(f2d)
+
+
+def adaptive_gaussian_v0(ionos, wgt, size_max, size_min):


Is this function used anywhere?

No worries about the late review.
This is an older version of the filter. I am not sure if I will be using it in the future, so I didn't remove it from the file.

ok no worries

hfattahi · 2020-10-24T16:39:13Z

@yunjunz and @piyushrpt would you please take a look at this PR. A lot of great contributions from @CunrenLiang are all squeezed in this PR. Of course it would have been nice to see smaller PRs. Maybe for future PRs.
@CunrenLiang you may need to rebase with main to fix the build fails.

1. updated filtering of ionospheric phase 2. updated mosaicking of subswath subband interferograms.

fixed bugs in getting range bandwidth, chirp slope, pulse duration and range sampling frequency

CunrenLiang · 2020-11-08T01:32:36Z

@yunjunz and @piyushrpt would you please take a look at this PR. A lot of great contributions from @CunrenLiang are all squeezed in this PR. Of course it would have been nice to see smaller PRs. Maybe for future PRs.
@CunrenLiang you may need to rebase with main to fix the build fails.

It would be great if this pull request can get merged early.

I did the rebase. Still one check failed. I think it might be related to docker? I am new to docker. Here is the output

#!/bin/sh -eo pipefail
cd images
source env.sh
docker load -i isce2.tar
docker tag isce/isce2:$TAG isce/isce2:latest
docker login -u $DOCKER_USER -p $DOCKER_PASS
docker push isce/isce2:$TAG
docker push isce/isce2:latest


e09ab3c0: Loading layer  211.1MB/211.1MB
de09d0c7: Loading layer  2.048kB/2.048kB
65112c6b: Loading layer  2.773GB/2.773GB
e56654fd: Loading layer  2.048kB/2.048kB
e1d62512: Loading layer   2.56kB/2.56kB
a0927620: Loading layer  68.24MB/68.24MB
fd0f5a83: Loading layer  316.3MB/316.3MB
324abdc2: Loading layer  9.728kB/9.728kB
efa43047: Loading layer  25.55MB/25.55MB
Loaded image: isce/isce2:b341c66/1.23GB
Password: EOF

Exited with code exit status 1

CircleCI received exit code 1

rtburns-jpl

The new features and fixes look great, but I want to echo @hfattahi's comment about smaller PRs - it's essentially unfeasible for us to review +13k line diffs. We're able to review and merge more granular PRs with limited scope much more quickly.

components/isceobj/Alos2Proc/runIonCorrect.py

components/isceobj/Alos2Proc/runIonFilt.py

components/isceobj/Alos2Proc/runIonSubband.py

components/isceobj/Sensor/ALOS.py

minor update

remove author info

delete functions not used

CunrenLiang · 2020-11-23T05:08:17Z

@yunjunz and @piyushrpt would you please take a look at this PR. A lot of great contributions from @CunrenLiang are all squeezed in this PR. Of course it would have been nice to see smaller PRs. Maybe for future PRs.
@CunrenLiang you may need to rebase with main to fix the build fails.

It would be great if this pull request can get merged early.

I did the rebase. Still one check failed. I think it might be related to docker? I am new to docker. Here is the output
#!/bin/sh -eo pipefail
cd images
source env.sh
docker load -i isce2.tar
docker tag isce/isce2:$TAG isce/isce2:latest
docker login -u $DOCKER_USER -p $DOCKER_PASS
docker push isce/isce2:$TAG
docker push isce/isce2:latest


e09ab3c0: Loading layer  211.1MB/211.1MB
de09d0c7: Loading layer  2.048kB/2.048kB
65112c6b: Loading layer  2.773GB/2.773GB
e56654fd: Loading layer  2.048kB/2.048kB
e1d62512: Loading layer   2.56kB/2.56kB
a0927620: Loading layer  68.24MB/68.24MB
fd0f5a83: Loading layer  316.3MB/316.3MB
324abdc2: Loading layer  9.728kB/9.728kB
efa43047: Loading layer  25.55MB/25.55MB
Loaded image: isce/isce2:b341c66/1.23GB
Password: EOF

Exited with code exit status 1

CircleCI received exit code 1

@rtburns-jpl @piyushrpt , I think this docker deploy failure is related to docker credentials? If so, I am just leaving it here until the credentials are updated.

rtburns-jpl · 2020-11-23T19:57:01Z

Could you try merging main into your branch to see if that fixes the deploy issue?

use if false instead of commenting out codes

remove update info from code.

for resolving conflicts with recent commit

CunrenLiang · 2020-11-24T05:58:25Z

Could you try merging main into your branch to see if that fixes the deploy issue?

Still failed at password step @rtburns-jpl .

rtburns-jpl · 2020-11-24T06:48:08Z

ping @piyushrpt - didn't we run into something similar a while ago? I don't recall what we did to solve it :/

efa43047: Loading layer  25.55MB/25.55MB
Loaded image: isce/isce2:b341c66/1.23GB
Password: EOF

Exited with code exit status 1

CircleCI received exit code 1

add comments about identifying bad pairs

correct variable name

lijun99 · 2021-01-09T00:01:15Z

I think the problem might be with "docker login -u $DOCKER_USER -p $DOCKER_PASS". These two envs need to set somewhere. For failsafe, it's better to use -n check before calling it.

Anyway, I advised @CunrenLiang to rename his branch from main; that might be the reason to force a deploy check on his PR.

ping @piyushrpt - didn't we run into something similar a while ago? I don't recall what we did to solve it :/
efa43047: Loading layer  25.55MB/25.55MB
Loaded image: isce/isce2:b341c66/1.23GB
Password: EOF

Exited with code exit status 1

CircleCI received exit code 1

rtburns-jpl · 2021-02-25T20:17:46Z

Hey, sorry about all the CI issues. I think they're all resolved now, so if you merge main, you should get all green checkmarks.

CunrenLiang · 2021-02-26T04:02:33Z

Hey, sorry about all the CI issues. I think they're all resolved now, so if you merge main, you should get all green checkmarks.

Thank you for the update @rtburns-jpl . Looks like the same problem in deploy is still there. The error message is the same.

rtburns-jpl · 2021-02-26T05:28:44Z

Ugh, yep. Well, at least the build job is working now. Anyway, I'll try to look this over to the best of my ability next week so we can merge this soon.

CunrenLiang · 2021-02-26T05:37:50Z

I think Lijun's suggestions might be promising. I tried to change my branch name from 'main' to something else, but it looks like it's not easy to do it without deleting my current branch and re-fork. So perhaps we can try to change the circleci configure file.

rtburns-jpl · 2021-02-26T06:00:26Z

Ah yeah, maybe the deploy job is just checking if the branch name is 'main' without also checking that it's not running on a fork. Honestly, don't worry about it. The build/test jobs were what I was concerned about and those are all fine now.

CunrenLiang · 2021-02-26T07:41:30Z

Ah yeah, maybe the deploy job is just checking if the branch name is 'main' without also checking that it's not running on a fork. Honestly, don't worry about it. The build/test jobs were what I was concerned about and those are all fine now.

Yes, Lijun just explained it with more details. It should be OK to just merge it. Then I can rename my fork to other names instead of 'main', so that it won't deploy again for my future PRs.

rtburns-jpl

Between this and PyCuAmpcor, we have a good number of new features in isce2, so I'll plan to cut a v2.5 release after merging.

CunrenLiang · 2021-03-04T00:50:07Z

It may be better to include this pull request in v2.5, as many people are waiting to use the new features in this pull request. Right now they have to use the code in my repository.

Between this and PyCuAmpcor, we have a good number of new features in isce2, so I'll plan to cut a v2.5 release after merging.

piyushrpt requested review from dbekaert and hfattahi September 13, 2020 00:06

hfattahi approved these changes Oct 4, 2020

View reviewed changes

hfattahi requested a review from vbrancat October 4, 2020 14:02

hfattahi approved these changes Oct 9, 2020

View reviewed changes

CunrenLiang changed the title ~~updated ionospheric phase filtering for ALOS-2~~ new alosStack, PRF change of ALOS-1, update of TOPS ionospheric correction, new ionospheric filter Oct 20, 2020

yunjunz mentioned this pull request Oct 22, 2020

support for alosStack in ISCE insarlab/MintPy#464

Merged

5 tasks

hfattahi requested review from piyushrpt, rtburns-jpl and yunjunz October 22, 2020 04:43

yunjunz mentioned this pull request Oct 24, 2020

dir structure of ISCE/alosStack insarlab/MintPy#465

Merged

CunrenLiang and others added 10 commits November 7, 2020 16:06

updated filtering of ALOS-2 ionospheric phase

0eda642

1. updated filtering of ionospheric phase 2. updated mosaicking of subswath subband interferograms.

minor bug fix

0fac7ff

fix TerraSAR-X parameter bugs

3aa234c

fixed bugs in getting range bandwidth, chirp slope, pulse duration and range sampling frequency

new alosStack application

e9bd7ed

handling PRF change for ALOS-1 raw data

9837bf3

option for discarding burst properties in TOPS ionosphere estimation

a7b0287

put temporary files in a better place

5e4d160

add executable permission

6cecbe9

add new files to cmakelists

6b0a0ac

Add files via upload

b341c66

use for loops for dates/pairs processing

78a5dde

rtburns-jpl requested changes Nov 9, 2020

View reviewed changes

CunrenLiang and others added 4 commits November 15, 2020 01:00

Update ion_filt.py

4e47909

minor update

Update ion_check.py

5634f5f

Update runIonCorrect.py

2637bb6

remove author info

Update runIonFilt.py

fd28d2e

delete functions not used

CunrenLiang closed this Nov 23, 2020

CunrenLiang reopened this Nov 23, 2020

CunrenLiang added 6 commits November 23, 2020 20:13

Update runIonSubband.py

54626df

use if false instead of commenting out codes

Merge branch 'main' of https://github.com/CunrenLiang/isce2 into main

59b2738

Update ALOS.py

b9894e4

remove update info from code.

remove comments about update from codes

6a43886

Update SConscript

e51f25a

for resolving conflicts with recent commit

Merge remote-tracking branch 'upstream/main' into main

5a8bf76

CunrenLiang added 2 commits November 26, 2020 23:40

Update create_cmds.py

ab8b0f4

add comments about identifying bad pairs

Update ion_subband.py

fb96c54

correct variable name

CunrenLiang added 2 commits February 3, 2021 13:24

Merge remote-tracking branch 'upstream/main' into main

7b0856f

Merge remote-tracking branch 'upstream/main' into main

1d44e16

Merge remote-tracking branch 'upstream/main' into main

89973d7

CunrenLiang closed this Feb 26, 2021

CunrenLiang reopened this Feb 26, 2021

rtburns-jpl approved these changes Mar 3, 2021

View reviewed changes

CunrenLiang closed this Mar 4, 2021

CunrenLiang reopened this Mar 4, 2021

rtburns-jpl merged commit d0ec8c1 into isce-framework:main Mar 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new alosStack, PRF change of ALOS-1, update of TOPS ionospheric correction, new ionospheric filter #175

new alosStack, PRF change of ALOS-1, update of TOPS ionospheric correction, new ionospheric filter #175

CunrenLiang commented Aug 4, 2020 •

edited

Loading

hfattahi left a comment

hfattahi Oct 4, 2020

CunrenLiang Oct 9, 2020

hfattahi Oct 9, 2020

hfattahi commented Oct 24, 2020

CunrenLiang commented Nov 8, 2020

rtburns-jpl left a comment

CunrenLiang commented Nov 23, 2020 •

edited

Loading

rtburns-jpl commented Nov 23, 2020

CunrenLiang commented Nov 24, 2020

rtburns-jpl commented Nov 24, 2020

lijun99 commented Jan 9, 2021

rtburns-jpl commented Feb 25, 2021

CunrenLiang commented Feb 26, 2021 •

edited

Loading

rtburns-jpl commented Feb 26, 2021

CunrenLiang commented Feb 26, 2021 •

edited

Loading

rtburns-jpl commented Feb 26, 2021

CunrenLiang commented Feb 26, 2021

rtburns-jpl left a comment

CunrenLiang commented Mar 4, 2021 •

edited

Loading

		return f2d/np.sum(f2d)


		def adaptive_gaussian_v0(ionos, wgt, size_max, size_min):

new alosStack, PRF change of ALOS-1, update of TOPS ionospheric correction, new ionospheric filter #175

new alosStack, PRF change of ALOS-1, update of TOPS ionospheric correction, new ionospheric filter #175

Conversation

CunrenLiang commented Aug 4, 2020 • edited Loading

hfattahi left a comment

Choose a reason for hiding this comment

hfattahi Oct 4, 2020

Choose a reason for hiding this comment

CunrenLiang Oct 9, 2020

Choose a reason for hiding this comment

hfattahi Oct 9, 2020

Choose a reason for hiding this comment

hfattahi commented Oct 24, 2020

CunrenLiang commented Nov 8, 2020

rtburns-jpl left a comment

Choose a reason for hiding this comment

CunrenLiang commented Nov 23, 2020 • edited Loading

rtburns-jpl commented Nov 23, 2020

CunrenLiang commented Nov 24, 2020

rtburns-jpl commented Nov 24, 2020

lijun99 commented Jan 9, 2021

rtburns-jpl commented Feb 25, 2021

CunrenLiang commented Feb 26, 2021 • edited Loading

rtburns-jpl commented Feb 26, 2021

CunrenLiang commented Feb 26, 2021 • edited Loading

rtburns-jpl commented Feb 26, 2021

CunrenLiang commented Feb 26, 2021

rtburns-jpl left a comment

Choose a reason for hiding this comment

CunrenLiang commented Mar 4, 2021 • edited Loading

CunrenLiang commented Aug 4, 2020 •

edited

Loading

CunrenLiang commented Nov 23, 2020 •

edited

Loading

CunrenLiang commented Feb 26, 2021 •

edited

Loading

CunrenLiang commented Feb 26, 2021 •

edited

Loading

CunrenLiang commented Mar 4, 2021 •

edited

Loading