support for alosStack in ISCE #464

CunrenLiang · 2020-10-21T20:45:34Z

Description of proposed changes

This PR adds support for ALOS-2 (and ALOS-4 in the future) stack processed with contrib/stack/alosStack in ISCE-2 (isce-framework/isce2#175). The data loading part is similar to the product from topsStack and stripmapStack, whose sub-type is identified automatically based on the input metadata file via mintpy.load.metaFile.

Detailed changes are as follows:

utils/isce_utils.py:
- modify get_processor() to detect alosStack based on f1_*/*.frame.xml file.
- add the following new functions to extract metadata for alosStack:
  - extract_alosStack_metadata()
  - alos2_acquisition_mode()
  - extract_image_size_alosStack()
  - load_track()
  - read_alosStack_baseline()
prep_isce.py:
- adjust prepare_stack() and prepare_geometry() for alosStack.
- cmd_line_parse(): 1) use argparse required argument to replace the checking in cmd_line_parse(); 2) translate wildcard in input metaFile
add docs/examples/input_files/NCalAlos2Scan.txt as the example template file for loading alosStack products
objects/stackDict.geometry.write2hdf5(): convert waterBody to waterMask while loading data
utils/readfile.read_binary_file(): use band2 by default for *.cor file with 2 bands to support reading complex coherence files generated by isce-2.

Reminders

Pass Codacy code review (green)
Pass Circle CI test (green)
Make sure that your code follows our style. Use the other functions/files as a basis.
If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
If adding new functionality, add a detailed description to the documentation and/or an example.

welcome · 2020-10-21T20:45:35Z

💖 Thanks for opening this pull request! Please check out our contributing guidelines. 💖
Keep in mind that all new features should be documented. It helps to write the comments next to the code or below your functions describing all arguments, and return types before writing the code. This will help you think about your code design and usually results in better code.

example input file for ingesting data processed by alosStack in ISCE

mintpy/utils/isce_utils.py

yunjunz · 2020-10-22T03:37:22Z

This is awesome @CunrenLiang!

yunjunz · 2020-10-22T03:47:41Z

I have some very minor suggestions, it will be much easier for me to edit on the pull request directly, could you give me the permission @CunrenLiang?

Here is the instruction from GitHub for that: https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork

CunrenLiang · 2020-10-22T04:26:20Z

I have some very minor suggestions, it will be much easier for me to edit on the pull request directly, could you give me the permission @CunrenLiang?

Here is the instruction from GitHub for that: https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork

OK, you should have the permission to edit now.

CunrenLiang · 2020-10-22T04:28:53Z

Thanks for the detailed comments about the changes @yunjunz

yunjunz · 2020-10-22T05:10:51Z

Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- mintpy/objects/stackDict.py  2
- mintpy/utils/isce_utils.py  3
- mintpy/prep_isce.py  2
- mintpy/load_data.py  1

See the complete overview on Codacy

yunjunz

This looks great! Thank you @CunrenLiang.

welcome · 2020-10-22T05:24:50Z

🎉 🎉 🎉 Congrats on merging your first pull request! We here at behaviorbot are proud of you! 🎉 🎉 🎉

* More complex models for velocity estimation are added: - polynomial functions with customized polynomial degree - periodic function(s) with customized period(s) for seasonal fitting - step function(s) with customized date(s) for coseismic fitting These models are not callable using smallbaseline.py But they have been made available using timeseries2velocity.py with arguments Check timeseries2velocity.py -h for more information under "velocity models of interest" * generic STD estimation for time func params + stack.timeseries(): - rename get_design_matrix4average_velocity() to get_design_matrix4time_func() for more meaningful name, and inside sub-functions. - set model as dict type to support future time functions. - update usage of get_design_matrix4*() in objects.gps, simulation.simulation. + timeseris2velocity: - set model as dict type, consistent with objects.stack() - add standard error estimation for all time function parameters, based on error propagation law, as a generic extension to the linear velocity STD from equation (10) in Fattahi and Amelung (2015, JGR). This is currently support via non-bootstrap approach only. * add writefil.write_isce_xml/file() + add utils.writefile.write_isce_xml() to support xml/vrt file writing + add writefile.write_isce_file() to write binary data file and metadata files in isce format. + ts2vel: add Yuan-Kai to the author list * rename file for the num of triplets with non-zero int ambiguity from `numNonzeroIntClosure.h5` to `numTriNonzeroIntAmbiguity.h5`. * add docs/FAQs page * indexing readfile.read() call in view.py Support fancy indexing / slicing in utils.readfile.read() with x/ystep, to merge the reading and multilooking process into one. This significantly reduce the memory load of view.py for large 3D dataset, from 9 GB to less than 500 MB in the case of GalapagosSenDT128 dataset. * isce_utils.py: bug fix for spacecraftname in metadata (insarlab#430) * tropo_pyaps3: add --custom-height for testing purpose (insarlab#431) + view: auto update vlim after masking for multi-subplots for better display when the non-zero values are largely off from zero, which is the case of the absolute tropo delay in ERA5.h5 file. + tropo_pyaps3.py: add --custom-height for testing purpose + test_smallbaselineApp.py: take codacy suggestion + docs/README: one line for the badges + docs/dask.md: fix inproper display of dask performance figure on readthedocs * utils: add med_abs_dev() separate median_abs_deviation() from median_abs_deviation_threshold() to support 2D matrix calculation for MAD. * isce_utils: update calc for CENTER_LINE_UTC and HEADING + use sensingMid for CENTER_LINE_TUC calculation, instead of the previous startUTC. Difference is in seconds, with no expected impact on tropo_pyaps3.py with GAM resolution of hours. + calc HEADING from orbit instead of reading los file to be more generic, precise and simple + use meta instead of metadata to shorter code * save_hdfeos5.py and smallbaselineApp.py: add avgSpatialCoh.h5 to S1*.he5 file (insarlab#439) + smallbaselineApp: add avgSpatialCoh.h5 to the default geocode file list + save_hdfeos5: add `quality/avgSpatialCoh` - add `--asc / --avg-spatial-coh` for the average spatial coherence - rename `-t` to `--tc / --temp-coh` for the temporal coherence to avoid confusion with the newly added avgSpatialCoh.h5 + update docs/hdfeos5.md for the updated HDF-EOS5 data structure * bugfix for view.py --coastline option (insarlab#440) * bugfix for view.py --coastline option * bugfix in view.py --multilook-num (insarlab#441) * bugfix in view.py --multilook-num, which causes the initiation shape mismatch due to the recently introduced multilooking via indexing. * view: set coastline default to None instead of 'no' * Improve memory efficiency in the steps of calculating ramp, deramp, residual_RMS (insarlab#436) * To enhance the memory efficiency in calculating ramp, deramp, and time-series RMS, by reading/estimating(/writing) one date at a time. * Fixed a syntax error in readfile.py at line 799 Before: is not "none" Now: != "none" * writefile.py: add `print_msg=True` to `layout_hdf5()` and `write_hdf5_block()`. * isce_utils.py: fix typo burst to frame * timeseries: add precise UTC time info to `self.times` from CENTER_LINE_UTC * view: tie --coastline with --lalo-label If --coastline option is turned ON, enable --lalo-label as well, to show the lat/lon label; otherwise, it's blank by default by cartopy. * add writefile.layout_hdf5(ref_file) utils.writefile.layout_hdf5(): + support ref_file as an alternative of ds_name_dict and metadata, for easy use + support auxliary data writing, for less call of write_hdf5_block() + use snake_case to replace the camelCase for consistency Simplify the code using the new `writefile.layout_hdf5()` in the following functions: + utils.utils1.run_deramp() + ifgram_inversion.ifgram_inversion() + prep_fringe.prepare_timeseries() pyaps: force era5 correctio in the 1st pass * skip pixels with zero avgSpatialCoh/tempCoh to speedup + ifg_inv: - skip pixel with zero avgSpatialCoh, which is caused by data missing in the processing (usually if not all) - msg bugfix in calc_temp_coh() when the patch is very small - change default --cluster value to 'no', to be consistent with default template value. + unwrap_phase_phase_closure.calc_T_int(): mask out pixels with zero average spatial coherence, which is caused by missing data + dem_err: skip pixel with zero temporal coherence, due to water body, shadow, zero average spatial coherence, etc. + decorrelation: update msg + multilook: ignore runtime warnning msg * ifgramStack_coherence_spatialCoh.txt to coherenceSpatialAvg.txt + rename ifgramStack_coherence_spatialCoh.txt to coherenceSpatialAvg.txt for shorter filename + use camelCase for output figure name in plot_network.py + msg update for load_data.py when checking inconsistent file size * improved auto skip for inv_net and quick_overview + ifg_inv: run if TS is partly written only, due to the incomplete running of ifgram_inversion. + unwrap_error_phase_closure: add checking of modification time, and ref_y/x metadata for numTriNonzeroIntAmbiguity calculation * readfile.read_hdf5_file(): speedup x/ystep indexing + utils.readfile.read_hdf5_file(): speedup x/ystep indexing by using numpy indexing instead of h5py indexing. The latter has significant slow down for large index. For a time-series with 20*2400*5500, it's 1 sec vs 44 secs. + .gitignore: add *.dSYM* + view: - update ref_y/x for multilooking - support date1_date2 for TS file to be able to reconstruct interferogram from time-series on the fly * decor: use coh_step to replace epsilon decor: use coh_step to replace epsilon to avoid dividing a number close to zero * decor.cal_coh4phase_pdf_bias: numerically robust + use poly_deg of 10 for L < 10; and 40 for 10 <= L < 20, for more suitable redundancy + add poly_rcond for easy modification + limit output coh_cal to [0, 1] + attach coh_sim/est = 1 to the end of the bias data for a more robust calibration for values close to 1 * tropo_pyaps3: check date_list in existing tropo h5 file * new ASCII logo + version: add a new ASCII logo to replace the previous one + docs/README: add mintpy prounciation in International Phonetic Alphabet (IPA) style. + docs/resources/colormaps/README: update link to sci colormap png + delete the unused mintpy/.travis.yml + delete the impractical docs/examples/dev, which is meant for early stage develop only + .gitignore: add mintpy/objects/solid_f.*.so* * ifg_inv: bugfix for custom output filenames ifg_inv: bugfix for custom output filenames and update example usage for offset sbApp: print plot_netowrk cmd always * time function fitting: k! in the denominator of the polynomial function (insarlab#450) * time function fitting: k! in the denominator of the polynomial function * dem_err: rm duplicated design matrix func for time func + remove the duplicated design matrix func for time func and use the one from stack.timeseries objects instead, because it's more generic and more clear coding. + use G for the design matrix symbol instead of A, to be consistent with the convension in Yunjun et al. (2019) * bugfix on the prep_fringe.py for the updated isce_utils.extract_geometry_metadata() args (insarlab#455) There is a bug on line 140 of the prep_fringe.py causing the code to stop working during the extraction of the metadata on the reference xml files. * asc_desc2horz_vert: add --dset option add --dset option to be able to customize the dataset to read/use to asc/desc 2 horz/vert conversion, for input file with more than one dataset. * net.select_pairs_*(): use date_format instead of date12_format for simplicity * move resources from docs to mintpy + move resources folder from docs to mintpy as they are used in the code, to make mintpy folder independent from docs folder + rename docs/resources/colormaps/README.md to docs/api/colormaps.md and update its usage in mkdocs.yml + update the location of resources files in save_kmz_timeseries.py, objects/colors.py + remove unused test/unwrap_error_bridging in .gitignore * prep_aria: support subset / update mode / compression + writefile.layout_hdf(): add compression argument + smallbaselineApp: do not forcely pass --update to prep_aria.py + prep_aria: - add read_subset_box() to support loading part of ARIA products in 'load_data' step using `mintpy.subset.yx/lalo` options - add --compression option - more robust run_or_skip() considering the corrupted h5 files * prep_aria: bugfix when no template file input + bugfix when there is no template file input + update example usage * add pip to docs/conda.txt + add pip to conda.txt since it's used in the installation of pykml + remove ${CONDA_PREFIX} while calling conda since it's now removed in the custom environment from the latest version of conda distribution * prep_gamma: support *-* folder style while searching for *.par files to grab LAT/LONO_REF1/2/3/4 metadata * prep_aria: update metadata after subseting * view: break date1_date2 into date1\ndate2 for number subplots in (20, 50] for clearer display * ifg_inv: NaN as no-data value for offset/phase + ifgram_inversion; - use np.nan value as the no-data value for observations (phase or offset) to replace the previous version of using both (zero and nan), by: - setting masked out pixel values as np.nan instead of 0 and - setting zero phase (not offset) value to nan. - The above changes results in the following improvements: - simplified checking of masks and pixels with no-data values in some (not all) interferograms. - remove `skip_zero_value` from estimate_timeseries() due to the above change - skip pixels with zero avgSpatialSnr values for offset - more comments + objects/stack.ifgramStack.temporal_average(): - split the big 3D matrix in row direction instead of ifgram/time dimension. - the above change allows to use nanmean to replace manual mean calculation, to better handle the nan value, for improved avgSpatialSnr.h5 calculation. + adjust printout msg in writefile.layout_hdf5() and view.py * support for alosStack in ISCE (insarlab#464) Adds support for ALOS-2 (and ALOS-4 in the future) stack processed with contrib/stack/alosStack in ISCE-2. The data loading part is similar to the product from topsStack and stripmapStack, whose sub-type is identified automatically based on the input metadata file via mintpy.load.metaFile. Detailed changes are as follows: + utils/isce_utils.py: - modify get_processor() to detect alosStack based on f1_*/*.frame.xml file. - add the following new functions to extract metadata for alosStack: - extract_alosStack_metadata() - alos2_acquisition_mode() - extract_image_size_alosStack() - load_track() - read_alosStack_baseline() + prep_isce.py: - adjust prepare_stack() and prepare_geometry() for alosStack. - cmd_line_parse(): 1) use argparse required argument to replace the checking in cmd_line_parse(); 2) translate wildcard in input metaFile + add docs/examples/input_files/NCalAlos2Scan.txt as the example template file for loading alosStack products + objects/stackDict.geometry.write2hdf5(): convert waterBody to waterMask while loading data + utils/readfile.read_binary_file(): use band2 by default for *.cor file with 2 bands to support reading complex coherence files generated by isce-2. * dir structure of ISCE/alosStack (insarlab#465) * docs/dir_structure: directory structure of ISCE/alosStack added * docs/examples/input_files: Rename template file of ISCE/alosStack * ifg_inv for offset: consider A/RLOOKS for az/rg_pixel_size + ifgram_inversion: update az/rg_pixel_size to the resolution before multilooking for meaningful value + add utils.utils0.vtec2range_delay() to convert the zenith TEC to the predicted range delay in SAR, based on Chen and Zebker (2012) + add utils.isce_utils.get_IPF() for Sentinel-1 data + view: show lat/lon in the status bar for file in radar coord, if the geometryRadar.h5 file is in the default location. + tsview: make REF_Y/X optional to support off time-series + update HDF-EOS5 usage in save_hdfeos5.py and docs/hdfeos5.md + remove ECMWF from smallbaselineApp.cfg since it's not been updated anymore and is replaced by ERA-5; and update comments in tropo_pyaps(3).py * ifg_recon: bugfix + stack.ifgramStack.get_design_matrix4timeseries(): add refDate='no' to disable the column redunction due to temporal referencing. + ifgram_reconstruction.py: - fix the bug of design matrix due to the reference date - update example usage * fix bool array index bug for h5 dataset + readfile & stack: fix a bug of fancy indexing with numpy boolean array in h5py dataset (https://docs.h5py.org/en/stable/high/dataset.html#fancy-indexing), which is used in utils/readfile.py and objects/stack.py. This seems to be a bug in h5py side because it was working before. + prep_aria: fix a bug in gdal*.ReadAsArray() from numpy array int64 type to native python int16 type * bugfix for loading SNAP DEM with subset enabled + load_data: fix a bug of subsetting geometry file in geo-coord if input dataset is in geo-coord as well. + stackDict: acknowledge the -32768 no data for DEM * example dataset & template file for SNAP + add an example input dataset processed with SNAP provided by Andre Theron + add mintpy/data/input_files/WCapSenAT29.txt + add SNAP dataset to test_smallbaselineApp.py and test/configs/WCapSenAT29.txt + view: do not update disp_min/max if --wrap is enabled for multiple subplots, to be consistent with the one for single subplot. * add setup.py for pip install and pypi distribution + add pyproject.toml to use setuptools and wheel for build system + add MANIFEST.in + add setup.py with the following updates: - auto grabbed version, (long_)description - dependencies info - data files + move /sh to /mintpy/sh for easy registration with setup.py + move /docs/examples/input_files into /mintpy/data/input_files + move /mintpy/resources into /mintpy/data + version: add release_description to replace the previous description; and add new description. + bump python min version in doc/setup from 3.5 to 3.6 to be consistent with the docs/conda.txt * example dataset for ARIA on San Francisco + update mintpy/data/input_files/SanFranSenDT42.txt: enable unwrapping error correction with bridging and disable network modification via interferogram index. + docs/demo_dataset: add ARIA section with zenodo link using the San Francisco dataset prepared by Heresh Fattahi using ARIA-tools + view.prep_slice(): update example comments regarding the cartopy projection * aria support in load_data with autoPath + load_data: - move the call of prep_aria.main() from smallbaselineApp.py into load_data.py, so that prep_aria could appears to be the same as other prep_*.py in the user side, a.k.a. "load_data.py -t template file" works for ARIA as well. - support 'mintpy.load.autoPath' for aria - ignore failed prep_aria and continue load_data, becuase 1) the ARIA product could have been loaded into mintpy and deleted already and 2) the potential missing loaded HDF5 files will be reported laterward in smallbaselineApp.py, so it's okay to ignore it here. + utils1.update_template_file(): fix a bug when the option name contains the option value, i.e. mintpy.load.autoPath = auto. + add SanFranSenDT42.txt to the test_smallbaselinApp.py + mintpy/sh/plot_smallbaselineApp: plot ts_demErr.h5 * add setup.py for pip install and pypi distribution + add pyproject.toml to use setuptools and wheel for build system + add MANIFEST.in + add setup.py with the following updates: - auto grabbed version, (long_)description - dependencies info - data files + move /sh to /mintpy/sh for easy registration with setup.py + move /examples from /docs to /mintpy for easy management via pip/setup.py + version: add release_description to replace the previous description; and add new description. * more comments on extract_alosStack_metadata() + more comments on isce_utils.extract_alosStack_metadata() + update docs/api/attribute.md * run_isce_stack: support topsStack + add run_isce_stack.py to drive the isce-2 topsStack and stripmapStack processors. Compared with process_isce_stack_v1.py: - remove the job submission code since I can not test and use it - add topsStack - simplify the code - add dem prep based on dem(_gsi).py and auto opt values - replace cmd call of stackSentinel/StripMap.py with python call - add --text_cmd to sh file execution to avoid running load_tops/stripmap_stack + add fill_and_translate_template_auto_value() to replace mintpy.utils.check_template_auto_value() - support input template option value of auto, which will be translated into default value + add isce.numThread option for topsStack.topo.py + delete obsolete process_isce_stack*.py Co-authored-by: Yuankailiu <[email protected]> Co-authored-by: Zhang Yunjun <[email protected]> Co-authored-by: Sara Mirzaee <[email protected]> Co-authored-by: Zhang Yunjun <[email protected]> Co-authored-by: ranneylxr <[email protected]> Co-authored-by: Yuan-Kai Liu <[email protected]> Co-authored-by: Sara Mirzaee <[email protected]> Co-authored-by: Bryan Marfito <[email protected]> Co-authored-by: CunrenLiang <[email protected]>

support for alosStack in ISCE

f98e99e

example input file

4c6eeef

example input file for ingesting data processed by alosStack in ISCE

yunjunz self-requested a review October 21, 2020 20:52

yunjunz reviewed Oct 22, 2020

View reviewed changes

mintpy/utils/isce_utils.py Outdated Show resolved Hide resolved

update module_hierarchy

e3a8b12

yunjunz approved these changes Oct 22, 2020

View reviewed changes

yunjunz merged commit 5840680 into insarlab:main Oct 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support for alosStack in ISCE #464

support for alosStack in ISCE #464

CunrenLiang commented Oct 21, 2020 •

edited by yunjunz

Loading

welcome bot commented Oct 21, 2020

yunjunz commented Oct 22, 2020

yunjunz commented Oct 22, 2020

CunrenLiang commented Oct 22, 2020

CunrenLiang commented Oct 22, 2020

yunjunz commented Oct 22, 2020

yunjunz left a comment

welcome bot commented Oct 22, 2020

support for alosStack in ISCE #464

support for alosStack in ISCE #464

Conversation

CunrenLiang commented Oct 21, 2020 • edited by yunjunz Loading

welcome bot commented Oct 21, 2020

yunjunz commented Oct 22, 2020

yunjunz commented Oct 22, 2020

CunrenLiang commented Oct 22, 2020

CunrenLiang commented Oct 22, 2020

yunjunz commented Oct 22, 2020

yunjunz left a comment

Choose a reason for hiding this comment

welcome bot commented Oct 22, 2020

CunrenLiang commented Oct 21, 2020 •

edited by yunjunz

Loading