Wrap x2sys_init and x2sys_cross #546

weiji14 · 2020-07-23T09:56:21Z

Description of proposed changes

Wrapping the x2sys_cross function (which also requires x2sys_init), implemented under x2sys.py. X2SYS is a tool for analyzing intersecting track data (e.g. ships measuring ocean bathymetry, satellite altimeters measuring land elevation)!

Live documentation preview for this Pull Request is at https://pygmt-git-x2syscross.gmt.vercel.app/api/generated/pygmt.x2sys_cross.html.

Example code:

import pygmt

fname: str = pygmt.which("@tut_ship.xyz", download="a")
pygmt.x2sys_init(tag="GMT", fmtfile="xyz", force=True)
df: pd.DataFrame = x2sys_cross(tracks=[fname], tag=tag, coe="i")

assert df.shape == (14294, 12)

This is useful for performing crossover analysis (e.g. finding the elevation error between intersecting tracks). It will be tremendously useful for some of my ICESat-2 satellite laser altimetry work, and having used it in GMT 5 back in 2015 for ICESat-1, I think it's time to bring it to the Python world!

TODO:

Fixes #545

Reminders

Run make format and make check to make sure the code follows the style guide.
Add tests for new features or tests that would have caught the bug that you're fixing.
Add new public functions/methods/classes to doc/api/index.rst.
Write detailed docstrings for all functions/methods.
If adding new functionality, add an example to docstrings or tutorials.

Wrapping the x2sys_cross function (and x2sys_init), implemented under x2sys.py. Original GMT `x2sys_cross` documentation can be found at https://docs.generic-mapping-tools.org/6.1/supplements/x2sys/x2sys_cross.html. Sample test cases stored under test_x2sys_cross.py, and they uses the load_sample_bathymetry dataset. Aliased fmtfile (D) and force (F) for x2sys_init. Aliased tag (T) and coe (Q) for x2sys_cross.

pygmt/tests/test_x2sys_cross.py

weiji14 · 2020-07-23T10:53:23Z

pygmt/x2sys.py

+        elif kind == "matrix":
+            file_context = lib.virtualfile_from_matrix(track.values)


Making a virtualfile from a pandas.DataFrame and passing that into x2sys_cross doesn't work yet. It might just be because I'm using ASCII with x2sys_init, not sure if BINARY would work (see https://docs.generic-mapping-tools.org/6.1/supplements/x2sys/x2sys_init.html#format-definition-files). Error message is as follows:

x2sys_cross [WARNING]: No time column, use dummy times Not a valid unit: c [meter assumed] Not a valid unit: c [meter assumed] x2sys_cross [ERROR]: x2sys_read_file : Cannot find track @GMTAPI@-S-I-D-M-T-N-000000 (null): Error from fclose [@GMTAPI@-S-I-D-M-T-N-000000]

As I understand how GMT works, I think x2sys related modules don't support virtual files. Perhaps you could open an issue in the GMT repository.

Not surprised actually, I'll open an issue! Edit: Done at GenericMappingTools/gmt#3717

Support only filename inputs into x2sys_cross for now, because passing in matrix type virtualfiles into GMT C API doesn't work properly yet.

Aliased suffix (E), discontinuity (G), spacing (I), units (I), region (R), gap (W), distcalc (j) for x2sys_init. Also checked that passing a list into region (R), spacing (I), units (N), and gap (W) works properly.

Aliased combitable (A), runtimes(C), interpolation (I), region (R), speed(S), numpoints (W) for x2sys_cross. Also checked that setting the region (R), interpolation (I) and numpoints (W) args work properly.

weiji14 · 2020-07-26T01:01:03Z

pygmt/x2sys.py

+    A="combitable",
+    C="runtimes",
+    # D="",
+    I="interpolation",
+    R="region",
+    S="speed",
+    T="tag",
+    Q="coe",
+    V="verbose",
+    W="numpoints",
+    # Z="",


Ok, almost done aliasing all the arguments for x2sys_cross. Any good names for D or Z?

My bad suggestions:

D (True/False, N or S) = override/overridepole

Z (True/False) = trackvalues/rawreport/rawvalue/xvalue

Edit: Going with override (D) and trackvalues (Z)!

On Linux, z_X.mean() = -139.196212, On macOS/Windows, z_X.mean() = -139.20221.

seisman · 2020-07-27T05:19:39Z

Maybe better to open separate PRs for other unrelated changes, e.g., pygmt/helpers/decorators.py and .pylintrc?

weiji14 · 2020-07-27T10:43:32Z

Maybe better to open separate PRs for other unrelated changes, e.g., pygmt/helpers/decorators.py and .pylintrc?

Ok, opened #550 for verbose (V) and #551 for distcalc (j). I've renamed df to table/dataframe so the .pylintrc change isn't needed anymore.

seisman

I never used these modules, but the code quality looks good to me.

weiji14 · 2020-07-27T22:55:51Z

Cool, thanks @seisman! It is quite a tricky library to setup, I only got back to X2SYS a few days ago because me and my friend @arrran were getting into some crossover analysis work on satellite altimetry datasets. I'll try and get the pandas.DataFrame input and tutorial for this once I have a play with it a little bit more (probably next month).

seisman · 2020-07-28T05:09:32Z

pygmt/tests/test_x2sys_cross.py

+        for i in range(2):
+            np.random.seed(seed=i)
+            with open(os.path.join(os.getcwd(), f"track_{i}.xyz"), mode="w") as fname:
+                np.savetxt(fname=fname, X=np.random.rand(10, 3))
+
+        output = x2sys_cross(
+            tracks=["track_0.xyz", "track_1.xyz"], tag=tag, coe="e", verbose="i"
+        )


@weiji14

This test produces two files track_0.xyz and track_1.xyz in the tmp-test-dir-with-unique-name directory.

I'm not fully understand what the test does. It should either output to a temporary directory, or delete these two files at the end of the test.

Move the x2sys modules originally wrapped in #546. The single x2sys.py has been separated into x2sys_init.py and x2sys_cross.py inside the src/ folder as per #807.

…oss (#951) Move the x2sys modules originally wrapped in #546. The single x2sys.py has been separated into x2sys_init.py and x2sys_cross.py inside the src/ folder as per #807.

…oss (GenericMappingTools#951) Move the x2sys modules originally wrapped in GenericMappingTools#546. The single x2sys.py has been separated into x2sys_init.py and x2sys_cross.py inside the src/ folder as per GenericMappingTools#807.

weiji14 added the feature Brand new feature label Jul 23, 2020

Fix pylint style errors

5e4d419

vercel bot temporarily deployed to Preview July 23, 2020 10:26 Inactive

weiji14 commented Jul 23, 2020

View reviewed changes

vercel bot temporarily deployed to Preview July 23, 2020 11:23 Inactive

Check that filename input/output works in x2sys_cross

007e957

weiji14 force-pushed the x2sys_cross branch from fe528d4 to 007e957 Compare July 23, 2020 11:25

vercel bot temporarily deployed to Preview July 23, 2020 11:26 Inactive

weiji14 mentioned this pull request Jul 23, 2020

Passing in virtual files into the supplementary x2sys modules GenericMappingTools/gmt#3717

Open

weiji14 added 2 commits July 24, 2020 13:11

Alias verbose (V)

4a5d5db

Pass in remote filename directly to x2sys_cross, turn on debug mode too

383134f

vercel bot temporarily deployed to Preview July 24, 2020 01:17 Inactive

Let x2sys_file handle multiple input tracks and external crossovers

b5aa8a8

vercel bot temporarily deployed to Preview July 24, 2020 12:21 Inactive

NotImplementedError for pd.DataFrame, GMTInvalidInput for invalid types

4309346

Support only filename inputs into x2sys_cross for now, because passing in matrix type virtualfiles into GMT C API doesn't work properly yet.

vercel bot temporarily deployed to Preview July 24, 2020 23:04 Inactive

vercel bot temporarily deployed to Preview July 24, 2020 23:25 Inactive

Complete long aliases for x2sys_init

9a2b40b

Aliased suffix (E), discontinuity (G), spacing (I), units (I), region (R), gap (W), distcalc (j) for x2sys_init. Also checked that passing a list into region (R), spacing (I), units (N), and gap (W) works properly.

vercel bot temporarily deployed to Preview July 25, 2020 23:14 Inactive

weiji14 added 2 commits July 26, 2020 12:55

Merge branch 'master' into x2sys_cross

8f22648

Complete most long aliases for x2sys_cross

822e06c

Aliased combitable (A), runtimes(C), interpolation (I), region (R), speed(S), numpoints (W) for x2sys_cross. Also checked that setting the region (R), interpolation (I) and numpoints (W) args work properly.

vercel bot temporarily deployed to Preview July 26, 2020 00:57 Inactive

weiji14 commented Jul 26, 2020

View reviewed changes

Fix slightly different z_X.mean() values on different platforms

90776dd

On Linux, z_X.mean() = -139.196212, On macOS/Windows, z_X.mean() = -139.20221.

vercel bot temporarily deployed to Preview July 26, 2020 01:43 Inactive

vercel bot temporarily deployed to Preview July 27, 2020 01:35 Inactive

Parse datetimes on the 3rd and 4th columns

f61e289

weiji14 force-pushed the x2sys_cross branch from 5533535 to f61e289 Compare July 27, 2020 01:38

vercel bot temporarily deployed to Preview July 27, 2020 01:38 Inactive

vercel bot temporarily deployed to Preview July 27, 2020 04:08 Inactive

This was referenced Jul 27, 2020

Add common alias verbose (V) #550

Merged

Add common alias distcalc (j) #551

Merged

Rename df to dataframe or table to stop pylint complaining

e7f42fd

vercel bot temporarily deployed to Preview July 27, 2020 10:41 Inactive

Merge branch 'master' into x2sys_cross

49114fc

vercel bot temporarily deployed to Preview July 27, 2020 21:59 Inactive

seisman added this to the 0.2.x milestone Jul 27, 2020

seisman approved these changes Jul 27, 2020

View reviewed changes

weiji14 merged commit a851624 into master Jul 27, 2020

weiji14 deleted the x2sys_cross branch July 27, 2020 22:58

seisman reviewed Jul 28, 2020

View reviewed changes

weiji14 mentioned this pull request Jul 28, 2020

Cleanup track files in test_x2sys_cross_input_two_filenames #552

Merged

5 tasks

weiji14 mentioned this pull request Aug 13, 2020

Analysis of ICESat-2 crossover track height errors weiji14/deepicedrain#146

Merged

3 tasks

This was referenced Sep 2, 2020

RFC Allow for io.StringIO inputs to certain modules #576

Closed

Allow passing in pandas dataframes to x2sys_cross #591

Merged

weiji14 mentioned this pull request Feb 22, 2021

Move x2sys.py to src folder and split it into x2sys_init and x2sys_cross #951

Merged

5 tasks

weiji14 mentioned this pull request Aug 2, 2021

Wrapper for x2sys_solve #1406

Open

weiji14 mentioned this pull request Oct 5, 2021

Refactor x2sys_cross to remove verbose info and return output statements #1572

Merged

5 tasks

weiji14 mentioned this pull request Mar 16, 2024

Get rid of temporary files from pygmt functions and plotting methods #2730

Closed

31 tasks

seisman mentioned this pull request Apr 8, 2024

x2sys_cross: Refactor to get rid of temporary files and have consistent table-like output behavior #3160

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wrap x2sys_init and x2sys_cross #546

Wrap x2sys_init and x2sys_cross #546

weiji14 commented Jul 23, 2020 •

edited

Loading

weiji14 Jul 23, 2020

seisman Jul 23, 2020

weiji14 Jul 23, 2020 •

edited

Loading

weiji14 Jul 26, 2020

weiji14 Jul 26, 2020 •

edited

Loading

seisman commented Jul 27, 2020

weiji14 commented Jul 27, 2020

seisman left a comment

weiji14 commented Jul 27, 2020

seisman Jul 28, 2020 •

edited

Loading

		elif kind == "matrix":
		file_context = lib.virtualfile_from_matrix(track.values)

Wrap x2sys_init and x2sys_cross #546

Wrap x2sys_init and x2sys_cross #546

Conversation

weiji14 commented Jul 23, 2020 • edited Loading

weiji14 Jul 23, 2020

Choose a reason for hiding this comment

seisman Jul 23, 2020

Choose a reason for hiding this comment

weiji14 Jul 23, 2020 • edited Loading

Choose a reason for hiding this comment

weiji14 Jul 26, 2020

Choose a reason for hiding this comment

weiji14 Jul 26, 2020 • edited Loading

Choose a reason for hiding this comment

seisman commented Jul 27, 2020

weiji14 commented Jul 27, 2020

seisman left a comment

Choose a reason for hiding this comment

weiji14 commented Jul 27, 2020

seisman Jul 28, 2020 • edited Loading

Choose a reason for hiding this comment

weiji14 commented Jul 23, 2020 •

edited

Loading

weiji14 Jul 23, 2020 •

edited

Loading

weiji14 Jul 26, 2020 •

edited

Loading

seisman Jul 28, 2020 •

edited

Loading