Dolfyn updates #212

jmcvey3 · 2022-12-20T01:54:05Z

A number of updates:

Fixes for TRDI reader: Dolfyn processing errors #182, Dolfyn reader estimates the number of pings in the file based on the length in bytes of the first ping #206, Dolfyn incorrectly processing irregular Bin sizes #207
Fix for VMDAS files with interleaved pings
Motion correction for duty-cycled ADVs
lncorporated logging module for into reader debuggers
Ensure datatypes are saved as float32 instead of float64 to reduce filesize
Added another ADCP clean function
Updated docstrings to mhkit formatting
Other minor bugfixes

into dolfyn-updates

ssolson · 2023-01-03T16:06:40Z

James thank you for submitting the PR. To start the review I have a couple of questions about large log files:

1.16 MB examples/data/dolfyn/test_data/AWAC_test01.log
6.74 MB examples/data/dolfyn/test_data/vector_data_imu01.log
3.05 MB examples/data/dolfyn/test_data/RDI_withBT.log

These add about 11 MB of log data. Is there a way we could reduce these log file sizes such as checking against a smaller dataset (and therefore a smaller associated log file), checking only a portion of the produced log, or mocking the data somehow?

ssolson

A couple of comments for consideration. Still working my way through these so this is just what I have so far.

ssolson · 2023-01-03T16:14:53Z

mhkit/dolfyn/adp/clean.py

@@ -170,19 +171,24 @@ def nan_beyond_surface(ds, val=np.nan):
      The adcp dataset to clean
    val : nan or numeric
      Specifies the value to set the bad values to (default np.nan).
+    inplace : bool (default: False)


Its been a while since I have built the docs but I believe we typically put the default in the description because when we do this the text is interpreted incorrectly by the interpreter our docs use. I could be wrong though.

It looks like it puts everything after the first colon in italics (https://mhkit-software.github.io/MHKiT/mhkit-python/api.dolfyn.html#io), but I'll look at the other docstring defaults

ssolson · 2023-01-03T16:46:41Z

mhkit/dolfyn/adp/clean.py

+    var : xarray.DataArray
+      Variable to clean
+    thresh : numeric
+      The maximum value of velocity to screen


Should specify default

ssolson · 2023-01-03T16:49:36Z

mhkit/dolfyn/adp/clean.py

+    var = var.copy(deep=True)
+
+    bd = np.zeros(var.shape, dtype='bool')
+    bd |= (np.abs(var.values) > thresh)


fancy. I had to look this operator up.

ssolson · 2023-01-03T16:50:12Z

mhkit/dolfyn/adp/clean.py

    method : string
      Interpolation method to use
    maxgap : numeric
-      Maximum length of missing data in seconds to interpolate across
+      Maximum gap of missing data to interpolate across


Maybe specify default

mhkit/dolfyn/adv/clean.py

ssolson · 2023-01-03T16:55:26Z

mhkit/dolfyn/adv/clean.py

      interpolation occurs over
    method : string
      Interpolation scheme to use (linear, cubic, pchip, etc)
-    max_gap : int
+    limit : int


I think this should be maxgap

ssolson · 2023-01-03T16:56:56Z

mhkit/dolfyn/adv/clean.py


    return u


-def _interp_nan(da, npt, method, max_gap):
+def _interp_nan(da, npt, method, maxgap):


consider adding method and maxgap to the docstring as well as a Returns section

ssolson · 2023-01-03T18:04:51Z

mhkit/dolfyn/adv/motion.py

+        if rng > 2 or (mean > interval+1 and mean < interval-1):
+            raise Exception("Bad duty cycle detected")
+
+        # If this passes, it means we're save to blindly skip n_burst for every integral


I think it should be "safe" instead of save

jmcvey3 · 2023-01-05T00:42:11Z

James thank you for submitting the PR. To start the review I have a couple of questions about large log files:

1.16 MB examples/data/dolfyn/test_data/AWAC_test01.log

6.74 MB examples/data/dolfyn/test_data/vector_data_imu01.log

3.05 MB examples/data/dolfyn/test_data/RDI_withBT.log

These add about 11 MB of log data. Is there a way we could reduce these log file sizes such as checking against a smaller dataset (and therefore a smaller associated log file), checking only a portion of the produced log, or mocking the data somehow?

Good point, yes I can reduce the size of these.

ssolson

@jmcvey3 a couple of minor comments. Great job on getting the docstring consistent.

Is there anything I missed that you would like to talk through before we approve the PR?

ssolson · 2023-01-19T16:22:17Z

mhkit/dolfyn/adv/motion.py

-
+      acceleration.  Default = 1/3 of `accel_filtfreq`
    """
+
    _default_accel_filtfreq = 0.03


Is _default_accel_filtfreq the default referred to as equaling 1/3 in the docstring?

No, there's two filter frequencies here. One for acceleration and one for velocity. The velocity filter frequency defaults to 1/3 of the acceleration frequency (0.03), so its default is 0.01.

ssolson · 2023-01-19T16:43:11Z

mhkit/dolfyn/adv/turbulence.py

@@ -259,12 +221,12 @@ def dissipation_rate_LT83(self, psd, U_mag, f_range=[6.28, 12.57]):
        out = (psd.isel(freq=idx) *
               freq.isel(freq=idx)**(5/3) / a).mean(axis=-1)**(3/2) / U

-        out = xr.DataArray(out, name='dissipation_rate',
+        out = xr.DataArray(out.astype('float32'),


Did you mean to remove the name parameter?

Yes, turns out the name is immediately overwritten once a dataArray is assigned a variable name in a dataset

ssolson · 2023-01-19T16:44:50Z

mhkit/dolfyn/adv/turbulence.py

                DAA[L] = np.nanmean((up[L:] - up[:-L]) ** 2, dtype=np.float64)
            cv2 = DAA / (lag ** (2 / 3))
            cv2m = np.median(cv2[np.logical_not(np.isnan(cv2))])
            out[slc[:-1]] = (cv2m / 2.1) ** (3 / 2)

-        return xr.DataArray(out, name='dissipation_rate',
+        return xr.DataArray(out.astype('float32'),


Guessing you did intend to remove the name parameter since it is gone here as well.

ssolson · 2023-01-19T16:49:50Z

mhkit/dolfyn/io/base.py

@@ -25,6 +25,7 @@ def _get_filetype(fname):
    with open(fname, 'rb') as rdr:
        bytes = rdr.read(40)
    code = bytes[:2].hex()
+    #print("{} - {}".format(fname.rsplit('/')[-1], bytes))


This looks leftover from debugging

Thanks, yes it is

ssolson · 2023-01-19T17:46:10Z

mhkit/dolfyn/io/rdi_lib.py

+    # def checksum(self,):
+    #     """
+    #     The next byte(s) are the expected checksum.  Perform the checksum.
+    #     """
+    #     if self.cs:
+    #         cs = self.read(1, self.cs._frmt)
+    #         self.cs(cs, True)
+    #     else:
+    #         raise Exception('CheckSum not requested for this file')


Is this something you plan to bring back in the future?

ssolson · 2023-01-19T17:47:20Z

mhkit/dolfyn/io/rdi_lib.py

+# ics = 0  # This is a holder for the checksum index
+# class checksum():
+#     # Checksum for TRDI


Similar to above is a checksum() class going to be added in a future update?

No this is old. Looks like I forgot to drop it

ssolson · 2023-01-19T17:52:08Z

mhkit/dolfyn/io/rdi.py

+        # if cfg['prog_ver'] >= 56:
+        #     fd.seek(1, 1)
+        #     pings_per_ensemble = fd.read_ui16(1)
+        #     exact_freq = fd.read_ui8(3)
+        #     #cfg['exact_freq'] = int("".join(str(x) for x in exact_freq))
+        #     self._nbyte += 6
+


Looks like this was replaced by the definition above. Do we want to keep this?

I'll remove it for now from the main branch until it's needed. I'm not entirely sure why it's in their documentation.

jmcvey3 · 2023-02-01T01:06:07Z

@ssolson In case I hadn't mentioned it, I think this PR should now be covered. I'll need to update the docs, but I believe it's easiest to wait until this is merged into Develop?

ssolson · 2023-02-01T15:08:12Z

Thanks James. I'm going to merge into Develop. Please update the Docs as you mentioned.

* Merge Master into Develop (#179) * Include last day of year (#160) Include the last day of the year when using years parameter. Fixes #154. * Timezone Bug fix: remove `.replace()` for `.astimezone()` (#161) * Bug fix: remove timezone replace for astimezone * add pytz to required packages * Fix: Pandas latest (#159) * Plot each col in DataFrame individually * Remove numpy and pandas version requirements * WDRT (#141) * Working version of Gaussian Copula * Create dedicated function for iso probs and quantiles * Contour plots worsk for single and multiple contours * Gumbel Copula. * Clayton Copula * Rosenblatt Copula * Guass and gumbel general copula * Add support for nonparametric gaussian, clayton, and gumbel copulas * Require statsmodel for nonparametric KDE copulas * Small changes * Fix bug in KDE log transformation * Adding docstrings * All doc strings updated * Add markers option to contours plotting * Add Copula tests comparing to WDRTresults * Fix x1, x2 pts bug in KDE contour, clean up * Example showing the calculation of all copla methods and comparing to WDRT results * Simplifications and lanuguage cleanup * Add statsmodel * Add testing data files * In plot envrionmental contors convert x1,x2 to values if a Series is passed * Docstring typo corrections * Corrected notebook description typos. * module and example for short-term extreme distributions * docstrings for short-term extreme functions * fixes #140 Speedup surface elevation calculation. * correction to surface_elevation * elevation * minor fixes based on review comments * add environmental_contours to init * Remove WDRT functionality * All WDRT contour functionality can be called from one function * Rework tests for new contour functions setup and location * Move all contour example into this file * Update to work with new structure. * Remove all commented out wdrt functions from resource * No changes made. Reverting back. * Have PCA method use general fit. Adds PCAmethod to fit method but can also accept the PCA dictionary. General Docstring Cleanup * Update discussion around the use of the copula method. * Remove copula stand alone notebook * minor formmating changes * Remove test bugs created from merge with origin * Import env contours into resource module and adjust test, examples, ect to function with tthe new structure * Uncommented tests * Remove reference to import the env contours module * fixing minor typo * Move env contours to _file and adjust package to handle new structure * Minor formatting cleanup * Cleanup unsed packages and variables * intial mler upload * formalize functions * add tests and example * sampling rate to averaging period * rename MLER example * working * merging * WDRT functions and examples finalized. Missing tests. * integrate mler into extreme * fixes tests? * fix tests * fix tests * fix full sea state example? * test contour samples * test sample full seastate * random seed * test long term extreme * expand MLER functionality * test short-term extremes * fix mler tests * MLER test error * mler fix 2 * update formats and asserts * further cleanup of countours.py for consistency, typos, and pep8 * run tests? * fix test? * change lists to np.array so they pass the assert statements * clean extreme.py for consistency, pep8 * points_per_interval should be int * typing in examples * full sea state example: make it easier to follow, add more explanation, and use consistent naming * contour example: make it easier to follow, add more explanation, and use consistent naming * Alias ste function and minor formatting. * naming convention * rename example * ste example * allow weights to be np.array * allow weights to be np.ndarray Co-authored-by: ssolson <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> * DOLfYN IO (#126) * input output files + testing * Code update * Test reformatting * Testing paths * dependency switch from h5netcdf to netcdf4 * Code cleanup * Test import fix * Minor docstring edits * Add dolfyn test data * Test fixes for mhkit * Removing uneeded test files, improving consistency * Not sure why these files didn't upload * Organizational changes and clarifications. Request for additional clarification * Dolfyn codebase updates * Dolfyn example notebooks update * Testing updates * Update example datafile * Latest dolfyn v0.13.0 updates * Name change * input output files + testing * Code update * Organizational changes and clarifications. Request for additional clarification * Dolfyn codebase updates * fix bug and make faster. Un-hard-code default seed/phase. * fix tests * delete commented out tests * retrigger checks * Checkpoint push * New dolfyn data files * Reorganize dolfyn testing * pytest install for warnings tests * Final update for IO * Remove future updates * Ensure file compression runs * Update numpy dependency * trying numpy v1.22 * changing numpy requirements * Remake unittest test cases * Save format options * Add dolfyn view to notebooks * Decode remaining binary ad2cp variables * DOLfYN 1.0.0 dependency Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: Michelen <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> * regenerate WDRT examples (#163) * updating version number in package * fixing dolfyn imports from pip * updating version number for bug fix * NDBC Metocean data (#152) * catch extra header in ndbc data * NDBC cwind example * update unit catch in request_data * update gust plot to show more data * add tests to read cwind with and without units * move graphics import to ndbc module and fix capitalization * use resample, add gridlines, rename variables, replace NDBC nans up front * rename wind_example to metocean_example * adding "Develop" branch to PR Co-authored-by: Carlos A. Michelén Ströfer <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: Adam Keester <[email protected]> * Test Suite Restructure (#174) * test_wave file to folder with individual feature files * Move io tests into io folder * Send plots to a plots folder * Move resource characterizations plot tests into resource metrics file * move load tests to folder * move power tests to folder * move river tests to folder * move tidal tests to folder * move utils tests to folder * add inits so that tests pick up the tests in subfolders * Require previous netCDF4 release. * FIx data directory path * Fix data directory path * Fix load data directory path * Remove relpath call to see if it fixes Windows test suite issues * Remove relpath from the rest of io tests in wave * DOLfYN source code (#169) * Initial push * Latest bugfixes * Remove phase-in text * Update docstrings * Readability, imports * Make functions public * Docstring updates * Fix test oversight * Fix docstring * Missing timestamp fix for classic Nortek * Docstring updates * Renaming functions per mhkit standards * Single array vs binned array functions * Reorganizing analysis code * Update invalid orientation matrix warning * Add config details to dataset attributes * dolfyn bugfixes Co-authored-by: jmcvey3 <[email protected]> * Delft 3D Timestep to Seconds Function (#168) * fixing file history * fixing file history * fixing file history * made revisions from ssolson review * changed time_stamp to timestamp * updated varible names * updated variable names * fixed syntax line 63 * umdated variable name * fixed typos in doc strings * ssolson review 5-26-2022 * deleted excessive code in get_all_data * chaged TI to a % from a fraction * adjusted TI test to be a % * fixed doc string for TI function * Docstring adjustments * updated doc strings * Make convert_time a hidden func, make 2 functions to access it * Minor changes to get_lay_data docstring and formatting * small changes * remove print from create_points * example notebook clean up * fixed merge confilics * updated after merge * final developmet barnach merge updates * deleated space * Delete test_river.py * added coverage to test * updated min and max plot labels Co-authored-by: Browning <[email protected]> Co-authored-by: ssolson <[email protected]> * NetCDF4 (#181) * Use conda TEST env on Py 3.7, 3.8, 3.9 * Use pip install with Py 3.8 and 3.9. * Updates for WEC-Sim v5.0 (#185) * update read_wecsim for WEC-Sim v5.0 * add cable class * update notebook with results * add try-except for Morrison (v4.1) vs Morison (v4.2+) * add cable check and dataset to wave tests * update cable dataset * Fix pip install tests (#194) * Require previous version of NetCDF4. * Use original pre install for environemnt in pip tests * Only test pip * No loading shell PATH * Do not upgrade h5py * Set NETCDF to 1.5.8 * pytest and coverage * yaml syntax fix * Change need to pip-build * Control coverage and test via rc files * Move configuration file to workflows directory * Move configuration file to workflows directory * Specify the location of configuration files in coverage/pytest call * run conda & pip, pass both to coveralls * conda bash -l {0} * Add hindcast build which runs serial * configuration files for hindcast run * Specify hindcast configuration for hindcast build * Remove version specification from hindcast calls * Specify configuration files to include/omit hindcast accordingly * Change job name * Fix configuration to hindcast file not test * Use all OS and py versions in tests * pandas <=1.5.0 * xarray <=2022.9.0 * Only hindcast * Clean up package install * Fixing hypy and h5pyd to previous versions * Run limited other tests * No hindcast tests * remove hindcast comments and coveralls call * remove py3.8 from pip build * requirements.txt netCDF <= 1.5.8 * Add function to compute value at a given return period (#193) * Run CI on push to Develop (#200) * Dolfyn general updates (#186) * Initial push * Latest bugfixes * Remove phase-in text * Update docstrings * Readability, imports * Make functions public * Docstring updates * Fix test oversight * Fix docstring * Missing timestamp fix for classic Nortek * Docstring updates * Renaming functions per mhkit standards * Single array vs binned array functions * Reorganizing analysis code * Update invalid orientation matrix warning * Add config details to dataset attributes * dolfyn bugfixes * ADV updates : * Update compression option for netcdf4 * Remove compression options * Fix clean function * Set functions as private * Move stress functions to ADV dir * Add bottleneck to reqs * Minor changes * Remove old functions * Code simplification * Update examples * Make functions public * Add bottleneck to dependencies * Fix notebook * Bugfix for beam vars Co-authored-by: jmcvey3 <[email protected]> * Delft3D z calculation (#190) * updated variable names * updated variable names * updated variable names * fixed a few docstring typos * updated s1 to water level * updated s1 to waterlevel * updated s1 to waterlevel * updated s1 to water level * updated s1 to waterlever * fixed typo depth to waterdepth * added edges = nearest example * added edges = nearest option in variable interpolate * updated z to waterdepth * added edges= nearest example Co-authored-by: Browning <[email protected]> * Directional NDBC (#197) * working on directional NDBC * clean up functions. Write docstrings. Write assert statements. Write tests. Create Tutorial. * docstring and asserts for plotting function * Bug Fix: Averaging histogram bin and wave energy (#205) * Fix averaging bug * Add outline to bin counts text for better contrast * Update plotting function and plot in example notebook * Fix: update variable name to remove reference before assignment error (#208) * update wave.contours.samples_contour to not get variable referenced before assignment error Co-authored-by: Graham Penrose <[email protected]> * HSDS (#211) Adds hindcast tests back into the test suite. The NREL HSDS API issues were resolved by creating multiple calls to the API for direction wave spectrum requests. Additionally, an exponential back-off time was implemented to retry calls with an increased wait time between calls. Co-authored-by: Adam Keester <[email protected]> * Provide function to convert from Te to Tp using ITTC approximation (#210) * Provide function to convert from Te to Tp using ITTC approximation * Apply suggestions from code review Co-authored-by: Adam Keester <[email protected]> * Metocean module - WIND Toolkit (#187) * initial script * update wind_toolkit with 4 regions and 1-hr or 5-min data * update parameters of wind_toolkit functions * compare NDBC and WIND metocean data * fix typo in wpto hindcast example * finish metocean example and add results * add wind_toolkit to wave/io/__init__.py * update WIND Toolkit parameter list * initial test structure for WIND toolkit * finish metocean example * add function to plot each region * wind toolkit tests and test data * add tests for misc wind_toolkit MHKiT functions and cases * fix wind toolkit tests * add elevation_to_string utility function * misc cleanup * clarify available parameters * add users lat_lon to plot_region visualization * update example with new functions * add numpy dependency back in * Ingnore the new hindcast folder * Include only the new hindcast folder * Move hindcast tests to hindcast folder * Specify exclusion of hindcast from standard coverage * update hindcast coverage * remove unnecessary header * move hindcast and wind_toolkit to wave/io/hindcast * update .coveragerc files * update paths for new wave.io.hindcast directory Co-authored-by: ssolson <[email protected]> * Dolfyn updates (#212) * RDI reader, logging, duty cycle motion correction * Updates * Force float32 datatype * Docstring formatting * Change 'default' docstring default * Remove old comments * Trasect comparison (#199) * Require previous version of NetCDF4. * updated variable names * updated variable names * updated variable names * fixed a few docstring typos * updated s1 to water level * updated s1 to waterlevel * updated s1 to waterlevel * updated s1 to water level * updated s1 to waterlever * fixed typo depth to waterdepth * D3d Tanana Transect Example * added optional edge interpolation with nearest * updated z to water depth * RDI reader, logging, duty cycle motion correction * add dolfyn, currenly will not run on the branch need jmcvey3 trdi_5beam * found river bottom and removed data below * Updates * Force float32 datatype * added downsampelind comparison and tried contourf * D3D tanana data * tanana transect 2 * tanana transect 3 * Docstring formatting * Change 'default' docstring default * Remove old comments * updated code to match Energies paper * Review * Boat transect image * updated variable names * updated variable names * updated variable names added to discriptions * Clean up and suggestions * added to the examples * TRTS study edits * code and discription updates * Edits and TODOs * moved files, updated discriptions * readding Delft3D_example notebook * added USGS discharge * updated doc strings * example updates and move data files * updated doc strings and Error equations * updated doc strings * pulled down development branch --------- Co-authored-by: ssolson <[email protected]> Co-authored-by: Browning <[email protected]> Co-authored-by: jmcvey3 <[email protected]> --------- Co-authored-by: Carlos A. Michelén Ströfer <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: Adam Keester <[email protected]> Co-authored-by: Emily Browning <[email protected]> Co-authored-by: Browning <[email protected]> Co-authored-by: Mark Bruggemann <[email protected]> Co-authored-by: Graham Penrose <[email protected]>

* Require previous version of NetCDF4. * Merge Master into Develop (#179) * Include last day of year (#160) Include the last day of the year when using years parameter. Fixes #154. * Timezone Bug fix: remove `.replace()` for `.astimezone()` (#161) * Bug fix: remove timezone replace for astimezone * add pytz to required packages * Fix: Pandas latest (#159) * Plot each col in DataFrame individually * Remove numpy and pandas version requirements * WDRT (#141) * Working version of Gaussian Copula * Create dedicated function for iso probs and quantiles * Contour plots worsk for single and multiple contours * Gumbel Copula. * Clayton Copula * Rosenblatt Copula * Guass and gumbel general copula * Add support for nonparametric gaussian, clayton, and gumbel copulas * Require statsmodel for nonparametric KDE copulas * Small changes * Fix bug in KDE log transformation * Adding docstrings * All doc strings updated * Add markers option to contours plotting * Add Copula tests comparing to WDRTresults * Fix x1, x2 pts bug in KDE contour, clean up * Example showing the calculation of all copla methods and comparing to WDRT results * Simplifications and lanuguage cleanup * Add statsmodel * Add testing data files * In plot envrionmental contors convert x1,x2 to values if a Series is passed * Docstring typo corrections * Corrected notebook description typos. * module and example for short-term extreme distributions * docstrings for short-term extreme functions * fixes #140 Speedup surface elevation calculation. * correction to surface_elevation * elevation * minor fixes based on review comments * add environmental_contours to init * Remove WDRT functionality * All WDRT contour functionality can be called from one function * Rework tests for new contour functions setup and location * Move all contour example into this file * Update to work with new structure. * Remove all commented out wdrt functions from resource * No changes made. Reverting back. * Have PCA method use general fit. Adds PCAmethod to fit method but can also accept the PCA dictionary. General Docstring Cleanup * Update discussion around the use of the copula method. * Remove copula stand alone notebook * minor formmating changes * Remove test bugs created from merge with origin * Import env contours into resource module and adjust test, examples, ect to function with tthe new structure * Uncommented tests * Remove reference to import the env contours module * fixing minor typo * Move env contours to _file and adjust package to handle new structure * Minor formatting cleanup * Cleanup unsed packages and variables * intial mler upload * formalize functions * add tests and example * sampling rate to averaging period * rename MLER example * working * merging * WDRT functions and examples finalized. Missing tests. * integrate mler into extreme * fixes tests? * fix tests * fix tests * fix full sea state example? * test contour samples * test sample full seastate * random seed * test long term extreme * expand MLER functionality * test short-term extremes * fix mler tests * MLER test error * mler fix 2 * update formats and asserts * further cleanup of countours.py for consistency, typos, and pep8 * run tests? * fix test? * change lists to np.array so they pass the assert statements * clean extreme.py for consistency, pep8 * points_per_interval should be int * typing in examples * full sea state example: make it easier to follow, add more explanation, and use consistent naming * contour example: make it easier to follow, add more explanation, and use consistent naming * Alias ste function and minor formatting. * naming convention * rename example * ste example * allow weights to be np.array * allow weights to be np.ndarray Co-authored-by: ssolson <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> * DOLfYN IO (#126) * input output files + testing * Code update * Test reformatting * Testing paths * dependency switch from h5netcdf to netcdf4 * Code cleanup * Test import fix * Minor docstring edits * Add dolfyn test data * Test fixes for mhkit * Removing uneeded test files, improving consistency * Not sure why these files didn't upload * Organizational changes and clarifications. Request for additional clarification * Dolfyn codebase updates * Dolfyn example notebooks update * Testing updates * Update example datafile * Latest dolfyn v0.13.0 updates * Name change * input output files + testing * Code update * Organizational changes and clarifications. Request for additional clarification * Dolfyn codebase updates * fix bug and make faster. Un-hard-code default seed/phase. * fix tests * delete commented out tests * retrigger checks * Checkpoint push * New dolfyn data files * Reorganize dolfyn testing * pytest install for warnings tests * Final update for IO * Remove future updates * Ensure file compression runs * Update numpy dependency * trying numpy v1.22 * changing numpy requirements * Remake unittest test cases * Save format options * Add dolfyn view to notebooks * Decode remaining binary ad2cp variables * DOLfYN 1.0.0 dependency Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: Michelen <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> * regenerate WDRT examples (#163) * updating version number in package * fixing dolfyn imports from pip * updating version number for bug fix * NDBC Metocean data (#152) * catch extra header in ndbc data * NDBC cwind example * update unit catch in request_data * update gust plot to show more data * add tests to read cwind with and without units * move graphics import to ndbc module and fix capitalization * use resample, add gridlines, rename variables, replace NDBC nans up front * rename wind_example to metocean_example * adding "Develop" branch to PR Co-authored-by: Carlos A. Michelén Ströfer <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: Adam Keester <[email protected]> * Test Suite Restructure (#174) * test_wave file to folder with individual feature files * Move io tests into io folder * Send plots to a plots folder * Move resource characterizations plot tests into resource metrics file * move load tests to folder * move power tests to folder * move river tests to folder * move tidal tests to folder * move utils tests to folder * add inits so that tests pick up the tests in subfolders * Require previous netCDF4 release. * FIx data directory path * Fix data directory path * Fix load data directory path * Remove relpath call to see if it fixes Windows test suite issues * Remove relpath from the rest of io tests in wave * DOLfYN source code (#169) * Initial push * Latest bugfixes * Remove phase-in text * Update docstrings * Readability, imports * Make functions public * Docstring updates * Fix test oversight * Fix docstring * Missing timestamp fix for classic Nortek * Docstring updates * Renaming functions per mhkit standards * Single array vs binned array functions * Reorganizing analysis code * Update invalid orientation matrix warning * Add config details to dataset attributes * dolfyn bugfixes Co-authored-by: jmcvey3 <[email protected]> * Delft 3D Timestep to Seconds Function (#168) * fixing file history * fixing file history * fixing file history * made revisions from ssolson review * changed time_stamp to timestamp * updated varible names * updated variable names * fixed syntax line 63 * umdated variable name * fixed typos in doc strings * ssolson review 5-26-2022 * deleted excessive code in get_all_data * chaged TI to a % from a fraction * adjusted TI test to be a % * fixed doc string for TI function * Docstring adjustments * updated doc strings * Make convert_time a hidden func, make 2 functions to access it * Minor changes to get_lay_data docstring and formatting * small changes * remove print from create_points * example notebook clean up * fixed merge confilics * updated after merge * final developmet barnach merge updates * deleated space * Delete test_river.py * added coverage to test * updated min and max plot labels Co-authored-by: Browning <[email protected]> Co-authored-by: ssolson <[email protected]> * NetCDF4 (#181) * Use conda TEST env on Py 3.7, 3.8, 3.9 * Use pip install with Py 3.8 and 3.9. * Updates for WEC-Sim v5.0 (#185) * update read_wecsim for WEC-Sim v5.0 * add cable class * update notebook with results * add try-except for Morrison (v4.1) vs Morison (v4.2+) * add cable check and dataset to wave tests * update cable dataset * updated variable names * updated variable names * updated variable names * fixed a few docstring typos * updated s1 to water level * updated s1 to waterlevel * updated s1 to waterlevel * updated s1 to water level * updated s1 to waterlever * fixed typo depth to waterdepth * D3d Tanana Transect Example * Fix pip install tests (#194) * Require previous version of NetCDF4. * Use original pre install for environemnt in pip tests * Only test pip * No loading shell PATH * Do not upgrade h5py * Set NETCDF to 1.5.8 * pytest and coverage * yaml syntax fix * Change need to pip-build * Control coverage and test via rc files * Move configuration file to workflows directory * Move configuration file to workflows directory * Specify the location of configuration files in coverage/pytest call * run conda & pip, pass both to coveralls * conda bash -l {0} * Add hindcast build which runs serial * configuration files for hindcast run * Specify hindcast configuration for hindcast build * Remove version specification from hindcast calls * Specify configuration files to include/omit hindcast accordingly * Change job name * Fix configuration to hindcast file not test * Use all OS and py versions in tests * pandas <=1.5.0 * xarray <=2022.9.0 * Only hindcast * Clean up package install * Fixing hypy and h5pyd to previous versions * Run limited other tests * No hindcast tests * remove hindcast comments and coveralls call * remove py3.8 from pip build * requirements.txt netCDF <= 1.5.8 * Add function to compute value at a given return period (#193) * Run CI on push to Develop (#200) * added optional edge interpolation with nearest * updated z to water depth * Dolfyn general updates (#186) * Initial push * Latest bugfixes * Remove phase-in text * Update docstrings * Readability, imports * Make functions public * Docstring updates * Fix test oversight * Fix docstring * Missing timestamp fix for classic Nortek * Docstring updates * Renaming functions per mhkit standards * Single array vs binned array functions * Reorganizing analysis code * Update invalid orientation matrix warning * Add config details to dataset attributes * dolfyn bugfixes * ADV updates : * Update compression option for netcdf4 * Remove compression options * Fix clean function * Set functions as private * Move stress functions to ADV dir * Add bottleneck to reqs * Minor changes * Remove old functions * Code simplification * Update examples * Make functions public * Add bottleneck to dependencies * Fix notebook * Bugfix for beam vars Co-authored-by: jmcvey3 <[email protected]> * RDI reader, logging, duty cycle motion correction * Delft3D z calculation (#190) * updated variable names * updated variable names * updated variable names * fixed a few docstring typos * updated s1 to water level * updated s1 to waterlevel * updated s1 to waterlevel * updated s1 to water level * updated s1 to waterlever * fixed typo depth to waterdepth * added edges = nearest example * added edges = nearest option in variable interpolate * updated z to waterdepth * added edges= nearest example Co-authored-by: Browning <[email protected]> * Directional NDBC (#197) * working on directional NDBC * clean up functions. Write docstrings. Write assert statements. Write tests. Create Tutorial. * docstring and asserts for plotting function * add dolfyn, currenly will not run on the branch need jmcvey3 trdi_5beam * found river bottom and removed data below * Bug Fix: Averaging histogram bin and wave energy (#205) * Fix averaging bug * Add outline to bin counts text for better contrast * Update plotting function and plot in example notebook * Fix: update variable name to remove reference before assignment error (#208) * update wave.contours.samples_contour to not get variable referenced before assignment error Co-authored-by: Graham Penrose <[email protected]> * HSDS (#211) Adds hindcast tests back into the test suite. The NREL HSDS API issues were resolved by creating multiple calls to the API for direction wave spectrum requests. Additionally, an exponential back-off time was implemented to retry calls with an increased wait time between calls. Co-authored-by: Adam Keester <[email protected]> * Provide function to convert from Te to Tp using ITTC approximation (#210) * Provide function to convert from Te to Tp using ITTC approximation * Apply suggestions from code review Co-authored-by: Adam Keester <[email protected]> * Updates * Force float32 datatype * added downsampelind comparison and tried contourf * D3D tanana data * tanana transect 2 * tanana transect 3 * Docstring formatting * Metocean module - WIND Toolkit (#187) * initial script * update wind_toolkit with 4 regions and 1-hr or 5-min data * update parameters of wind_toolkit functions * compare NDBC and WIND metocean data * fix typo in wpto hindcast example * finish metocean example and add results * add wind_toolkit to wave/io/__init__.py * update WIND Toolkit parameter list * initial test structure for WIND toolkit * finish metocean example * add function to plot each region * wind toolkit tests and test data * add tests for misc wind_toolkit MHKiT functions and cases * fix wind toolkit tests * add elevation_to_string utility function * misc cleanup * clarify available parameters * add users lat_lon to plot_region visualization * update example with new functions * add numpy dependency back in * Ingnore the new hindcast folder * Include only the new hindcast folder * Move hindcast tests to hindcast folder * Specify exclusion of hindcast from standard coverage * update hindcast coverage * remove unnecessary header * move hindcast and wind_toolkit to wave/io/hindcast * update .coveragerc files * update paths for new wave.io.hindcast directory Co-authored-by: ssolson <[email protected]> * Change 'default' docstring default * Remove old comments * updated code to match Energies paper * Review * Boat transect image * updated variable names * Dolfyn updates (#212) * RDI reader, logging, duty cycle motion correction * Updates * Force float32 datatype * Docstring formatting * Change 'default' docstring default * Remove old comments * updated variable names * updated variable names added to discriptions * Clean up and suggestions * added to the examples * TRTS study edits * code and discription updates * Edits and TODOs * moved files, updated discriptions * readding Delft3D_example notebook * added USGS discharge * updated doc strings * example updates and move data files * updated doc strings and Error equations * updated doc strings * pulled down development branch * updated figure visibility * updated figure at L2 equation * restor old version * updated L2 eqation and intro fig * updated L2 eqation and intro fig * updated L2 equation * pulled in Develope branch * updated figure formate --------- Co-authored-by: ssolson <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: Carlos A. Michelén Ströfer <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: Adam Keester <[email protected]> Co-authored-by: Browning <[email protected]> Co-authored-by: Mark Bruggemann <[email protected]> Co-authored-by: Graham Penrose <[email protected]>

jmcvey3 added 4 commits November 8, 2022 08:59

RDI reader, logging, duty cycle motion correction

63db153

Updates

29b0f04

Force float32 datatype

040ee24

Merge branch 'Develop' of https://github.com/MHKiT-Software/MHKiT-Python

f6e0631

into dolfyn-updates

ssolson self-requested a review January 3, 2023 16:07

ssolson assigned jmcvey3 Jan 3, 2023

ssolson added enhancement New feature or request Clean Up Improve code consistency and readability dolfyn module labels Jan 3, 2023

This was linked to issues Jan 3, 2023

Dolfyn processing errors #182

Closed

Dolfyn reader estimates the number of pings in the file based on the length in bytes of the first ping #206

Closed

Dolfyn incorrectly processing irregular Bin sizes #207

Closed

ssolson reviewed Jan 3, 2023

View reviewed changes

jmcvey3 added 2 commits January 9, 2023 15:52

Docstring formatting

6acce9f

Change 'default' docstring default

e29deb4

ssolson reviewed Jan 19, 2023

View reviewed changes

Remove old comments

7246244

ssolson approved these changes Feb 1, 2023

View reviewed changes

ssolson merged commit 8658214 into MHKiT-Software:Develop Feb 1, 2023

jmcvey3 deleted the dolfyn-updates branch February 14, 2023 19:01

rpauly18 mentioned this pull request Jan 26, 2024

Matlab Parity with Python MHKiT-Software/MHKiT-MATLAB#110

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dolfyn updates #212

Dolfyn updates #212

jmcvey3 commented Dec 20, 2022 •

edited

Loading

ssolson commented Jan 3, 2023

ssolson left a comment

ssolson Jan 3, 2023

jmcvey3 Jan 5, 2023 •

edited

Loading

ssolson Jan 3, 2023

ssolson Jan 3, 2023

ssolson Jan 3, 2023

ssolson Jan 3, 2023

ssolson Jan 3, 2023

ssolson Jan 3, 2023

jmcvey3 commented Jan 5, 2023

ssolson left a comment

ssolson Jan 19, 2023

jmcvey3 Jan 19, 2023 •

edited

Loading

ssolson Jan 19, 2023

jmcvey3 Jan 19, 2023 •

edited

Loading

ssolson Jan 19, 2023

ssolson Jan 19, 2023

jmcvey3 Jan 19, 2023

ssolson Jan 19, 2023

ssolson Jan 19, 2023

jmcvey3 Jan 19, 2023

ssolson Jan 19, 2023

jmcvey3 Jan 19, 2023

jmcvey3 commented Feb 1, 2023

ssolson commented Feb 1, 2023

Dolfyn updates #212

Dolfyn updates #212

Conversation

jmcvey3 commented Dec 20, 2022 • edited Loading

ssolson commented Jan 3, 2023

ssolson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmcvey3 Jan 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmcvey3 commented Jan 5, 2023

ssolson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmcvey3 Jan 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmcvey3 Jan 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmcvey3 commented Feb 1, 2023

ssolson commented Feb 1, 2023

jmcvey3 commented Dec 20, 2022 •

edited

Loading

jmcvey3 Jan 5, 2023 •

edited

Loading

jmcvey3 Jan 19, 2023 •

edited

Loading

jmcvey3 Jan 19, 2023 •

edited

Loading