Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure interpolation values are within range when sampling contours #278

Merged
merged 7 commits into from
Nov 17, 2023

Conversation

ssolson
Copy link
Contributor

@ssolson ssolson commented Nov 15, 2023

Overview

In #218 @Graham-EGI uncovered a bug in mhkit.wave.contours.samples_contour

What he uncovered was quite simple:

  • Given: Hs and Tp values
  • If: you create an array from the Tp min and max to sample using samples_contour
  • Then: the function could throw ValueError: A value in x_new is above the interpolation range.

Reference Example from #218

from mhkit.wave.contours import samples_contour
import numpy as np

contour = {
        "hm_contour": [ <see #218 for values> ],
        ""period_contour": [ <see #218 for values> ],
]

min_tp = min(contour["period_contour"])
max_tp = max(contour["period_contour"])

# ten samples
period_samples = np.linspace(min_tp, max_tp, 10)

# error thrown about being above the interpolation range
hm_samples = samples_contour(period_samples, np.array(contour["period_contour"]), np.array(contour["hm_contour"]))

The import part to note in the code block that period_samples is defined using max_tp.

The reason this error is thrown is because the sampling is performed by breaking the passed contour into 2 halves. The upper and lower half called w1 and w2 in samples_contour.

image

samples_contour chooses the upper half of the contour:
image

As can be seen in the second plot the upper half ("chosen segment # 2") is not inclusive of the Max Tp value (red circle on right side). This max Tp value was used as the max of the Tp sample range. Therefore the interpolation throws an error as the sample is outside the data included for interpolation.

Solution

To fix this:

  1. The PR adds a check to ensure all samples are within the provided data range.
  2. After choosing the upper half of the contour the PR will add 1 index value to the chosen half if the min/max sample value is outside the range of the chosen half.

@ssolson ssolson added the bug Something isn't working label Nov 15, 2023
@ssolson ssolson self-assigned this Nov 15, 2023
@ssolson ssolson linked an issue Nov 15, 2023 that may be closed by this pull request
@ssolson
Copy link
Contributor Author

ssolson commented Nov 15, 2023

@cmichelenstrofer could you give me a quick sanity check on the approach I took to fix this bug?

@ssolson ssolson marked this pull request as ready for review November 15, 2023 22:40
@ssolson ssolson requested a review from akeeste November 16, 2023 15:54
@ssolson
Copy link
Contributor Author

ssolson commented Nov 16, 2023

@akeeste I am 99% certain what I did is correct and was mostly just checking with Carlos just in case. If you want to give a formal review while we wait I believe this PR is ready.

@cmichelenstrofer
Copy link
Contributor

Caveat: I didn't run it to make sure it works.

But the logic & implementation look good to me. If I understand correctly these are the steps in the code:

  • divide the contour into two pieces
  • select the top portion (higher max wave heights)
  • check if the maximum period is included here, if not increase the index by one
  • similar for the minimum period.

based on how the data is divided the max/minimum if missing should only be one index away.

Copy link
Contributor

@akeeste akeeste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ssolson I proposed a few changes to the logic check and how to catch both the minimum and maximum Te if either is missing. Before merging, I'd like to confirm with Graham's data tomorrow. I'll follow up tomorrow morning

mhkit/wave/contours.py Outdated Show resolved Hide resolved
Copy link
Contributor

@akeeste akeeste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ssolson I had a chance to think through this indexing more and agree that your implementation is correct, except for the minor suggestion on the logic check. After that, let's merge

@ssolson ssolson merged commit 2387a25 into MHKiT-Software:develop Nov 17, 2023
13 checks passed
ssolson added a commit to ssolson/MHKiT-Python that referenced this pull request Nov 21, 2023
akeeste pushed a commit that referenced this pull request Mar 13, 2024
* assert to exception

* Revert "Ensure interpolation values are within range when sampling contours (#278)"

This reverts commit 2387a25.

* black format

* add a dev requirements

* add check for black formatting

* add pre commit

* change API key

* Treat the None as string "None"

* black formatting

* remove unused imports, add some type checks

* no coverage on artifact load

* black formatting

* add scipy to TEST env

* remove no dependencies

* remove debug imports

* revert to original main.yaml

* black

* river io to usgs and d3d

* move d3d tidal into tidal/io

* quick test to call d3d from tidal

* snake_case

* `product` is deprecated as of NumPy 1.25.0

* fix import path

* increase test coverage

* black

* update test filename

* correct name

* wait fox James to respond

* increase mooring graphics coverage

* increase test coverage

* increase coverage

* catch strings passed as TypeError

* Increase coverage on dataTypes

* throw error on strings passed

* black

* remove unused imports

* additional input checks

* test env contours invalid inputs

* uncomment tests

* do not include cached API requests in coverage

* black

* fix & improve type checks

* fix bug on slicing after setting min_bins to 4

* increase test coverage

* black

* black

* remove debug

* black v24.1.0

* fix implementation from #284

* finalize tests for `cpsd_quasisync_1D`

* black

* remove "alt_raw" merge artifact

* remove positive frequencies test

* black formatting

* black

* black

* add back copula tests

* pylint: extreme to multiple files

* lint

* clean up module docstring

* fix too many local args

* linting

* match correct module name

* middle of getting linted versions working

* mler perfect lint

* fix type errors

* lint

* pylint loads 80%

* remove comments

* try this

* set to 7

* 100% peaks

* pylint 10/10 extereme

* should run perfect on extreme

* mhkit/loads/extreme/

* does this fix install issues?

* install package

* install

* remove [] empyty array input

* lint 10/10

* list works; throw error on string data

* entire loads module

* docstring

* 10/10

* type_handling function for checking numeric arrays

* typehints

* typehints
@ssolson ssolson deleted the contours branch April 16, 2024 14:55
@ssolson ssolson mentioned this pull request Apr 24, 2024
@ssolson ssolson mentioned this pull request May 6, 2024
ssolson added a commit that referenced this pull request May 8, 2024
# MHKiT v0.8.0
We're excited to announce the release of MHKiT v0.8.0, which brings a host of new features, enhancements, and bug fixes across various modules, ensuring compatibility with Python 3.10 and 3.11, and introducing full xarray support for more flexible data handling. Significant updates in the Wave and DOLfYN modules improve functionality and extend capabilities.

## Python 3.10 & 3.11 Support
MHKiT now supports python 3.10 and 3.11. Support for 3.12 will follow in the next minor update.
- #240


## Wave Module
### Enhancements:
**Automatic Threshold Calculation for Peaks-Over-Threshold**: We've introduced a new feature that automatically calculates the "best" threshold for identifying significant wave events. This method, originally developed by Neary, V. S., et al. in their 2020 study, has now been translated from Matlab to Python, enhancing our existing peaks-over-threshold functionality.

**Wave Heights Analysis**: A new function, `wave_heights`, has been added to extract the heights of individual waves from a time series. This function uses zero up-crossing analysis to accurately measure wave heights, improving upon our previous methods which only provided the maximum value between up-crossings.

**Enhanced Zero Crossing Analysis**: Building on the above, the zero crossing code previously embedded in `global_peaks` has been isolated into a helper function. This modular approach not only refines the codebase but also supports new functionalities such as calculating wave heights, zero crossing periods, and identifying crests.

### Bug Fixes:
**Contour Sampling Error in Wave Contours**: A bug identified in `mhkit.wave.contours.samples_contour` has been resolved. The issue occurred when period samples defined using the maximum period resulted in values outside the interpolation range of the contour data. This was corrected by ensuring that all sampling points are within the interpolation range and adjusting the contour data selection process accordingly.

- #268 
- #252 
- #278


## Xarray Support
MHKiT functions now fully support the use of xarray for passing and returning data.

- #279 
- #282
- #285
- #302
- #310


## DOLfYN

Thanks to the many user contributions and users of MHKiT the DOLFYN module include a significant number of enhancements and bug fixes. 

### Enhancements:
**Altimeter Support**: Enhanced the Nortek Signature Reader to add capability for reading ADCP dual profile configurations.

**Data Handling Improvements**: Introduced logic to skip messy header data that can accumulate during measurements collected via Nortek software on PCs and Macs.

**Instrument Noise Subtraction**: Added a function to subtract instrument noise from turbulence intensity estimation using RMS calculations, providing results that differ by approximately 1% from the existing standard deviation-based "I" property.

**Improved File Handling**: Updates for RDI files to handle changing "number of cells" and variable "cell sizes," which are now bin-averaged into the largest cell size.

### Bug Fixes:
**Power Spectra Calculation**: Fixed a bug where a given noise value was not being subtracted from the power spectra, and noise was inadvertently added as an input to dissipation rate calculations.

**Improved Header Handling**: Allowed RDI reader to skip junk headers effectively.

**Nortek Reader C Types Update**: Adjusted C types in the Nortek reader to handle below-zero water temperatures and to allow pitch and roll values to go negative.


- #280 
- #289
- #290
- #292
- #293
- #294
- #299


## River & Tidal: D3D
Added limits to `variable_interpolation` and added 3 array input capability to `create_points`
- #271

## Developer Experience
### Black formatting
Black formatting is now enforced on all MHKiT files. This ensures consistent formatting across the MHKiT package.
- #281

### Linting & Type Hints
MHKiT is in the process of enforcing pylint and adding type hints to all functions. Currently this has been achieved and is enforced in the Loads and Power modules.
- #288 
- #296 

### CI/CD
This release introduces significant reduction in testing time for development. This is achieved by reducing the number of tests for pulls against the develop branch and only running hindcast test when changes are made to it. A bug in the hindcast CI was fixed which only ran on changes to the hindcast tests instead of the hindcast module. Additionally the wave and wind hindcast needed to be separated in 2 jobs due to the excessive time taken to run a wind cache. This created a number of follow on PRs around solidifying the logic of these job. A special case for Python 3.8, pip, and Mac OS was added to use homebrew to install NetCDF and HDF5 to get tests to pass.
- #241
- #270
- #306
- #311
- #317
- #318
- #319
- #320
- #324

### Clean Up
MHKiT fixed an implementation error where functions used assert instead of built in errors for type and value checking. Additionally these PRs removed unused files, fixed typos, and created an argument which allows users to run CDIP API calls silently.
- #276
- #272
- #273
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Contour Sampling - interpolation errors
3 participants