Merge/add seabed mask (#86)

* Initial implementation of the seabed masking.

* Code for seabed masking

* Changed dilation source package from cv2 to skimage.

* some refactoring and add unit test

* Update echopype/mask/mask_seabed.py

Co-authored-by: Raluca Simedroni <[email protected]>

* Update echopype/tests/mask/test_mask_seabed.py

Co-authored-by: Raluca Simedroni <[email protected]>

* Re-added mask_transformation, with only the necessary functions kept

* Changed the tests to be able to autodownload its files - must check that I have the correct one though since the test isn't passing

* Cleaned up mask_transformation

* Copyright notices

Co-authored-by: Raluca Simedroni <[email protected]>

* Update echopype/mask/mask_seabed.py

Co-authored-by: Raluca Simedroni <[email protected]>

* Update echopype/mask/mask_seabed.py

Co-authored-by: Raluca Simedroni <[email protected]>

* Update incorrect file name

* Mask transformation consistency

* Mask transformation consistency

* Removing pytest req

* Refactoring

* Fix twod test

* Refactoring

* Test fixes

* Fixing merge compatibility

* Added dataset generation for datasets enriched with beam angle and location data

* Added dataset generation for datasets enriched with beam angle and location data

* Added channel selection.

* Modified Blackwell methods to use the physical beam angle rather than the raw EK60 beam angle - doing this using a correction on the angle itself (rather than modifying the constants) since the changes are nonlinear so if I want a linear transformation, I need to apply it on the angle data itself - angle * 22 * 128 / 180, which gets it close to how a decently calibrated EK60 instrument as used by Blackwell et al stores its raw angle data.

* Changed parser in `parse_azfp.py` to maintain consistency over other parsers. (OSOceanAcoustics#1135)

* replaced usage of xml.dom.minidom with xml.etree.ElementTree

* modified azfp parser and parsed all parameters

* refactored the parse_azfp code and added new tests

* small tweak in test, rename conversion function to _camel_to_snake

* updated parameter names

* Replace camel-to-snake conversion funcitonality (#1)

* Create utils camel-to-snake-case function in new misc.py, and use it in ek_raw_parsers

* Replace AZFP camel-to-snake function with new utils/misc.py function

* fixed minor error in code

* minor change in test_convert_azfp.py

* fixed failing tests

* fixed failing tests related to attribute name

* fixed minor bug

---------

Co-authored-by: Wu-Jung Lee <[email protected]>
Co-authored-by: Emilio Mayorga <[email protected]>

* Import fix

* ci: added support for running individual test files (OSOceanAcoustics#1166)

Added a support for testing individual test files also instead of having support for testing only subpackages.

* refactor(convert): refactor and cleanup parsed2zarr (OSOceanAcoustics#1070)

* Bump pypa/gh-action-pypi-publish from 1.6.4 to 1.8.1 (OSOceanAcoustics#999)

Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.6.4 to 1.8.1.
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.6.4...v1.8.1)

---
updated-dependencies:
- dependency-name: pypa/gh-action-pypi-publish
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/cache from 3.2.5 to 3.3.1 (OSOceanAcoustics#982)

Bumps [actions/cache](https://github.com/actions/cache) from 3.2.5 to 3.3.1.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v3.2.5...v3.3.1)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (OSOceanAcoustics#1022)

updates:
- [github.com/psf/black: 23.1.0 → 23.3.0](psf/black@23.1.0...23.3.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Bump pypa/gh-action-pypi-publish from 1.8.1 to 1.8.5 (OSOceanAcoustics#1021)

Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.8.1 to 1.8.5.
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.8.1...v1.8.5)

---
updated-dependencies:
- dependency-name: pypa/gh-action-pypi-publish
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/setup-python from 4.5.0 to 4.6.0 (OSOceanAcoustics#1036)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.5.0 to 4.6.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v4.5.0...v4.6.0)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pypa/gh-action-pypi-publish from 1.8.5 to 1.8.6 (OSOceanAcoustics#1041)

Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.8.5 to 1.8.6.
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.8.5...v1.8.6)

---
updated-dependencies:
- dependency-name: pypa/gh-action-pypi-publish
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump mamba-org/provision-with-micromamba from 15 to 16 (OSOceanAcoustics#1048)

Bumps [mamba-org/provision-with-micromamba](https://github.com/mamba-org/provision-with-micromamba) from 15 to 16.
- [Release notes](https://github.com/mamba-org/provision-with-micromamba/releases)
- [Commits](mamba-org/provision-with-micromamba@v15...v16)

---
updated-dependencies:
- dependency-name: mamba-org/provision-with-micromamba
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/setup-python from 4.6.0 to 4.6.1 (OSOceanAcoustics#1052)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.6.0 to 4.6.1.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v4.6.0...v4.6.1)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Extract out whether_write_to_zarr and remove use_swap arg

* Initial add of destination_path

* Start using destination path and storage options, modify cleanup

* Update docstrings, and add some checks

* Add sonar model attribute to parser object

* Pass sonar_model to parser and fix bug

* Move dataarrays creation to parsed2zarr

* Remove atexit registration

* Add default DEFAULT_ZARR_TEMP_DIR global var

* Initial test for parsed2zarr components

* Add explicit no swap for conversion test

* fix: add parsed2zarr_obj to self to fix bug

* fix: uncomment zarr store close to close the store

* fix: remove unneeded typing

* fix: no need to close fsmap zarr store anymore, removing code

* feat: write tx datagram to zarr in  during p2z

* feat: add  property for ek80 p2z

* feat: set  from swap array

* fix: missing transmit data when no swap

* fix: only delete col when it exists in p2z_ek80

* test(echodata): add simple P2Z object to utils mock

* fix: Import 'List' typehint

Added import for 'List' that is currently
in use but missing.

* fix: Remove elif for column removal

Changed the if elif to a for if so
it removes both 'power' and 'angle' columns
for RAW4 'tx_datagram_df' data.

* fix: Removed dependency to 'more-itertools'

Removed dependency to 'more-itertools' by using similar
method that uses 'numpy.array_split' instead to
evenly split data into desired chunks

* test: Set 'no_swap' for 'test_combine_echodata_combined_append'

Assign 'no_swap' to 'destination_path' during 'open_raw'
to ensure that everything is in memory since the new
parsed to zarr functionality is 'auto' by default.

* feat: Add 'auto' keyword to enable auto 'use_swap'

Changed the way that 'auto' determination of 'use_swap'
by specifying an 'auto' keyword, rather than by default.
Now defaulting back to 'no_swap' for empty 'destination_path'.

* revert: Removed 'no_swap' in 'test_combine_echodata_combined_append'

On OSOceanAcoustics@b6b79fa
'no_swap' was set, however because now the default is 'no_swap'
this shouldn't be needed!

* fix: Use convention yaml for 'backscatter_x'

Sets 'long_name' attributes for 'backscatter_r' and
'backscatter_i' from the convention yaml for p2z
outputs. Additionally, units changed from 'V' to 'dB' to
sync up with the "no_swap" counterpart.

Old tests for 'test_direct_to_zarr_integration' has been activated
again to ensure equivalency b/w the two methods.

* test: Added P2Z Arrays tests

Added testing for underlying methods that
gets called during a 'datagram_to_zarr'
call to ensure that zarr arrays actually gets
created.

Ref: OSOceanAcoustics#777

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix(commongrid):  improve 'compute_MVBS' using flox [all tests ci] (OSOceanAcoustics#1124)

* chore(deps): add flox dependency >=0.7.2

* fix(commongrid): fixes 'compute_MVBS' so it can work better and scale

Under the hood, binning along ping time and echo range now uses flox.
This allows for scalability and more community-maintained.

* docs: add small code comment

* refactor: change how ping_time index is retrieved

* refactor: remove for loop for channel

* test(mvbs): add mock Sv datasets and tests for dims (#2)

Note that @leewujung also changed mean to nanmean for skipping NaNs in each bin.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: change dask to numpy

Changed the use of dask for log10 to numpy instead since numpy can also handle dask array inputs properly.

* feat: Add method argument

Added 'method' argument to 'get_MVBS_along_channels'
and also expose additional keyword arguments control for
flox.

* fix(commongrid): Fixed to include lat lon

Fixed 'compute_MVBS' function to now
include latitude and longitude if the variables
exists in the Sv dataset.

Additionally, the flox method and keyword arguments
are now exposed within the 'compute_MVBS' function.

Ref: Issue OSOceanAcoustics#1002

* refactor: Set defaults to recommended

After some investigation, @lsetiawan
concluded that at this time the
method 'map-reduce', engine 'numpy',
and reindex True works the best,
so this is now set as default.

Also, getting echo range maximum is
through direct data slicing rather than
computation.

* feat(commongrid): Add 'range_var' argument to 'compute_MVBS'

Added a new argument 'range_var' so that user can set
the range variable to perform binning with.
There are 2 options of 'echo_range' and 'depth':

- 'echo_range': When this is set, variable 'water_level'
                is now included in the resulting MVBS dataset
- 'depth': A check is in place to ensure that this variable exists
           before moving forward and use this to perform range binning.

Ref: Issue OSOceanAcoustics#1002

* fix: Add missing attributes for lat lon

* test: Update test to use random generator

* fix: Add case for no 'water_level'

Added a case for dataset that doesn't have water level variable.

* test(nasc): Remove 'compute_NASC' import to avoid failure

* fix: Removed assumption on echo range max

Reverted back the echo range max computation
to computing on the fly since there may be some
NaN values.

* test: Extract api test and add markings

Extracted fixtures to conftest.py for commongrid.
Additionally, clean up unused functions and mark
tests b/w unit and integration. Added a new test module
called 'test_api.py' for 'commongrid.api'.

* test: Add latlon test for 'compute_MVBS'

Added a test for Sv dataset that contains latitude
and longitude going through 'compute_MVBS'
to ensure that those variables gets propagated through.

Ref: OSOceanAcoustics#1002

* test: Add small get_MVBS_along_channels test

Added test for 'get_MVBS_along_channels' with either 'depth'
as the 'range_var' or checking for 'has_positions' is True
or False.

* refactor: Integrate suggested changes

Integrated suggested changes from review such as
additional comments in code, fixing some variable names,
and extracting out the lin2log and log2lin functions.

Additionally, now echopype imports pint library
to start having unit checks in the input for compute_MVBS.

* test: Added check for position values

* test: Update range_meter_bin to strings

* test: Added 'compute_MVBS' values test

* Update echopype/tests/utils/test_processinglevels_integration.py

compute_MVBS now should preserve the processing level attributes. So, test for presence rather than absence

* test: Add 'nan' sprinkles

Sprinkled 'nan' values all over 'echo_range' to ensure that
computed values from 'compute_MVBS' doesn't take into account
the 'nan'. Added check for the expected distribution of 'nan'
in the resulting array.

Ref: OSOceanAcoustics#1124 (comment)

* revert: Revert the use of 'pint'

Removed dependency to 'pint' and use simple regex
to ensure that 'range_bin' input is unit 'm'.
Renamed 'range_meter_bin' argument to 'range_bin'.

Ref: OSOceanAcoustics#1124 (comment)

* feat: Allow 'range_bin' to have space

* fix: Apply suggestions from code review

Applied fix for regex not capturing decimal values by @emiliom

Ref: https://github.com/OSOceanAcoustics/echopype/pull/1124/files#r1320422121

Co-authored-by: Emilio Mayorga <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* test: Fix test text for wrong unit

* test: Remove the 'e.g.' part on pytest

Removed the part with '(e.g., '10m')' since
it's messing up pytests regex matching.

* test: Remove remnant for test_ek.py

* refactor: Extract range_bin parsing and add close arg

Extracts out the 'range_bin' string to float into a private function.
Additionally now there's a fine tune argument for bin close edges
so user can specify either close is 'left' or 'right'.
Bins are converted to pandas interval index before passing
into 'get_MVBS_along_channels'.

* refactor: Update arg types to include interval index

Added argument type options for 'range_interval' and 'ping_interval' to
also be interval index.

* test: Update tests to have brute force creation

Changed mock mvbs to be created by doing brute force
rather than hard coding.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* test: Fix brute force mvbs gen

Fixes the generation of expected mvbs with brute force
as well as tweaks to mvbs_along_channel test.

* chore: Clean up old code for doing compute MVBS

Removes old code that perfoms compute_MVBS since
now we've switched over to flox

* chore(pytest): Added custom markers 'unit' and 'integration'

* docs: Update docstring for `compute_MVBS`

Added options for literal arguments

* refactor: Change 'parse_range_bin' to 'parse_x_bin'

Make bin parsing to be more general by making it
to 'parse_x_bin'.

* chore: Update suggested changes

Update some texts from suggested review as discussed.

Ref: OSOceanAcoustics#1124 (review)

---------

Co-authored-by: Wu-Jung Lee <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Emilio Mayorga <[email protected]>

* Refactored seabed generation masks to use parameter dicts and function maps

* Added multichannel mask support to seabed generation

* Fixing an indent issue

* Fixing a build issue

* Fixed a merge issue

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Mihai Boldeanu <[email protected]>
Co-authored-by: Andrei Rusu <[email protected]>
Co-authored-by: Andrei Rusu <[email protected]>
Co-authored-by: Raluca Simedroni <[email protected]>
Co-authored-by: Praneeth Ratna <[email protected]>
Co-authored-by: Wu-Jung Lee <[email protected]>
Co-authored-by: Emilio Mayorga <[email protected]>
Co-authored-by: Don Setiawan <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

echopype/mask/__init__.py

@@ -1,4 +1,3 @@
 from .api import apply_mask, frequency_differencing, get_seabed_mask, shoal_weill
 
 __all__ = ["frequency_differencing", "apply_mask", "get_seabed_mask", "shoal_weill"]
-