Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge/add transient fixed conflicts #109

Merged
merged 75 commits into from
Nov 3, 2023
Merged
Show file tree
Hide file tree
Changes from 74 commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
e4bdb3b
added the feature described in issue 1
mihaiboldeanu Jul 26, 2023
f2d02b9
Moved code for the signal attenuation mask to new file
mihaiboldeanu Aug 1, 2023
a202b43
Resolved merge conflicts by accepting remote versions of files
mihaiboldeanu Aug 1, 2023
a2b7f64
Update mask_attenuated_signal.py
mihaiboldeanu Aug 1, 2023
47605b7
renamed utils
beatfactor Aug 10, 2023
74b90ac
Added unit test
beatfactor Aug 13, 2023
dba7fd2
Test can now automatically download its data
ruxandra-valcu Aug 17, 2023
107f8dd
Update echopype/utils/mask_transformation.py
ruxandra-valcu Aug 17, 2023
fc50d70
Update echopype/utils/mask_transformation.py
ruxandra-valcu Aug 17, 2023
3dca2b5
Update echopype/utils/mask_transformation.py
ruxandra-valcu Aug 17, 2023
de165f8
Update echopype/utils/mask_transformation.py
ruxandra-valcu Aug 17, 2023
37c23f1
Update echopype/utils/mask_transformation.py
ruxandra-valcu Aug 17, 2023
e633378
Update echopype/utils/mask_transformation.py
ruxandra-valcu Aug 17, 2023
fa2388d
Update echopype/utils/mask_transformation.py
ruxandra-valcu Aug 17, 2023
c2e8fc0
Cleaned up mask_transformation
ruxandra-valcu Aug 17, 2023
19924ea
moved files and rerouted importing paths accordingly (#36)
beatfactor Aug 29, 2023
ec60163
Feature/add transient noise mask (#35)
beatfactor Aug 29, 2023
482066c
Added extra tests for the utils/mask_transformation module to increas…
mihaiboldeanu Aug 31, 2023
b755cce
Merge branch 'OSOceanAcoustics:main' into feature/add_transient_noise…
mihaiboldeanu Sep 1, 2023
9d5f688
Reverted test_mask.py reformating
mihaiboldeanu Sep 1, 2023
0ab52ea
Moved impulse and transient noise functions to echopype/clean
mihaiboldeanu Sep 1, 2023
14bc3e7
Fixed import order
mihaiboldeanu Sep 1, 2023
0253649
Fixed missing import in clean/conftest.py
mihaiboldeanu Sep 1, 2023
2087084
Moved content of tests/clean/conftest.py to tests/conftest.py
mihaiboldeanu Sep 4, 2023
909a341
Refactoring
ruxandra-valcu Sep 5, 2023
998d2b2
Running some linters on the code
ruxandra-valcu Sep 6, 2023
94c4ba9
Merge branch 'OSOceanAcoustics:main' into feature/add_transient_noise…
ruxandra-valcu Sep 13, 2023
49d470e
Fix unused imports in conftest
ruxandra-valcu Sep 13, 2023
9b0f7f7
Update echopype/mask/mask_attenuated_signal.py
ruxandra-valcu Aug 17, 2023
47b1a2e
Added channel selection to signal attenuation mask
mihaiboldeanu Sep 14, 2023
465c61e
Merge pull request #43 from OceanStreamIO/feature/add-attenuation-mask
ruxandra-valcu Sep 14, 2023
eb2fdc1
Merge branch 'next' into feature/add_transient_noise_mask
ruxandra-valcu Sep 14, 2023
d11c0e2
Changed parser in `parse_azfp.py` to maintain consistency over other …
praneethratna Sep 18, 2023
ea01132
ci: added support for running individual test files (#1166)
praneethratna Sep 19, 2023
debb7c6
ci: Bump docker/setup-qemu-action from 2 to 3 (#1172)
dependabot[bot] Sep 19, 2023
2c45181
ci: Bump docker/login-action from 2 to 3 (#1175)
dependabot[bot] Sep 19, 2023
f42b722
ci: Bump docker/build-push-action from 4 to 5 (#1173)
dependabot[bot] Sep 19, 2023
7ec7a1c
ci: Bump docker/setup-buildx-action from 2 to 3 (#1174)
dependabot[bot] Sep 19, 2023
b453bba
Delinting
ruxandra-valcu Sep 20, 2023
2e564e9
refactor(convert): refactor and cleanup parsed2zarr (#1070)
lsetiawan Sep 21, 2023
b887be8
fix(commongrid): improve 'compute_MVBS' using flox [all tests ci] (#…
lsetiawan Sep 21, 2023
ce1e614
ci: Fix run-test.py for module runs (#1180)
lsetiawan Sep 22, 2023
b161ddc
Merge branch 'dev' into feature/add_transient_noise_mask
beatfactor Sep 25, 2023
4a6d80f
Impulse noise: modified ryan_iterable to use ryan directly
ruxandra-valcu Sep 26, 2023
2dc1236
Impulse noise: renamed medianf
ruxandra-valcu Sep 26, 2023
4294926
Consolidating functions
ruxandra-valcu Sep 26, 2023
759df07
Impulse noise refactoring: using a parameter dictionary and a functio…
ruxandra-valcu Sep 26, 2023
0b1c8ea
Frequency filtering
ruxandra-valcu Sep 27, 2023
cb0b1e1
Frequency filtering (temp add to transient noise and signal attenuati…
ruxandra-valcu Sep 27, 2023
14767dc
Signal attenuation refactor
ruxandra-valcu Sep 27, 2023
696b03a
Transmission noise refactor
ruxandra-valcu Sep 27, 2023
eb164b9
Impulse noise default params
ruxandra-valcu Sep 27, 2023
f6d7c2e
Multichannel noise masks
ruxandra-valcu Sep 27, 2023
35f258a
Made the noise masks able to handle taking a file path as an input
ruxandra-valcu Sep 28, 2023
2742a4d
Initial mods
ruxandra-valcu Oct 2, 2023
aba7db9
Lin, log, downsample - upsample not yet functional
ruxandra-valcu Oct 4, 2023
6374af6
Temporarily readded skimage to requirements.txt (will remove it later…
ruxandra-valcu Oct 4, 2023
3b15b6d
Added xarray upsample
ruxandra-valcu Oct 4, 2023
69ec5e5
Added xarray ryan/ryan_iterable
ruxandra-valcu Oct 6, 2023
13c2bce
Removed wang method for impulse noise removal. The reason for this is…
ruxandra-valcu Oct 9, 2023
07c170d
ci: Bump actions/setup-python from 4.7.0 to 4.7.1 (#1187)
dependabot[bot] Oct 9, 2023
68ba151
Fix build bug
ruxandra-valcu Oct 10, 2023
593a7e1
[pre-commit.ci] pre-commit autoupdate (#1188)
pre-commit-ci[bot] Oct 12, 2023
bb90309
Bug/transient attenuation mask type (#93)
simedroniraluca Oct 16, 2023
e30f49e
Mask transformation updates
ruxandra-valcu Oct 24, 2023
974da73
xarray transient method (#104)
ruxandra-valcu Oct 31, 2023
080fc92
xarray signal attenuation method (#105)
ruxandra-valcu Oct 31, 2023
0de62b0
[pre-commit.ci] pre-commit autoupdate (#1197)
pre-commit-ci[bot] Oct 31, 2023
25227c1
Merge branch 'OSOceanAcoustics:main' into feature/add_transient_noise…
ruxandra-valcu Nov 2, 2023
7500201
Removed scikit
ruxandra-valcu Nov 2, 2023
235e10b
Added a test for get_dataset
ruxandra-valcu Nov 2, 2023
6f15a62
Documentation modifications
ruxandra-valcu Nov 2, 2023
6d8e57b
Removed a file unused in the current PR
ruxandra-valcu Nov 2, 2023
c7df676
Merge branch 'next-dev' into merge/add_transient
simedroniraluca Nov 3, 2023
fe0b018
Test fixes
ruxandra-valcu Nov 3, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/packit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
fetch-depth: 0

- name: Set up Python
uses: actions/[email protected].0
uses: actions/[email protected].1
with:
python-version: 3.9

Expand Down Expand Up @@ -52,7 +52,7 @@ jobs:
needs: build-artifact
runs-on: ubuntu-20.04
steps:
- uses: actions/[email protected].0
- uses: actions/[email protected].1
name: Install Python
with:
python-version: 3.9
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pypi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
fetch-depth: 0

- name: Set up Python
uses: actions/[email protected].0
uses: actions/[email protected].1
with:
python-version: 3.9

Expand Down Expand Up @@ -56,7 +56,7 @@ jobs:
needs: build-artifact
runs-on: ubuntu-20.04
steps:
- uses: actions/[email protected].0
- uses: actions/[email protected].1
name: Install Python
with:
python-version: 3.9
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/windows.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
# Check data endpoint
curl http://localhost:8080/data/
- name: Setup Python
uses: actions/[email protected].0
uses: actions/[email protected].1
with:
python-version: ${{ matrix.python-version }}
architecture: x64
Expand Down
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ exclude: |
)
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
rev: v4.5.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
Expand All @@ -24,7 +24,7 @@ repos:
args: ["--profile", "black", "--filter-files"]

- repo: https://github.com/psf/black
rev: 23.9.1
rev: 23.10.1
hooks:
- id: black

Expand All @@ -34,7 +34,7 @@ repos:
- id: flake8

- repo: https://github.com/codespell-project/codespell
rev: v2.2.5
rev: v2.2.6
hooks:
- id: codespell
# Checks spelling in `docs/source` and `echopype` dirs ONLY
Expand Down
4 changes: 2 additions & 2 deletions docs/source/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ This diagram depicts the complete workflow we use in the source GitHub repositor
- ``doc patch``: Updates to the documentation that refer to the current ``echopype``
release can be pushed out immediately to the
`echopype documentation site <https://echopype.readthedocs.io>`_
by contibuting patches (PRs) to the ``stable`` branch. See `Documentation development`_
by contributing patches (PRs) to the ``stable`` branch. See `Documentation development`_
below for more details.
- ``code patch``: Code development is carried out as patches (PRs) to the ``dev``
branch; changes in the documentation corresponding to changes in the code can be
Expand Down Expand Up @@ -231,7 +231,7 @@ To view the HTML files generated by Jupyter Book, open the
``docs/_build/html/index.html`` in your browser.

Jupyter Book `configurations <https://jupyterbook.org/en/stable/customize/config.html>`_ can be found in the ``docs/source/_config.yml`` file.
The `table of contents <https://jupyterbook.org/en/stable/structure/toc.html>`_ arragements for the sidebar can be found in ``docs/source/_toc.yml`` file.
The `table of contents <https://jupyterbook.org/en/stable/structure/toc.html>`_ arrangements for the sidebar can be found in ``docs/source/_toc.yml`` file.

When ready to commit your changes, please pull request your changes to the `stable` branch. Once the PR is submitted, the `pre-commit` CI will run for basic spelling and formatting check (See the `pre-commit hooks section <contributing.html#pre-commit-hooks>`_ for more details). Any changes from the `pre-commit` check have to be pulled to your branch (via `git pull`) before your push further commits. You will also be able to view the newly built doc in the PR via the "docs/readthedocs.org:echopype" entry shown below.

Expand Down
4 changes: 2 additions & 2 deletions docs/source/data-proc.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Data processing

Echopype data processing funcionalities are structured into different subpackages with expandability and a series of [data processing levels](processing-levels) in mind. Once the data is converted from the raw instrument data files to standardized [`EchoData` objects](data-format:echodata-object) (or stored in `.zarr` or `.nc` format) and calibrated, the core input and output of most subsequent functions are generic [xarray `Datasets`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html). This design allows new processing functions be easily added without needing to understand specialized objects, other than functions needing access of data stored only in the raw-converted `EchoData` objects.
Echopype data processing functionalities are structured into different subpackages with expandability and a series of [data processing levels](processing-levels) in mind. Once the data is converted from the raw instrument data files to standardized [`EchoData` objects](data-format:echodata-object) (or stored in `.zarr` or `.nc` format) and calibrated, the core input and output of most subsequent functions are generic [xarray `Datasets`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html). This design allows new processing functions be easily added without needing to understand specialized objects, other than functions needing access of data stored only in the raw-converted `EchoData` objects.

The section [**Data processing functionalities**](data-proc:functions) provides information for current processing functions and their usage.

Expand All @@ -9,6 +9,6 @@ The section [**Additional information for processed data**](data-proc:additional
(data-proc:format)=
## Format of processed data

Once raw data (represented by the `EchoData` objects) are calibrated (via [`compute_Sv`](echopype.calibrate.compute_Sv)), the calibrated data and the outputs of all subsequent [processing functions](data-process:funcionalities) are generic [xarray Datasets](https://docs.xarray.dev/en/stable/user-guide/data-structures.html#dataset).
Once raw data (represented by the `EchoData` objects) are calibrated (via [`compute_Sv`](echopype.calibrate.compute_Sv)), the calibrated data and the outputs of all subsequent [processing functions](data-process:functionalities) are generic [xarray Datasets](https://docs.xarray.dev/en/stable/user-guide/data-structures.html#dataset).
We currently do not follow any specific conventions for processed data, but we retain provenance information in the dataset, including the [data processing levels](./processing-levels.md).
However, whether and how data variables used in the processing will be stored remain to be determined.
25 changes: 11 additions & 14 deletions echopype/clean/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,21 @@
import pathlib
from typing import Union

import xarray as xr
from pandas import Index
import pathlib
from typing import Union

import xarray as xr
from pandas import Index

from ..utils.io import get_dataset
from ..utils.misc import frequency_nominal_to_channel
from ..utils.io import get_dataset
from ..utils.misc import frequency_nominal_to_channel
from ..utils.prov import add_processing_level, echopype_prov_attrs, insert_input_processing_level
from . import impulse_noise, signal_attenuation, transient_noise
from . import impulse_noise, signal_attenuation, transient_noise
from .noise_est import NoiseEst


Expand Down Expand Up @@ -92,7 +100,7 @@ def get_transient_noise_mask(
method: str = "ryan",
) -> xr.DataArray:
"""
Create a mask based on the identified signal attenuations of Sv values at 38KHz.
Create a transient noise mask.
This method is based on:
Ryan et al. (2015) ‘Reducing bias due to noise and attenuation in
open-ocean echo integration data’, ICES Journal of Marine Science,
Expand Down Expand Up @@ -166,7 +174,6 @@ def get_impulse_noise_mask(
Can contain the following:
thr: Union[Tuple[float, float], int, float]
User-defined threshold value (dB) (ryan and ryan iterable)
or a 2-element tuple with the range of threshold values (wang).
m: Optional[Union[int, float]] = None,
Vertical binning length (in number of samples or range)
(ryan and ryan iterable).
Expand All @@ -175,17 +182,8 @@ def get_impulse_noise_mask(
Number of pings either side for comparisons (ryan),
or a 2-element tuple specifying the range (ryan iterable).
Defaults to None.
erode: Optional[List[Tuple[int, int]]] = None,
Window size for each erosion cycle (wang).
Defaults to None.
dilate: Optional[List[Tuple[int, int]]] = None,
Window size for each dilation cycle (wang).
Defaults to None.
median: Optional[List[Tuple[int, int]]] = None,
Window size for each median filter cycle (wang).
Defaults to None.
method: str, optional
The method (ryan, ryan_iterable or wang) used to mask impulse noise.
The method (ryan, ryan_iterable) used to mask impulse noise.
Defaults to 'ryan'.

Returns
Expand All @@ -200,7 +198,7 @@ def get_impulse_noise_mask(
mask_map = {
"ryan": impulse_noise._ryan,
"ryan_iterable": impulse_noise._ryan_iterable,
"wang": impulse_noise._wang,
# "wang": impulse_noise._wang,
}
if method not in mask_map.keys():
raise ValueError(f"Unsupported method: {method}")
Expand Down Expand Up @@ -449,4 +447,3 @@ def get_attenuation_mask_multichannel(
mask_list.append(mask)
mask = create_multichannel_mask(mask_list, channel_list)
return mask

Loading
Loading