Skip to content

Commit

Permalink
Merge/add transient fixed conflicts (#109)
Browse files Browse the repository at this point in the history
  • Loading branch information
simedroniraluca authored and beatfactor committed May 24, 2024
1 parent 7d9f6d2 commit 459385b
Show file tree
Hide file tree
Showing 21 changed files with 1,122 additions and 553 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/packit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
fetch-depth: 0

- name: Set up Python
uses: actions/[email protected].0
uses: actions/[email protected].1
with:
python-version: 3.9

Expand Down Expand Up @@ -52,7 +52,7 @@ jobs:
needs: build-artifact
runs-on: ubuntu-20.04
steps:
- uses: actions/[email protected].0
- uses: actions/[email protected].1
name: Install Python
with:
python-version: 3.9
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pypi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
fetch-depth: 0

- name: Set up Python
uses: actions/[email protected].0
uses: actions/[email protected].1
with:
python-version: 3.9

Expand Down Expand Up @@ -56,7 +56,7 @@ jobs:
needs: build-artifact
runs-on: ubuntu-20.04
steps:
- uses: actions/[email protected].0
- uses: actions/[email protected].1
name: Install Python
with:
python-version: 3.9
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/windows.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
# Check data endpoint
curl http://localhost:8080/data/
- name: Setup Python
uses: actions/[email protected].0
uses: actions/[email protected].1
with:
python-version: ${{ matrix.python-version }}
architecture: x64
Expand Down
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ exclude: |
)
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
rev: v4.5.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
Expand All @@ -24,7 +24,7 @@ repos:
args: ["--profile", "black", "--filter-files"]

- repo: https://github.com/psf/black
rev: 23.9.1
rev: 23.10.1
hooks:
- id: black

Expand All @@ -34,7 +34,7 @@ repos:
- id: flake8

- repo: https://github.com/codespell-project/codespell
rev: v2.2.5
rev: v2.2.6
hooks:
- id: codespell
# Checks spelling in `docs/source` and `echopype` dirs ONLY
Expand Down
4 changes: 2 additions & 2 deletions docs/source/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ This diagram depicts the complete workflow we use in the source GitHub repositor
- ``doc patch``: Updates to the documentation that refer to the current ``echopype``
release can be pushed out immediately to the
`echopype documentation site <https://echopype.readthedocs.io>`_
by contibuting patches (PRs) to the ``stable`` branch. See `Documentation development`_
by contributing patches (PRs) to the ``stable`` branch. See `Documentation development`_
below for more details.
- ``code patch``: Code development is carried out as patches (PRs) to the ``dev``
branch; changes in the documentation corresponding to changes in the code can be
Expand Down Expand Up @@ -231,7 +231,7 @@ To view the HTML files generated by Jupyter Book, open the
``docs/_build/html/index.html`` in your browser.

Jupyter Book `configurations <https://jupyterbook.org/en/stable/customize/config.html>`_ can be found in the ``docs/source/_config.yml`` file.
The `table of contents <https://jupyterbook.org/en/stable/structure/toc.html>`_ arragements for the sidebar can be found in ``docs/source/_toc.yml`` file.
The `table of contents <https://jupyterbook.org/en/stable/structure/toc.html>`_ arrangements for the sidebar can be found in ``docs/source/_toc.yml`` file.

When ready to commit your changes, please pull request your changes to the `stable` branch. Once the PR is submitted, the `pre-commit` CI will run for basic spelling and formatting check (See the `pre-commit hooks section <contributing.html#pre-commit-hooks>`_ for more details). Any changes from the `pre-commit` check have to be pulled to your branch (via `git pull`) before your push further commits. You will also be able to view the newly built doc in the PR via the "docs/readthedocs.org:echopype" entry shown below.

Expand Down
4 changes: 2 additions & 2 deletions docs/source/data-proc.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Data processing

Echopype data processing funcionalities are structured into different subpackages with expandability and a series of [data processing levels](processing-levels) in mind. Once the data is converted from the raw instrument data files to standardized [`EchoData` objects](data-format:echodata-object) (or stored in `.zarr` or `.nc` format) and calibrated, the core input and output of most subsequent functions are generic [xarray `Datasets`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html). This design allows new processing functions be easily added without needing to understand specialized objects, other than functions needing access of data stored only in the raw-converted `EchoData` objects.
Echopype data processing functionalities are structured into different subpackages with expandability and a series of [data processing levels](processing-levels) in mind. Once the data is converted from the raw instrument data files to standardized [`EchoData` objects](data-format:echodata-object) (or stored in `.zarr` or `.nc` format) and calibrated, the core input and output of most subsequent functions are generic [xarray `Datasets`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html). This design allows new processing functions be easily added without needing to understand specialized objects, other than functions needing access of data stored only in the raw-converted `EchoData` objects.

The section [**Data processing functionalities**](data-proc:functions) provides information for current processing functions and their usage.

Expand All @@ -9,6 +9,6 @@ The section [**Additional information for processed data**](data-proc:additional
(data-proc:format)=
## Format of processed data

Once raw data (represented by the `EchoData` objects) are calibrated (via [`compute_Sv`](echopype.calibrate.compute_Sv)), the calibrated data and the outputs of all subsequent [processing functions](data-process:funcionalities) are generic [xarray Datasets](https://docs.xarray.dev/en/stable/user-guide/data-structures.html#dataset).
Once raw data (represented by the `EchoData` objects) are calibrated (via [`compute_Sv`](echopype.calibrate.compute_Sv)), the calibrated data and the outputs of all subsequent [processing functions](data-process:functionalities) are generic [xarray Datasets](https://docs.xarray.dev/en/stable/user-guide/data-structures.html#dataset).
We currently do not follow any specific conventions for processed data, but we retain provenance information in the dataset, including the [data processing levels](./processing-levels.md).
However, whether and how data variables used in the processing will be stored remain to be determined.
25 changes: 11 additions & 14 deletions echopype/clean/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,21 @@
import pathlib
from typing import Union

import xarray as xr
from pandas import Index
import pathlib
from typing import Union

import xarray as xr
from pandas import Index

from ..utils.io import get_dataset
from ..utils.misc import frequency_nominal_to_channel
from ..utils.io import get_dataset
from ..utils.misc import frequency_nominal_to_channel
from ..utils.prov import add_processing_level, echopype_prov_attrs, insert_input_processing_level
from . import impulse_noise, signal_attenuation, transient_noise
from . import impulse_noise, signal_attenuation, transient_noise
from .noise_est import NoiseEst


Expand Down Expand Up @@ -92,7 +100,7 @@ def get_transient_noise_mask(
method: str = "ryan",
) -> xr.DataArray:
"""
Create a mask based on the identified signal attenuations of Sv values at 38KHz.
Create a transient noise mask.
This method is based on:
Ryan et al. (2015) ‘Reducing bias due to noise and attenuation in
open-ocean echo integration data’, ICES Journal of Marine Science,
Expand Down Expand Up @@ -166,7 +174,6 @@ def get_impulse_noise_mask(
Can contain the following:
thr: Union[Tuple[float, float], int, float]
User-defined threshold value (dB) (ryan and ryan iterable)
or a 2-element tuple with the range of threshold values (wang).
m: Optional[Union[int, float]] = None,
Vertical binning length (in number of samples or range)
(ryan and ryan iterable).
Expand All @@ -175,17 +182,8 @@ def get_impulse_noise_mask(
Number of pings either side for comparisons (ryan),
or a 2-element tuple specifying the range (ryan iterable).
Defaults to None.
erode: Optional[List[Tuple[int, int]]] = None,
Window size for each erosion cycle (wang).
Defaults to None.
dilate: Optional[List[Tuple[int, int]]] = None,
Window size for each dilation cycle (wang).
Defaults to None.
median: Optional[List[Tuple[int, int]]] = None,
Window size for each median filter cycle (wang).
Defaults to None.
method: str, optional
The method (ryan, ryan_iterable or wang) used to mask impulse noise.
The method (ryan, ryan_iterable) used to mask impulse noise.
Defaults to 'ryan'.
Returns
Expand All @@ -200,7 +198,7 @@ def get_impulse_noise_mask(
mask_map = {
"ryan": impulse_noise._ryan,
"ryan_iterable": impulse_noise._ryan_iterable,
"wang": impulse_noise._wang,
# "wang": impulse_noise._wang,
}
if method not in mask_map.keys():
raise ValueError(f"Unsupported method: {method}")
Expand Down Expand Up @@ -449,4 +447,3 @@ def get_attenuation_mask_multichannel(
mask_list.append(mask)
mask = create_multichannel_mask(mask_list, channel_list)
return mask

Loading

0 comments on commit 459385b

Please sign in to comment.