Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release v1.3.0 #343

Merged
merged 121 commits into from
Dec 27, 2024
Merged
Show file tree
Hide file tree
Changes from 109 commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
72cf054
Logs-to-excel accepts multiple keys and includes folder logs
Shettland Oct 24, 2024
ef7b207
Logs-to-excel accepts multiple keys and includes folder logs. Linting
Shettland Oct 24, 2024
481f4c4
Updated CHANGELOG.md
Shettland Oct 24, 2024
7ac93bd
Better logging messages for duplicate sampleids
Shettland Oct 25, 2024
2846184
Now can provide samples files path when no samples_data is provided.
Shettland Oct 25, 2024
68f9169
Included files-folder option for read-lab-metadata
Shettland Oct 25, 2024
b76b4ba
Better logging. Stop if no samples files found
Shettland Oct 25, 2024
5d37b3a
Better logging messages for duplicate sampleids. Linting
Shettland Oct 25, 2024
72ac4df
Included files-folder arg for read-lab-metadata.linting
Shettland Oct 25, 2024
3841239
Included files-folder arg for read-lab-metadata.linting2
Shettland Oct 25, 2024
58aa92f
Updated CHANGELOG.md
Shettland Oct 25, 2024
f61d4cf
Changed sample_id verification order
Shettland Oct 25, 2024
b7c28e2
Included new folder_names argument to specify folders
Shettland Oct 31, 2024
42590ed
Included a string_to_date() to extract date from a string
Shettland Oct 31, 2024
7151011
Included datetime module import
Shettland Oct 31, 2024
b6ddf52
Small logging improvements
Shettland Oct 31, 2024
0980b4f
Better handling of missing config params
Shettland Oct 31, 2024
bf36ad0
Included group_by_fields list in pipeline_manager prop
Shettland Oct 31, 2024
5400ea8
fixed docstring in split_data_by_key()
Shettland Oct 31, 2024
6381072
Now split by the given fields in config and process list of input fol…
Shettland Oct 31, 2024
8e23b84
Now string_to_date() searches iteratively
Shettland Oct 31, 2024
b9f580d
new argument folder_names. Linting
Shettland Oct 31, 2024
71bac10
Now string_to_date() searches iteratively. Added _
Shettland Oct 31, 2024
58d431d
Updated CHANGELOG.md
Shettland Nov 4, 2024
63cf2ab
Mail sender DRAFT
OPSergio Oct 11, 2024
818d37a
added jinja2 to work with the template.
OPSergio Oct 22, 2024
1bb1dcb
Modified logic to work with the send-mail module.
OPSergio Oct 22, 2024
d92a89a
Created jinja2 template to send validation report.
OPSergio Oct 22, 2024
1f7d6ba
Added information to configuration.json for mail module functionality…
OPSergio Oct 22, 2024
84296ae
Pass Black format
OPSergio Oct 22, 2024
c11065d
Remove load_config_json method from EmailSender and use read_json_fil…
OPSergio Oct 30, 2024
2d1c919
Remove load_credentials method from EmailSender and use read_yml_file…
OPSergio Oct 30, 2024
e13e676
Catch only JSON decode and permission errors in get_invalid_count for…
OPSergio Oct 30, 2024
4424a80
Use read_json_file from utils to load validate_data in EmailSender __…
OPSergio Oct 30, 2024
39605a5
Use read_json_file in get_institution_info to load JSON data
OPSergio Oct 30, 2024
9c69ec2
Add template and credential paths as attributes to reduce repetition
OPSergio Oct 30, 2024
e69c814
Expand path to read_yml_file method
OPSergio Oct 30, 2024
8f44ae7
Enhance error clarity by raising exceptions instead of returning None
OPSergio Oct 30, 2024
98dad26
Replaced direct JSON loading with ConfigJson class in send_mail
OPSergio Oct 30, 2024
af93fee
Apply PEP 8 with autopep8
OPSergio Oct 30, 2024
3b05c66
Make validate_file optional outside CLI, process validate_data in __m…
OPSergio Oct 31, 2024
985aa41
Converted get_invalid_count to a static method in log_summary.py
OPSergio Oct 31, 2024
b4cead1
Update CHANGELOG.md and apply Black formatting
OPSergio Oct 31, 2024
84fb36f
Refactor: Simplified handling of instituion info
OPSergio Nov 11, 2024
1906764
Apply Black formatting
OPSergio Nov 11, 2024
814d5c2
Merge pull request #328 from OPSergio/develop
OPSergio Nov 13, 2024
459364a
Add SQK-RBK114-96 to library_preparation_kit schema
OPSergio Nov 13, 2024
0938898
Update CHANGELOG.md
OPSergio Nov 13, 2024
1c976e8
Corrected the Submitting_institution and added Hospital General de To…
OPSergio Nov 14, 2024
396af9a
Updated CHANGELOG.md
OPSergio Nov 14, 2024
377f177
Added Tomelloso to geo_loc_cities.json
OPSergio Nov 14, 2024
103ba38
Fixed linting was still waiting for .py files
Shettland Nov 14, 2024
82eab9e
Updated changelog
Shettland Nov 14, 2024
388bafb
Now files-folder arg works with relative paths
Shettland Nov 26, 2024
a420eeb
Homogeneized style of report global report sheet
Shettland Nov 26, 2024
7928eba
Included a try-except for every module to catch unexpected errors
Shettland Nov 26, 2024
0e0a7cd
Now check-gzip-integrity() catches any exception
Shettland Nov 26, 2024
77ed96a
Updating version for release 1.3.0
Shettland Nov 26, 2024
c6782c2
Included whole traceback when handling module errors
Shettland Nov 26, 2024
7bc5fa6
Updated CHANGELOG for 1.3.0 release
Shettland Nov 26, 2024
b15bc45
Included whole traceback when handling module errors. Linting
Shettland Nov 26, 2024
183a511
added enums to collecting institutions
svarona Nov 22, 2024
bbdc83e
added nine new originating laboratories
svarona Nov 22, 2024
b4ce1bd
fixed typo
svarona Nov 22, 2024
55906e7
Removed not cities
svarona Nov 26, 2024
ddf3430
Removed duplicated by accent
svarona Nov 26, 2024
5088560
Removed duplicated by different names
svarona Nov 26, 2024
b3405dd
Added new labs
svarona Nov 26, 2024
2a43ad4
Fixed some laboratory names based on REGCESS, removed duplicates and …
svarona Nov 26, 2024
b92ae3a
Fixed typos
svarona Nov 26, 2024
f5aa0b8
Kept Vitoria un cities json
svarona Nov 26, 2024
b28093c
Fixed enums based on REGCESS
svarona Nov 26, 2024
68c5bb0
Updated changelog
svarona Nov 26, 2024
cc810be
Fixed folder_names arg
Shettland Nov 27, 2024
97181dc
Now latest_date from folder-list is parsed without time
Shettland Nov 27, 2024
af1e718
- Updated `__main__.py` to handle multiple email addresses extracted …
OPSergio Dec 4, 2024
07b2d29
Replaced temporary path with path to Relecov contact directory.
OPSergio Dec 4, 2024
be21a56
Added template for successful validation
OPSergio Dec 5, 2024
53cf75b
Added template for validation with errors
OPSergio Dec 5, 2024
2f4cc2f
Modified module logic to integrate template choice
OPSergio Dec 5, 2024
a465622
Reformatted using black
OPSergio Dec 5, 2024
bd093a1
Updated CHANGELOG.md
OPSergio Dec 5, 2024
a64a92d
Linting
OPSergio Dec 5, 2024
f4dd9e3
Merge pull request #349 from OPSergio/develop
OPSergio Dec 9, 2024
b1fc99c
make check_gzip_integrity before
svarona Dec 11, 2024
9576227
remove corrupted before checking successful_files
svarona Dec 11, 2024
731a759
remove instead of moving after checking integrity
svarona Dec 11, 2024
83a0b36
fixed black
svarona Dec 11, 2024
dc416ce
added version to log everytime tools are executed
svarona Dec 12, 2024
a863183
added a sample with corrupted gzip for testing
svarona Dec 12, 2024
36ebe81
Now custom sheet name can be included
Shettland Dec 17, 2024
d808e32
Now default sheet name is extracted from config
Shettland Dec 17, 2024
875be25
Now default sheet name is extracted from config.
Shettland Dec 17, 2024
0860f32
Included arg excel_sheet in validate module
Shettland Dec 17, 2024
1f3556f
updated CHANGELOG.md
Shettland Dec 17, 2024
aa7922f
Added param to pass template path
svarona Dec 20, 2024
8a13589
Added email password that was being unused
svarona Dec 20, 2024
390cd20
explained help better
svarona Dec 20, 2024
d635a41
added samples batch
svarona Dec 20, 2024
cefb2d7
added batch and minnor fix
svarona Dec 20, 2024
c780373
added invalid samples explanation
svarona Dec 20, 2024
1d1b460
added additional notes
svarona Dec 20, 2024
d13fae7
minnor fixes
svarona Dec 20, 2024
5440bfd
Removed email subject from body
svarona Dec 20, 2024
5ca0c0b
Updated changelog
svarona Dec 20, 2024
9e31c3f
fixed black
svarona Dec 20, 2024
5f8aa06
Removed log error
svarona Dec 20, 2024
418f3aa
fixed sugestion
svarona Dec 20, 2024
b3232f8
fixed black
svarona Dec 20, 2024
77735ff
Inclusion of a national hospital catalogue
OPSergio Dec 9, 2024
5b06f13
Added collecting_institutions_enums.json as dictionary for enums in s…
OPSergio Dec 9, 2024
90e14c6
Added entries and including information about the submitting institution
OPSergio Dec 9, 2024
e974135
Renamed laboratory_address.json as copy for comparison (can be deleted).
OPSergio Dec 9, 2024
2329f01
Correction for the use of the include in the schema.
OPSergio Dec 12, 2024
8784202
Corrected path to collecting_institution_enums.json
OPSergio Dec 12, 2024
714eeef
Included terms directly in the schema and removed unnecessary documents.
OPSergio Dec 12, 2024
f683c1a
Removed backslashes
OPSergio Dec 23, 2024
a763e14
Fixed names
OPSergio Dec 23, 2024
b878f45
Added cities to geo_loc_cities.json
OPSergio Dec 23, 2024
3436a13
Merge pull request #352 from OPSergio/develop
OPSergio Dec 23, 2024
cf8c587
Updated release date
svarona Dec 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/workflows/python_lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,10 @@ name: python_lint

on:
push:
paths:
- '**.py'
branches: "**"
pull_request:
paths:
- '**.py'
types: [opened, reopened, synchronize, closed]
branches: "**"

jobs:
flake8_py3:
Expand All @@ -19,6 +18,8 @@ jobs:
architecture: x64
- name: Checkout PyTorch
uses: actions/checkout@master
with:
fetch-depth: 0
- name: Install flake8
run: pip install flake8
- name: Check for Python file changes
Expand All @@ -40,6 +41,8 @@ jobs:
steps:
- name: Setup
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Install black in jupyter
run: pip install black[jupyter]
- name: Check for Python file changes
Expand Down
76 changes: 73 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,75 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.2.0] - 2024-10-11 : https://github.com/BU-ISCIII/relecov-tools/releases/tag/1.2.0
## [1.X.0] - 202X-XX-XX : https://github.com/BU-ISCIII/relecov-tools/releases/tag/

### Credits

Code contributions to the release:

### Modules

#### Added enhancements

#### Fixes

#### Changed

#### Removed

### Requirements

## [1.3.0] - 2024-11-26 : https://github.com/BU-ISCIII/relecov-tools/releases/tag/v1.3.0
svarona marked this conversation as resolved.
Show resolved Hide resolved

### Credits

Code contributions to the release:

- [Pablo Mata](https://github.com/Shettland)
- [Sergio Olmos](https://github.com/OPSergio)
- [Sarai Varona](https://github.com/svarona)

### Modules

- Included files-folder option for read-lab-metadata when no samples_data.json is provided [#330](https://github.com/BU-ISCIII/relecov-tools/pull/330)
- Included folder_names multiple arg for pipeline_manager to specify names of folders to process [#331](https://github.com/BU-ISCIII/relecov-tools/pull/331)
- Include send-mail. Automated email notification module to generate and send validation reports. [#328](https://github.com/BU-ISCIII/relecov-tools/pull/328)

#### Added enhancements

- Now logs-to-excel can handle logs with multiple keys and includes folder logs [#329](https://github.com/BU-ISCIII/relecov-tools/pull/329)
- Improved logging messages for duplicated sample IDs in read-lab and download modules [#330](https://github.com/BU-ISCIII/relecov-tools/pull/330)
- Included a new method string_to_date() in utils to search for a date pattern within a string [#331](https://github.com/BU-ISCIII/relecov-tools/pull/331)
- Integrated jinja2 for template rendering in the mail module. [#328](https://github.com/BU-ISCIII/relecov-tools/pull/328)
- Configurations for the mail module added to configuration.json [#328](https://github.com/BU-ISCIII/relecov-tools/pull/328)
- Added static method get_invalid_count in log_summary.py [#328](https://github.com/BU-ISCIII/relecov-tools/pull/328)
- Included a try-except for every module to catch unexpected errors in __main__.py [#339](https://github.com/BU-ISCIII/relecov-tools/pull/339)
- Added, removed and renamed collecting institutions and their cities [#340](https://github.com/BU-ISCIII/relecov-tools/pull/340)
- Updated contact directory to integrate additional institution data. [#349](https://github.com/BU-ISCIII/relecov-tools/pull/349)
- Added support for multiple recipients in the email_receiver field. [#349](https://github.com/BU-ISCIII/relecov-tools/pull/349)
- Introduced a new Jinja template for successful and error validation reports. [#349](https://github.com/BU-ISCIII/relecov-tools/pull/349)
- Modified the module logic to dynamically select and render email templates based on user input. [#349](https://github.com/BU-ISCIII/relecov-tools/pull/349)
- Enhanced email formatting and added a default CC to [email protected]. [#349](https://github.com/BU-ISCIII/relecov-tools/pull/349)
- Validate module now takes an optional argument to select the name of the sheet to check in excel file [#357](https://github.com/BU-ISCIII/relecov-tools/pull/357)
- Fixed email module [#361](https://github.com/BU-ISCIII/relecov-tools/pull/361)

#### Fixes

- Fixed python linting workflow was still waiting for .py files[#335](https://github.com/BU-ISCIII/relecov-tools/pull/335)
- Now files-folder arg works with relative paths in read-lab-metadata [#339](https://github.com/BU-ISCIII/relecov-tools/pull/339)
- Now check-gzip-integrity() catches any exception in utils.py as it only needs to return True when file can be decompressed [#339](https://github.com/BU-ISCIII/relecov-tools/pull/339)
- Now validate modules does not crash when no METADATA_LAB sheet is found. [#357](https://github.com/BU-ISCIII/relecov-tools/pull/357)

#### Changed

- Pipeline-manager fields_to_split is now in configuration.json to group samples by those fields [#331](https://github.com/BU-ISCIII/relecov-tools/pull/331)
- Homogeneized style of report global report sheet in logs-excel [#339](https://github.com/BU-ISCIII/relecov-tools/pull/339)

#### Removed

### Requirements

## [1.2.0] - 2024-10-11 : https://github.com/BU-ISCIII/relecov-tools/releases/tag/v1.2.0

### Credits

Expand Down Expand Up @@ -35,7 +103,9 @@ Code contributions to the release:
- When no samples_data.json is given, read-lab-metadata now creates a new one [#324](https://github.com/BU-ISCIII/relecov-tools/pull/324)
- Handling for missing sample ids in read-lab-metadata [#324](https://github.com/BU-ISCIII/relecov-tools/pull/324)
- Better logging for download, read-lab-metadata and wrapper [#324](https://github.com/BU-ISCIII/relecov-tools/pull/324)

- Add SQK-RBK114-96 to library_preparation_kit schema [#333](https://github.com/BU-ISCIII/relecov-tools/pull/333)
- Corrected the Submitting_institution for Hospital de Valdepeñas, Centro de Salud Altagracia and Hospital General de Santa Bárbara.[#334](https://github.com/BU-ISCIII/relecov-tools/pull/334)
- Added Hospital General de Tomelloso. [#334](https://github.com/BU-ISCIII/relecov-tools/pull/334)
#### Fixes

- Fixed wrong city name in relecov_tools/conf/laboratory_address.json [#320](https://github.com/BU-ISCIII/relecov-tools/pull/320)
Expand Down Expand Up @@ -64,7 +134,7 @@ Code contributions to the release:

### Requirements

## [1.1.0] - 2024-09-13 : https://github.com/BU-ISCIII/relecov-tools/releases/tag/1.1.0
## [1.1.0] - 2024-09-13 : https://github.com/BU-ISCIII/relecov-tools/releases/tag/v1.1.0

### Credits

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ $ relecov-tools --help
\ \ / |__ / |__ | |___ | | | \ /
/ / \ | \ | | | | | | \ /
/ |--| | \ |___ |___ |___ |___ |___| \/
RELECOV-tools version 1.2.0
RELECOV-tools version 1.3.0
Usage: relecov-tools [OPTIONS] COMMAND [ARGS]...

Options:
Expand Down
Loading
Loading