feat: add option to disable normal hardfilter

[mathiasbio]

Description

Some customers have requested a feature to get better control over the normal filtration in Scout. We can achieve this by transforming the matched normal filter to as soft-filter that can be optionally applied in Scout.

Linked MTP-BALSAMIC story: https://github.com/Clinical-Genomics/MTP-BALSAMIC/issues/1

To enable this feature we need to do two more things:

1. enable the new argument in CG: add soft-filter-normal argument to paired tga analyses cg#4157
2. activate the filtering by matched normal soft-filters in Scout: Enable filtering on matched normal scout#5201

After applying this soft filter argument there will be 2 new filters appearing in the final VCF:

• germline_risk (from TNscope)
• in_normal (from custom bcftools normal af / tumor af fraction filter)

Added

• new argument --soft-filter-normal to disable hard filtration of matched normal filters

Changed

• refactored variant-filter constants and reworked bcftools filters
• renamed high_normal_tumor_af_frac to in_normal

Documentation

• N/A
• Updated Balsamic documentation to reflect the changes as needed for this PR.
  • [balsamic_filters.rst] with new soft-filter normal behaviour

Tests

Feature Tests

• Ensure that all workflows are still filtering properly, with or without the new soft-filter normal option. See google sheet: https://docs.google.com/spreadsheets/d/15oU6JoLUybFwY76iRDOnvnEDuybjDElhu07oB3V6vR0/edit?gid=0#gid=0

• With soft-filter

• TN TGA

• TO TGA

• TN WES

• TO WES

• TN UMI

• TO UMI

• TN WGS

• TO WGS

• Without soft-filter

• TN TGA

• TO TGA

• TN WES

• TO WES

• TN UMI

• TO UMI

• TN WGS

• TO WGS

• Pipeline still works after changing filter name to "in_normal", sanity check

• TN TGA completes successfully

Pipeline Integrity Tests

• Report deliver (generation of the .hk file)
  • N/A
  • Verified
• TGA T/O Workflow
  • N/A
  • Verified
• TGA T/N Workflow
  • N/A
  • Verified
• UMI T/O Workflow
  • N/A
  • Verified
• UMI T/N Workflow
  • N/A
  • Verified
• WGS T/O Workflow
  • N/A
  • Verified
• WGS T/N Workflow
  • N/A
  • Verified
• QC Workflow
  • N/A
  • Verified
• PON Workflow
  • N/A
  • Verified

Clinical Genomics Stockholm

Documentation

• Atlas documentation
  • N/A
  • Updated: [Link]
• Web portal for Clinical Genomics
  • N/A
  • Updated: [Link]

Panel of Normal specific criteria

• The PR includes the addition of a new Panel of Normals
• The samples have been verified to adhere to the sample selection criteria on Atlas PoN creation instructions for Balsamic

User Changes

• N/A
• This PR affects the output files or results.
  • User feedback is considered unnecessary because [Justification].
  • Affected users have been included in the development process and given a chance to provide feedback.

Infrastructure Changes

• Stored files in Housekeeper
  • N/A
  • Updated: [Link]
• CG (CLI and delivered/uploaded files)
  • N/A
  • Updated: add soft-filter-normal argument to paired tga analyses cg#4157
• Servers (configuration files on Hasta)
  • N/A
  • Updated: [Link]
• Scout interface
  • N/A
  • Updated: Enable filtering on matched normal scout#5201

Checklist

Important

Ensure that all checkboxes below are ticked before merging.

For Developers

• PR Description
  • Provided a comprehensive description of the PR.
  • Linked relevant user stories or issues to the PR.
• Documentation
  • Verified and updated documentation if necessary.
• Tests
  • Described and tested the functionality addressed in the PR.
  • Ensured integration of the new code with existing workflows.
  • Confirmed that meaningful unit tests were added for the changes introduced.
  • Checked that the PR has successfully passed all relevant code smells and coverage checks.
• Review
  • Addressed and resolved all the feedback provided during the code review process.
  • Obtained final approval from designated reviewers.

For Reviewers

• Code
  • Code implements the intended features or fixes the reported issue.
  • Code follows the project's coding standards and style guide.
• Documentation
  • Pipeline changes are well-documented in the CHANGELOG and relevant documentation.
• Tests
  • The author provided a description of their manual testing, including consideration of edge cases and boundary
    conditions where applicable, with satisfactory results.
• Review
  • Confirmed that the developer has addressed all the comments during the code review.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.50%. Comparing base (7d529e6) to head (8490657).
Report is 38 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1509      +/-   ##
===========================================
+ Coverage    99.48%   99.50%   +0.02%     
===========================================
  Files           40       40              
  Lines         1932     2036     +104     
===========================================
+ Hits          1922     2026     +104     
  Misses          10       10              

Flag	Coverage Δ
unittests	99.50% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mathiasbio]

This PR isn't finished yet in terms of all the testing and documentation, but I can take some reviews on the code!

[fevac]

First round of reviews. I think the way how filters and configurations are set is quite messy. It would be nice to try to clean up as much as possible to make it more readable. In my opinion it would be nicer to use classes instead, but the classes should be properly defined and with a clear scope so there is no confusion between them. It is already a bit confusing the VariantCallerFilters and VarCallerFilters. It would be good to either merge, rename or refactor to know what is going on

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[fevac]

Nice improvements! 🌟 Just some questions below

[fevac]

Thanks for addressing it. I'm still unsure if there would be any unexpected behaviour from the code. But it seems that you have tested this and it's good to go. So 🚀

[mathiasbio]

Thanks a lot for the review and the great suggestions @fevac ! ❤️