#3591 panel app vus/gus relationship (#1088)

* Panal App VUS/GUS relationship download

* Panal App VUS/GUS relationship download

* Providing methods to find more useful relationships between term and gene symbol for report

* issue #2647 - liftover refactor

* issue #2647 - start of bcftools liftover

* Issue #2647 - BCFTools liftover

* Issue #2647 - Remove vestigial NCBI remap traces

* Issue #2647 - enable in vg test

* Stop warning that doesn't apply here

* Issue #2647 - be able to create variant for long ones...

* This occasionally failed - execution continued after asking to change window

* issue #1054 - make bcftools liftover part of standard ANNOTATION settings

* issue #1054 - misc liftover issues

* Issue #1052 - analysis template version

* Make VEP have version in path

* Shariant test config - enable BCFTools liftover

* Upload initially shows error wrongly in pipeline race condition

* issue #818 - don't uniq on preprocess anymore

* issue #1056 - vcf_clean_and_filter convert contig headers

* issue #1057 - VEP deployment change to explicit version

* vg test config

* issue #980 - karyomapping use symbol instead of gene

* Renamed counts to Counts(germline only) on overlaps page (#1046)

* Fix issue where alleleOriginToggle was being called with undefined

* WIP fixing conda dependencies

* Update of conda environment (works again in conda)

* Add new evidence key for somatic testing

* Show better quick clinical significance values (for somatic and germline)

* Fix recently introduced bug in suggested terms for an allele

* Allow link for condition matching to appear on a classification even when not in edit mode

* issue #1059 - all variants node

* issue #1059 - all variants node

* Adjust node counts a little

* issue #1043 - export VCF

* Tidy up server status page removing redundant data (now go to Overall Status)

* Fix the evidence key values

* Get c.HGVS showing properly on variant details page again

* issue #758 - configure quick links via settings

* More comments in conda file

* Move condition text match from classification.html to JSON to keep things in sync
More requirements on when you will be linked to condition resolution

* Style fixes for new liftover

* Put liftover date on form

* Move overall data to the splash server status page

* Add Clinical Trails gov to quick links

* Add liftover to settings menu if variants menu disabled

* Update shariant prod settings to use VEP v108

* Make liftover page side menu change based on settings

* Better wording for changing clinical contexts within a discordance

* Tighten up the handling of embedded card or modal (used for triage but affected other modals by mistake)

* Better safety around condition text relationships

* For view metrics use heading of "Users" not "User"

* Add overall stats to liftover pages

* Slight formatting on liftover note

* Rework the internals of the view user activity reports

* More styling on liftover runs page

* More styling on liftover pages

* Tiny column alignment issue on liftover

* linting

* Fix exclusion of blank searches in metrics

* Subtle change of header wording for activity report

* Improve the formatting of missing IDs when making a batch for ClinVar

* Proper int to string handling for missing IDs

* issue #758 - quick links via settings

* Reworking of Lab Differences using new zippable functionality for ExportRow

* Put the allele URL into the lab compare (I think it was there before)

* Enable SEARCH_HGVS_GENE_SYMBOL_USE_MANE for shariant test

* Also try enabling SEARCH_HGVS_GENE_SYMBOL

* Fix bug where condition URL was calculated before seeing if we had condition text

* Enable extra variant annotations for testing

* Attempt at parsing OncogenicityClassification for pulling in ClinVar records

* issue #1193 - analysis audit log

* issue #1193 - analysis audit log

* issue #1193 - analysis audit log

* issue #1193 - analysis audit log

* issue #1193 - analysis audit log

* variantgrid_private#1193 - Audit log for analysis

* Get audit log name right in requirements

* issue #1060  - variant details page, create allele for all variants not just shorty clingen ones

* Sometimes can't reload analysis #1061

* issue #1193 - disable audit log for template copies etc

* issue #1193 - Audit log for analysis - click to expand and see JSON

* issue #1193 - Don't audit node cloning (was causing tests to fail)

* temp fix for #1053 - will work out data fix later

* Better categorising of ClinVar's Somatic/Germline records

* issue #1053 - cohort genotype versions - make common cohort version match cohort.version

* linting - format whitespace

* linting - remove unused imports

* Unused file

* linting - f string w/o interpolation

* linting - use generator

* Increase the ClinVar parser version to purge old cache

* Add django-audit-log to conda environment

* ClinVar REcords More work on parsing condition

* issue #1053 - missing import, make sure we retrieve only 1 cgc per cohort

* issue #877 - Export column - c_hgvs_compat

* Add MeSH as a non-local ontology set

* Ability to render multiple conditions against a single ClinVar record

* Don't abbreviate ref/alt in VCF

* Fix bug with filtering out non-human clinvar records

* Count homo sapiens, homo-sapiens, homosapiens the same as human for ClinVar records

* A script to close manually raised flags

* Raise classification change flag for both Classification and Somatic Clinical Significance

* Filter based on creator of flag

* Added prefixed clinvar batch export CB_ and clinvar export CE_ search functionality (#1066)

* issue #3604 - COSMIC search

* Update changelog

* Tidy up code for close flags, allow it to re-open flags

* Bug fixes for the condition checking code

* panalapp work

* PanelApp Compare: Fixes to stop infinite recurssion, correct method signatures.

* #3591 panel app export

---------

Co-authored-by: TheMadBug <[email protected]>
Co-authored-by: Dave Lawrence <[email protected]>

classification/views/classification_export_view.py

@@ -96,6 +96,7 @@ def _export_view_context(request: HttpRequest) -> dict:
     format_json = {'id': 'json', 'name': 'JSON'}
     format_spelling = {'id': 'spelling', 'name': 'Spelling Report', 'admin_only': True}
     format_lab_compare = {'id': 'lab_compare', 'name': 'Lab Compare', 'admin_only': True}
+    format_condition_resolution = {'id': 'condition_resolution', 'name': 'Condition Resolution', 'admin_only': True}
     format_redcap = {'id': 'redcap', 'name': 'REDCap'}
     format_vcf = {'id': 'vcf', 'name': 'VCF'}
     formats = [
@@ -105,6 +106,7 @@ def _export_view_context(request: HttpRequest) -> dict:
         format_clinvar_expert_compare,
         format_spelling,
         format_lab_compare,
+        format_condition_resolution,
         format_json,
         format_mvl
     ]

classification/views/exports/__init__.py

@@ -7,5 +7,6 @@
 from classification.views.exports.classification_export_formatter_mvl import ClassificationExportFormatterMVL
 from classification.views.exports.classification_export_formatter_redcap import ClassificationExportFormatterRedCap
 from classification.views.exports.classification_export_formatter_spelling import ClassificationExportFormatterSpelling
+from classification.views.exports.classification_export_formatter_condition_resolution import ClassificationExportFormatterConditionResolution
 from classification.views.exports.classification_export_formatter_vcf import ClassificationExportFormatterVCF
 from classification.views.exports.classification_export_formatter_lab_compare import ClassificationExportInternalCompare

classification/views/exports/classification_export_formatter_condition_resolution.py

@@ -0,0 +1,163 @@
+from dataclasses import dataclass
+from typing import List, Optional
+
+from django.http import HttpRequest
+from django.urls.base import reverse
+
+from classification.enums import SpecialEKeys
+from classification.models import ClassificationModification, Classification
+from classification.views.exports.classification_export_decorator import register_classification_exporter
+from classification.views.exports.classification_export_filter import ClassificationFilter, AlleleData
+from classification.views.exports.classification_export_formatter import ClassificationExportFormatter
+from library.django_utils import get_url_from_view_path
+from library.utils import ExportRow, export_column, delimited_row
+from ontology.models import OntologyTerm, OntologyTermRelation, OntologyRelation, OntologyImportSource, \
+    PanelAppClassification, OntologySnake, OntologyService, GeneDiseaseClassification
+
+
+@dataclass(frozen=True)
+class ClassificationConditionResolutionRow(ExportRow):
+    classification: ClassificationModification
+    condition: OntologyTerm
+    gene_symbol: str
+    panel_app_strength: Optional[set[PanelAppClassification]]
+    gencc_strength: Optional[set[GeneDiseaseClassification]]
+    mondo_strength: Optional[set[str]]
+    all_relations: Optional[set[OntologyTerm]] = None
+
+    @property
+    def cm(self):
+        return self.classification
+
+    @property
+    def vc(self):
+        return self.classification.classification
+
+    @export_column('Classification ID')
+    def classification_id(self):
+        return self.vc.id
+
+    @export_column('Lab')
+    def lab(self):
+        return self.vc.lab.name
+
+    @export_column('c.HGVS')
+    def c_hgvs(self):
+        return self.vc.c_parts.full_c_hgvs
+
+    @export_column('Allele Origin')
+    def allele_origin(self):
+        return self.vc.allele_origin_bucket
+
+    @export_column('Provided Condition')
+    def condition(self):
+        return self.condition
+
+    @export_column('Date Curated')
+    def date_curated(self):
+        return self.cm.curated_date.date()
+
+    @export_column('Gene Symbol')
+    def gene_symbol(self):
+        return self.gene_symbol
+
+    @export_column('Panel App Strength')
+    def panel_app_strength(self):
+        if self.panel_app_strength:
+            return ', '.join(relation.label for relation in self.panel_app_strength)
+        else:
+            return '-'
+
+    @export_column('Gencc Strength')
+    def gencc_strength(self):
+        if self.gencc_strength:
+            return ', '.join(relation.label for relation in self.gencc_strength)
+        else:
+            return '-'
+
+    @export_column('MONDO Strength')
+    def mondo_strength(self):
+        if self.mondo_strength:
+            return ', '.join(self.mondo_strength)
+        else:
+            return '-'
+
+    @export_column('Matching Condition')
+    def matching_relations(self):
+        if self.all_relations:
+            return ', '.join(str(relation) for relation in self.all_relations)
+        else:
+            return '-'
+
+
+@register_classification_exporter("condition_resolution")
+class ClassificationExportFormatterConditionResolution(ClassificationExportFormatter):
+
+    @classmethod
+    def from_request(cls, request: HttpRequest) -> 'ClassificationExportFormatterConditionResolution':
+        return ClassificationExportFormatterConditionResolution(
+            classification_filter=ClassificationFilter.from_request(request),
+        )
+
+    def content_type(self) -> str:
+        return "text/csv"
+
+    def extension(self) -> str:
+        return "csv"
+
+    def header(self) -> List[str]:
+        return [delimited_row(ClassificationConditionResolutionRow.csv_header())]
+
+    def footer(self) -> List[str]:
+        return []
+
+    def row(self, allele_data: AlleleData) -> list[str]:
+        rows: list[str] = []
+        for vcm in allele_data.cms:
+            if condition_obj := vcm.classification.condition_resolution_obj:
+                if not condition_obj.is_multi_condition and condition_obj.terms and condition_obj.terms[0].ontology_service in {OntologyService.MONDO, OntologyService.OMIM}:
+                    # only work for single conditions in MONDO or OMIM
+                    condition_term = condition_obj.terms[0]
+                    for gene_symbol in vcm.classification.allele_info.gene_symbols:
+                        # on the off chance there are 2 gene symbols, we can make results for each gene symbol individually
+                        all_relationships = []
+                        for snake in OntologySnake.get_all_term_to_gene_relationships(condition_term, gene_symbol):
+                            if snake_relations := snake.get_import_relations:
+                                all_relationships.append(snake_relations)
+
+                        panel_app_relationships = [relation for relation in all_relationships if relation.relation == OntologyRelation.PANEL_APP_AU]
+                        # we now have a list of potential PanelApp, MONDO, GenCC relationships all of different strengths to the exact condition term
+                        has_direct_panel_app_relationship = bool(panel_app_relationships)
+                        all_relations = set()
+                        panel_app_strength, gencc_strength, mondo_strength = set(), set(), set()
+
+                        if all_relationships:
+                            # make one row, including the biggest strength of each type of relationship
+                            for rel in all_relationships:
+
+                                source = rel.from_import.import_source
+                                if source in OntologyImportSource.PANEL_APP_AU:
+                                    panel_app_strength.add(rel.gencc_quality)
+                                    all_relations.add(rel.source_term)
+                                elif source == OntologyImportSource.MONDO:
+                                    mondo_strength.add(rel.relation)
+                                    all_relations.add(rel.source_term)
+                                elif source == OntologyImportSource.GENCC:
+                                    gencc_strength.add(rel.gencc_quality)
+                                    all_relations.add(rel.source_term)
+
+                        if not has_direct_panel_app_relationship:
+                            # there may or may not have been any relationships, but there wasn't one from panel app, so now lets look at all conditions for the gene symbol and make a row for each one
+                            if all_relationships_snakes := OntologySnake.terms_for_gene_symbol(gene_symbol=gene_symbol, desired_ontology=OntologyService.MONDO, min_classification=GeneDiseaseClassification.DISPUTED):
+                                all_relationships = all_relationships_snakes.leaf_relations(OntologyRelation.PANEL_APP_AU)
+                                for relation in all_relationships:
+                                    all_relations.add(f'({relation.gencc_quality.label}): {relation.source_term}')
+
+                        if all_relations:
+                            row = ClassificationConditionResolutionRow(vcm, condition_term, gene_symbol,
+                                                                       panel_app_strength, gencc_strength,
+                                                                       mondo_strength, all_relations)
+
+                            rows.append(delimited_row(row.to_csv()))
+
+        return rows

ontology/admin.py

@@ -1,6 +1,6 @@
 from django.contrib import admin
 
-from ontology.models import OntologyTerm, OntologyImport
+from ontology.models import OntologyTerm, OntologyImport, OntologyTermRelation
 from snpdb.admin_utils import ModelAdminBasics
 
 
@@ -28,3 +28,12 @@ def is_readonly_field(self, f) -> bool:
 
     def has_add_permission(self, request):
         return False
+
+
+@admin.register(OntologyTermRelation)
+class OntologyImportAdmin(ModelAdminBasics):
+    search_fields = ('relation', 'from_import__import_source', 'from_import__context')
+    list_filter = ('from_import__import_source',)
+
+    def has_add_permission(self, request):
+        return False

ontology/models/models_ontology.py

@@ -9,7 +9,7 @@
 from collections import defaultdict
 from dataclasses import dataclass
 from functools import cached_property
-from typing import Optional, Union, Iterable, Any
+from typing import Optional, Union, Iterable, Any, Iterator
 
 from cache_memoize import cache_memoize
 from django.conf import settings
@@ -154,6 +154,31 @@ class OntologyRelation:
     "http://purl.obolibrary.org/obo/RO_0004030": "disease arises from structure"
     """
 
+class PanelAppClassification(models.TextChoices):
+    GREEN = "1", "Expert Review Green"
+    AMBER = "2", "Expert Review Amber"
+    RED = "3", "Expert Review Red"
+
+    @property
+    def is_strong_enough(self) -> bool:
+        return self == PanelAppClassification.GREEN
+
+    @staticmethod
+    def get_by_label_pac(label: str) -> 'PanelAppClassification':
+        for pac in PanelAppClassification:
+            if pac.label == label:
+                return pac
+        raise ValueError(f"No PanelAppClassification for {label}")
+
+    @staticmethod
+    def get_above_min(min_classification: str) -> list[str]:
+        classifications = []
+        for e in reversed(PanelAppClassification):
+            classifications.append(e.label)
+            if e.value == min_classification:
+                break
+        return classifications
+
 
 class GeneDiseaseClassification(models.TextChoices):
     # @see https://thegencc.org/faq.html#validity-termsdelphi-survey - where sort order comes from
@@ -594,10 +619,14 @@ def relation_display(self):
         return OntologyRelation.DISPLAY_NAMES.get(self.relation, self.relation)
 
     @property
-    def gencc_quality(self) -> Optional[GeneDiseaseClassification]:
+    def gencc_quality(self) -> GeneDiseaseClassification | PanelAppClassification | None:
         if extra := self.extra:
             if strongest := extra.get('strongest_classification'):
-                return GeneDiseaseClassification.get_by_label(strongest)
+                try:
+                    label = GeneDiseaseClassification.get_by_label(strongest)
+                except ValueError:
+                    label = PanelAppClassification.get_by_label_pac(strongest)
+                return label
         return None
 
     @staticmethod
@@ -911,6 +940,14 @@ def leaf_relationship(self) -> OntologyTermRelation:
     def start_source(self) -> OntologyImportSource:
         return self.show_steps()[0].relation.from_import.import_source
 
+    @cached_property
+    def get_import_relations(self) -> Optional[OntologyTermRelation]:
+        for step in self.show_steps():
+            if step.relation.from_import.import_source in {OntologyImportSource.PANEL_APP_AU,
+                                                           OntologyImportSource.GENCC,
+                                                           OntologyImportSource.MONDO}:
+                return step.relation
+
     @staticmethod
     def check_if_ancestor(descendant: OntologyTerm, ancestor: OntologyTerm, max_levels=4) -> list['OntologySnake']:
         if ancestor == descendant:
@@ -1110,7 +1147,8 @@ def terms_for_gene_symbol(gene_symbol: Union[str, GeneSymbol], desired_ontology:
 
     @staticmethod
     def has_gene_relationship(term: Union[OntologyTerm, str], gene_symbol: Union[GeneSymbol, str], quality: Optional[GeneDiseaseClassification] = GeneDiseaseClassification.STRONG) -> bool:
-        # TODO, do this with hooks
+        # TODO, have this run off get_all_term_to_gene_relationships
+        # just need to filter through results until we reach one of a high enough quality
         from ontology.panel_app_ontology import update_gene_relations
         update_gene_relations(gene_symbol)
         if isinstance(term, str):
@@ -1140,6 +1178,39 @@ def has_gene_relationship(term: Union[OntologyTerm, str], gene_symbol: Union[Gen
             report_exc_info()
             return False
 
+    def get_all_term_to_gene_relationships(term: Union[OntologyTerm, str], gene_symbol: Union[GeneSymbol, str], try_related_terms: bool = True) -> Iterator['OntologySnake']:
+        # iterates all ontology term relationships between the term and the gene symbol (as well as any relationships to the equiv MONDO/OMIM)
+        from ontology.panel_app_ontology import update_gene_relations
+        update_gene_relations(gene_symbol)
+        if isinstance(term, str):
+            term = OntologyTerm.get_or_stub(term)
+            if term.is_stub:
+                return None
+        try:
+            gene_term = OntologyTerm.get_gene_symbol(gene_symbol)
+            # try direct link first
+            otr_qs = OntologyVersion.get_latest_and_live_ontology_qs()
+            for relationship in otr_qs.filter(source_term=term, dest_term=gene_term):
+                yield OntologySnake(source_term=term, leaf_term=gene_term, paths=[relationship])
+
+            if not try_related_terms:
+                return None
+
+            # optimisations for OMIM/MONDO
+            if term.ontology_service in {OntologyService.MONDO, OntologyService.OMIM}:
+                if term.ontology_service == OntologyService.MONDO:
+                    if omim := OntologyTermRelation.as_omim(term):
+                        for relation in OntologySnake.get_all_term_to_gene_relationships(omim, gene_symbol, try_related_terms=False):
+                            yield relation
+
+                elif term.ontology_service == OntologyService.OMIM:
+                    if mondo := OntologyTermRelation.as_mondo(term):
+                        for relation in OntologySnake.get_all_term_to_gene_relationships(mondo, gene_symbol, try_related_terms=False):
+                            yield relation
+        except ValueError:
+            report_exc_info()
+            return None
+
     @staticmethod
     def get_children(term: OntologyTerm):
         relationships = OntologyVersion.get_latest_and_live_ontology_qs().filter(dest_term=term, relation=OntologyRelation.IS_A).select_related("source_term")