Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pydantic ValidationError in Chemical To Disease evaluation #211

Closed
caufieldjh opened this issue Sep 20, 2023 · 1 comment · Fixed by #226
Closed

Pydantic ValidationError in Chemical To Disease evaluation #211

caufieldjh opened this issue Sep 20, 2023 · 1 comment · Fixed by #226
Labels
bug Something isn't working evaluation Approaches for evaluating OntoGPT functions

Comments

@caufieldjh
Copy link
Member

caufieldjh commented Sep 20, 2023

Chemical to Disease evaluation as defined in ontogpt/src/ontogpt/evaluation/ctd/eval_ctd.py encounters a pydantic_core._pydantic_core.ValidationError in tests.integration.test_evaluation.test_eval_ctd.TestCTD. I think this is specific to the types used in the evaluation rather than anything defined in the test or text fixtures.
Details:

======================================================================
ERROR: test_eval (tests.integration.test_evaluation.test_eval_ctd.TestCTD)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/harry/ontogpt/tests/integration/test_evaluation/test_eval_ctd.py", line 47, in test_eval
    eos = evaluator.eval()
  File "/home/harry/ontogpt/src/ontogpt/evaluation/ctd/eval_ctd.py", line 187, in eval
    docs = list(self.load_test_cases())
  File "/home/harry/ontogpt/src/ontogpt/evaluation/ctd/eval_ctd.py", line 170, in load_cases
    yield ChemicalToDiseaseDocument(publication=pub, triples=triples)
  File "/home/harry/ontogpt/.venv/lib/python3.9/site-packages/pydantic/main.py", line 165, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for ChemicalToDiseaseDocument
publication
  Input should be a valid dictionary or instance of Publication [type=model_type, input_value=Publication(id=None, titl...xt=None, full_text=None), input_type=Publication]
    For further information visit https://errors.pydantic.dev/2.3/v/model_type
triples.0
  Input should be a valid dictionary or instance of ChemicalToDiseaseRelationship [type=model_type, input_value=Triple(subject='MESH:D015..., object_qualifier=None), input_type=Triple]
    For further information visit https://errors.pydantic.dev/2.3/v/model_type

======================================================================
ERROR: test_load_test_sets (tests.integration.test_evaluation.test_eval_ctd.TestCTD)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/harry/ontogpt/tests/integration/test_evaluation/test_eval_ctd.py", line 39, in test_load_test_sets
    docs = list(self.engine.load_test_cases())
  File "/home/harry/ontogpt/src/ontogpt/evaluation/ctd/eval_ctd.py", line 170, in load_cases
    yield ChemicalToDiseaseDocument(publication=pub, triples=triples)
  File "/home/harry/ontogpt/.venv/lib/python3.9/site-packages/pydantic/main.py", line 165, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for ChemicalToDiseaseDocument
publication
  Input should be a valid dictionary or instance of Publication [type=model_type, input_value=Publication(id=None, titl...xt=None, full_text=None), input_type=Publication]
    For further information visit https://errors.pydantic.dev/2.3/v/model_type
triples.0
  Input should be a valid dictionary or instance of ChemicalToDiseaseRelationship [type=model_type, input_value=Triple(subject='MESH:D015..., object_qualifier=None), input_type=Triple]
    For further information visit https://errors.pydantic.dev/2.3/v/model_type
@caufieldjh caufieldjh added bug Something isn't working evaluation Approaches for evaluating OntoGPT functions labels Sep 21, 2023
@caufieldjh caufieldjh linked a pull request Sep 29, 2023 that will close this issue
@caufieldjh
Copy link
Member Author

One thing that makes a difference here, surprisingly, is that nested classes (like in this case, how one of the slots in ChemicalToDiseaseRelationship requires a Publication) need to be imported from the same set of source classes.
Importing Publication from core.py and ChemicalToDiseaseRelationship from ctd.py doesn't work despite the Publication class being identical in both - it won't validate because it isn't the "same" class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working evaluation Approaches for evaluating OntoGPT functions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant