⛩ Subsetting of data model / rules #987
Conversation
nikokaoja
commented
Feb 12, 2025
nikokaoja
commented
- [ALPHA] Ability to subset data model to desired concepts (classes/views)
☂️ Python Coverage
Overall Coverage
New Files
Modified Files
|
|
|
||
|
|
||
| @session_class_wrapper | ||
| class SubsetAPI: |
There was a problem hiding this comment.
I am wondering if there is a simpler name, subset is a technical term that might not encourage our less technical users.
| if self._state.rule_store.empty: | ||
| raise NeatSessionError("No rules to set the data model ID.") | ||
|
|
||
| warnings.filterwarnings("default") |
| raise NeatSessionError("Something went terrible wrong. Please contact the neat team.") | ||
|
|
||
| if not issues: | ||
| print(f"Subset to {after} concepts.") |
There was a problem hiding this comment.
after can risk not existing
| information = self._state.rule_store.provenance[-1].target_entity.information | ||
|
|
||
| if dms: | ||
| views = { |
There was a problem hiding this comment.
This selection logic should probably be in the transformer and not here
| after = len(self._state.rule_store.last_verified_dms_rules.views) | ||
|
|
||
| elif information: | ||
| classes = {ClassEntity(prefix=information.metadata.space, suffix=concept) for concept in concepts} |
There was a problem hiding this comment.
There is no warning about typos here. So if you mistype, you can easily drop all classes.
| @@ -101,6 +102,7 @@ def __init__( | |||
| self.inspect = InspectAPI(self._state) | |||
| self.mapping = MappingAPI(self._state) | |||
| self.drop = DropAPI(self._state) | |||
| self.subset = SubsetAPI(self._state) | |||
There was a problem hiding this comment.
Lets have a short discussion on name
| subset = subset.union(ancestors) | ||
|
|
||
| if not subset: | ||
| raise ValueError("None of the requested classes are defined in the rules!") |
There was a problem hiding this comment.
NeatValueError? This could happen as a result of poor user decisions?
| subset = subset.union(ancestors) | ||
|
|
||
| if not subset: | ||
| raise ValueError("None of the requested classes are defined in the rules!") |
There was a problem hiding this comment.
Same here. NeatValueError? This could happen as a result of poor user decisions?
| try: | ||
| return DMSRules.model_validate(subsetted_rules) | ||
| except ValidationError as e: | ||
| raise NeatValueError(f"Cannot subset rules: {e}") from e |
There was a problem hiding this comment.
Lets unpack the pydantic errors into nice, readable NeatErrors.
| raise NeatValueError(f"Cannot subset rules: {e}") from e | |
| neat_errors = from_pydantic_errors(e.errors()) | |
| body = "\n".join(f" - {neat_error.as_message()}" for neat_error in neat_errors) | |
| raise NeatValueError(f"Cannot subset rules: \n{body}") from e |
remember
from cognite.neat._issues._factory import from_pydantic_errorsThere was a problem hiding this comment.
Even better, make from_pydanti_errors return an IssueList and have a create_body method.
| try: | ||
| return InformationRules.model_validate(subsetted_rules) | ||
| except ValidationError as e: | ||
| raise NeatValueError(f"Cannot subset rules: {e}") from e |
There was a problem hiding this comment.
from cognite.neat._issues._factory import from_pydantic_errors| raise NeatValueError(f"Cannot subset rules: {e}") from e | |
| neat_errors = from_pydantic_errors(e.errors()) | |
| body = "\n".join(f" - {neat_error.as_message()}" for neat_error in neat_errors) | |
| raise NeatValueError(f"Cannot subset rules: \n{body}") from e |
