Show "missing" counts in confusion matrices #5187
Conversation
|
Warning Rate limit exceeded@imanjra has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 1 minutes and 0 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (2)
WalkthroughThe pull request modifies the Changes
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
brimoor
changed the title
brimoor
changed the title
brimoor
changed the title
brimoor
changed the title
brimoor
force-pushed
the
model-eval-show-missing-none
branch
from
f4bf6c5 to
cf54730
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's limits when posting large numbers of comments.
Actionable comments posted: 1
🧹 Nitpick comments (2)
fiftyone/operators/builtins/panels/model_evaluation/__init__.py (1)
252-278: Consider consolidating calls toresults._confusion_matrixto improve performance.In
get_confusion_matrices, the method callsresults._confusion_matrixmultiple times with different class orderings. If this operation is computationally expensive, consider caching the results or refactoring the logic to reduce redundant computations.app/packages/core/src/plugins/SchemaIO/components/NativeModelEvaluationView/Evaluation.tsx (1)
Line range hint
169-180: Define explicit TypeScript types forevaluationConfigand related variables.Currently, variables like
evaluationConfig,evaluationType, andevaluationMethodmay be implicitly typed asany. Defining explicit interfaces or types for these variables will enhance type safety and improve code readability.
🛑 Comments failed to post (1)
fiftyone/operators/builtins/panels/model_evaluation/__init__.py (1)
443-448:
⚠️ Potential issueHandle potential
Nonevalue forfieldbefore callingfield.upper().In the
load_viewmethod,field.upper()is called without checking iffieldis notNone. IffieldisNone, this will raise anAttributeError. Please add a check to ensurefieldis notNonebefore callingupper().Apply this diff to fix the issue:
if self.is_binary_classification(info): + if field is not None: uppercase_field = field.upper() + else: + # Handle the None case appropriately + uppercase_field = None view = ctx.dataset.match( {computed_eval_key: {"$eq": uppercase_field}} )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.if self.is_binary_classification(info): if field is not None: uppercase_field = field.upper() else: # Handle the None case appropriately uppercase_field = None view = ctx.dataset.match( {computed_eval_key: {"$eq": uppercase_field}} ) else:
brimoor
force-pushed
the
model-eval-show-missing-none
branch
from
cf54730 to
2a39e70
Compare
brimoor
changed the title
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's limits when posting large numbers of comments.
Actionable comments posted: 1
🧹 Nitpick comments (3)
fiftyone/operators/builtins/panels/model_evaluation/__init__.py (3)
249-275: Add error handling for edge casesWhile the implementation correctly handles missing classes, consider adding error handling for the following edge cases:
- Empty results (no classes)
- All classes being missing/other
- Invalid class types in results
def get_confusion_matrices(self, results): + if not results.classes or len(results.classes) == 0: + raise ValueError("No classes found in results") + default_classes = results.classes.tolist() freq = Counter(results.ytrue) if results.missing in freq: freq.pop(results.missing) + if not freq: + raise ValueError("All classes are missing/other")
329-330: Add type validation for missing stateConsider validating the type of
results.missingbefore setting it as state to ensure consistent behavior across different result types.- ctx.panel.set_state("missing", results.missing) + missing = results.missing + if not isinstance(missing, (str, int)): + missing = str(missing) + ctx.panel.set_state("missing", missing)
481-545: Improve code structure for better maintainabilityThe detection view filtering logic is complex and could benefit from being split into smaller, focused methods. This would improve readability and maintainability.
Consider extracting the following methods:
def _filter_class_view(self, view, x, gt_field, pred_field, gt_field2=None, pred_field2=None): """Handle class-specific view filtering.""" pass def _filter_matrix_view(self, view, x, y, missing, eval_key, gt_field, pred_field): """Handle confusion matrix cell filtering.""" pass def _filter_field_view(self, view, field, eval_key, gt_field, pred_field): """Handle field-specific view filtering.""" pass
🛑 Comments failed to post (1)
fiftyone/operators/builtins/panels/model_evaluation/__init__.py (1)
425-447: 🛠️ Refactor suggestion
Add evaluation compatibility validation
When comparing two evaluations, it's important to validate that they are compatible (same evaluation type, comparable metrics, etc.).
eval_key2 = view_state.get("compareKey", None) pred_field2 = None gt_field2 = None if eval_key2 is not None: info2 = ctx.dataset.get_evaluation_info(eval_key2) + if info2.config.type != info.config.type: + raise ValueError( + f"Incompatible evaluation types: {info.config.type} vs {info2.config.type}" + ) pred_field2 = info2.config.pred_field if info2.config.gt_field != gt_field: gt_field2 = info2.config.gt_field📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.view_options = ctx.params.get("options", {}) eval_key = view_state.get("key") eval_key = view_options.get("key", eval_key) eval_view = ctx.dataset.load_evaluation_view(eval_key) info = ctx.dataset.get_evaluation_info(eval_key) pred_field = info.config.pred_field gt_field = info.config.gt_field eval_key2 = view_state.get("compareKey", None) pred_field2 = None gt_field2 = None if eval_key2 is not None: info2 = ctx.dataset.get_evaluation_info(eval_key2) if info2.config.type != info.config.type: raise ValueError( f"Incompatible evaluation types: {info.config.type} vs {info2.config.type}" ) pred_field2 = info2.config.pred_field if info2.config.gt_field != gt_field: gt_field2 = info2.config.gt_field x = view_options.get("x", None) y = view_options.get("y", None) field = view_options.get("field", None) missing = ctx.panel.get_state("missing", "(none)")
brimoor
force-pushed
the
model-eval-show-missing-none
branch
from
2a39e70 to
683a612
Compare
brimoor
force-pushed
the
model-eval-fixes2
branch
from
94ed26c to
a9ea1c3
Compare
brimoor
force-pushed
the
model-eval-show-missing-none
branch
from
683a612 to
3168e30
Compare
|
@imanjra there's a couple frontend TODO items in the PR description that will need to be addressed before we can merge this. Could you take a look?? |
imanjra
force-pushed
the
model-eval-show-missing-none
branch
from
6f5dc95 to
76a1492
Compare
imanjra
force-pushed
the
model-eval-show-missing-none
branch
from
76a1492 to
ba1ecf6
Compare
imanjra
force-pushed
the
model-eval-show-missing-none
branch
from
ba1ecf6 to
325a0a0
Compare
Change log
When evaluating object detections, confusion matrices need the ability to show an additional row/column to capture “missing” values:
TODO
set_view()callbacks to correctly handle "missing" casesk=10classes, we need to display 11 total rows/cols: the 10 rows/cols representing the classes they requested PLUS the last row/col so that we're showing the "missing" data. This means the frontend logic for applying thelimit=klogic will need to select the firstkrows plus the last row of the appropriate confusion matrix, rather than just the firstkrowsExample usage