Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance Readability of validation check failures #4299

Merged
merged 7 commits into from
Dec 13, 2024
Merged

Conversation

shashank-elastic
Copy link
Contributor

@shashank-elastic shashank-elastic commented Dec 13, 2024

Pull Request

Issue link(s): - #3195

Summary - What I changed

  • Have a static error message which is a unique string and describe the error
  • The details of the error are logged properly before we throw the error and the format of the error details are outside of ValueError scope

How To Test

  • Simulated a Local validation Failure
  • Clear log messages and trace are simulated.
===================================================================== FAILURES =====================================================================
_____________________________________________ TestAlertSuppression.test_eql_non_sequence_support_only ______________________________________________
tests/base.py:85: in setUp
    self.fail(f'Rule loader failure: \n{RULE_LOADER_FAIL_MSG}')
E   AssertionError: Rule loader failure: 
E   Error in both stack and integrations checks
-------------------------------------------------------------- Captured stdout setup ---------------------------------------------------------------
Stack Error Trace: Error at line:3,column:4
Field not recognized for authentication event
sequence by host.id, source.ip, user.name with maxspan=15s
  [ authentication where host.os.type == "linux" and 
   event.actionS in ("ssh_login", "user_login") and event.outcome == "failure" and
   ^^^^^^^^^^^^^
stack: 8.18.0, beats: 8.16.1,ecs: 8.16.0, endgame: 8.4.0
Integrations Error Trace: Error at line:3,column:4
Field not recognized for authentication event
sequence by host.id, source.ip, user.name with maxspan=15s
  [ authentication where host.os.type == "linux" and 
   event.actionS in ("ssh_login", "user_login") and event.outcome == "failure" and
   ^^^^^^^^^^^^^
stack: 8.12.0, integration: None,ecs: 8.11.0, package: system, package_version: 1.63.0
Error loading rule in /Users/shashankks/elastic_workspace/detection-rules/rules/linux/credential_access_potential_linux_ssh_bruteforce_external.toml
================================================================= warnings summary =================================================================
.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:1276
.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:1276
  /Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:1276: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: typeguard
    self._mark_plugins_for_rewrite(hook)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================= short test summary info ==============================================================
FAILED tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only - AssertionError: Rule loader failure: 
============================================== 1 failed, 91 passed, 63 skipped, 2 warnings in 13.67s ===============================================
(.venv) 
detection-rules on  issue-3195 [$!?] is 📦 v0.3.6 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] took 14s 
  • On no failures all Unit test should pass.

Checklist

  • Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
  • Added the meta:rapid-merge label if planning to merge within 24 hours
  • Secret and sensitive material has been managed correctly
  • Automated testing was updated or added to match the most common scenarios
  • Documentation and comments were added for features that require explanation

Contributor checklist

@shashank-elastic shashank-elastic added the bug Something isn't working label Dec 13, 2024
@shashank-elastic shashank-elastic self-assigned this Dec 13, 2024
@botelastic botelastic bot added the python Internal python for the repository label Dec 13, 2024
@shashank-elastic shashank-elastic linked an issue Dec 13, 2024 that may be closed by this pull request
@protectionsmachine
Copy link
Collaborator

Bug - Guidelines

These guidelines serve as a reminder set of considerations when addressing a bug in the code.

Documentation and Context

  • Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
  • Include additional context or details about the problem.
  • Ensure the fix includes necessary updates to the release documentation and versioning.

Code Standards and Practices

  • Code follows established design patterns within the repo and avoids duplication.
  • Code changes do not introduce new warnings or errors.
  • Variables and functions are well-named and descriptive.
  • Any unnecessary / commented-out code is removed.
  • Ensure that the code is modular and reusable where applicable.
  • Check for proper exception handling and messaging.

Testing

  • New unit tests have been added to cover the bug fix or edge cases.
  • Existing unit tests have been updated to reflect the changes.
  • Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
  • Validate that any rules affected by the bug are correctly updated.
  • Ensure that performance is not negatively impacted by the changes.
  • Verify that any release artifacts are properly generated and tested.

Additional Checks

  • Ensure that the bug fix does not break existing functionality.
  • Review the bug fix with a peer or team member for additional insights.
  • Verify that the bug fix works across all relevant environments (e.g., different OS versions).
  • Confirm that all dependencies are up-to-date and compatible with the changes.
  • Confirm that the proper version label is applied to the PR patch, minor, major.

Copy link
Contributor

@eric-forte-elastic eric-forte-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit, looks good to me 👍

@shashank-elastic shashank-elastic merged commit 2ff2965 into main Dec 13, 2024
12 checks passed
@shashank-elastic shashank-elastic deleted the issue-3195 branch December 13, 2024 13:33
protectionsmachine pushed a commit that referenced this pull request Dec 13, 2024
protectionsmachine pushed a commit that referenced this pull request Dec 13, 2024
protectionsmachine pushed a commit that referenced this pull request Dec 13, 2024
protectionsmachine pushed a commit that referenced this pull request Dec 13, 2024
protectionsmachine pushed a commit that referenced this pull request Dec 13, 2024
protectionsmachine pushed a commit that referenced this pull request Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport: auto bug Something isn't working patch python Internal python for the repository
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Render the error message of failed validation as a string
4 participants