Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bignum test case generation script #6093

Merged
Merged
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
8b2df74
Add bignum test generation framework
wernerlewis Jul 8, 2022
69a92ce
Add test generation for bignum cmp variant
wernerlewis Jul 18, 2022
86caf85
Add test case generation for bignum add
wernerlewis Jul 18, 2022
a51fe2b
Sort tests when generating cases
wernerlewis Jul 20, 2022
b17ca8a
Remove set() to preserve test case order
wernerlewis Jul 20, 2022
c442f6a
Fix type issues
wernerlewis Jul 20, 2022
265e051
Remove is None from if statement
wernerlewis Jul 20, 2022
6a31396
Fix incorrect indentation
wernerlewis Jul 20, 2022
75ef944
Fix CMake change failures on Windows
wernerlewis Jul 21, 2022
383461c
Separate CMake targets for bignum and PSA
wernerlewis Aug 23, 2022
fbb75e3
Separate common test generation classes/functions
wernerlewis Aug 24, 2022
55e638c
Remove abbreviations and clarify attributes
wernerlewis Aug 23, 2022
92c876a
Remove unneeded list concatenation
wernerlewis Aug 23, 2022
6c70d74
Convert bools to int before arithmetic
wernerlewis Aug 24, 2022
169034a
Add details to docstrings
wernerlewis Aug 23, 2022
699e126
Use ABCMeta for abstract classes
wernerlewis Aug 24, 2022
2b527a3
Split generate_tests to reduce code complexity
wernerlewis Aug 24, 2022
cfd4768
Use __new__() for case counting
wernerlewis Aug 24, 2022
d03d2a3
Remove trailing whitespace in description
wernerlewis Aug 24, 2022
6300b4f
Add missing typing
wernerlewis Aug 24, 2022
9990b30
Use typing casts for fixed-width tuples
wernerlewis Aug 24, 2022
a195ce7
Disable pylint unused arg in __new__
wernerlewis Aug 24, 2022
6d654c6
Raise NotImplementedError in abstract methods
wernerlewis Aug 25, 2022
e3ad22e
Fix TARGET types and code style
wernerlewis Aug 25, 2022
a16b617
Disable abstract check in pylint
wernerlewis Aug 25, 2022
f156c43
Use argparser default for directory
wernerlewis Aug 25, 2022
6ef5436
Clarify documentation
wernerlewis Aug 25, 2022
9df9faa
Use argparser default for targets
wernerlewis Aug 25, 2022
76f4562
Fix trailing whitespace
wernerlewis Aug 25, 2022
3366ebc
Add test_generation.py dependency in builds
wernerlewis Aug 25, 2022
81f2444
Modify wording in docstrings
wernerlewis Aug 25, 2022
a4b7720
Use `combinations_with_replacement` for inputs
wernerlewis Aug 31, 2022
466f036
Add dependencies attribute to BaseTarget
wernerlewis Aug 31, 2022
aaf3b79
Use Python 3.5 style typing for dependencies
wernerlewis Aug 31, 2022
a4668a6
Rework TestGenerator to add file targets
wernerlewis Sep 2, 2022
5601308
Remove unused imports
wernerlewis Sep 2, 2022
855e45c
Use simpler int to hex string conversion
wernerlewis Sep 2, 2022
1fade8a
Move symbol definition out of __init__
wernerlewis Sep 12, 2022
3dc4519
Replace L/R inputs with A/B
wernerlewis Sep 12, 2022
34d6d3e
Update comments/docstrings in TestGenerator
wernerlewis Sep 14, 2022
858cffd
Add toggle for test case count in descriptions
wernerlewis Sep 14, 2022
00d0242
Remove argparser default for directory
wernerlewis Sep 14, 2022
b6e8091
Use typing.cast instead of unqualified cast
wernerlewis Sep 14, 2022
ac446c8
Add combination_pairs helper function
wernerlewis Sep 14, 2022
52ae326
Update references to file targets in docstrings
wernerlewis Sep 14, 2022
07c830c
Fix setting for default test suite directory
wernerlewis Sep 15, 2022
c2fb540
Use a script specific description in CLI help
wernerlewis Sep 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Split generate_tests to reduce code complexity
Previous implementation mixed the test case generation and the
recursive generation calls together. A separate method is added to
generate test cases for the current class' test function. This reduces
the need to override generate_tests().

Signed-off-by: Werner Lewis <[email protected]>
wernerlewis committed Aug 24, 2022
commit 2b527a394dd851905bb73a6ccfd810fbbd7c6d77
35 changes: 25 additions & 10 deletions scripts/mbedtls_dev/test_generation.py
Original file line number Diff line number Diff line change
@@ -25,7 +25,7 @@
import re

from abc import ABCMeta, abstractmethod
from typing import Callable, Dict, Iterable, List, Type, TypeVar
from typing import Callable, Dict, Iterable, Iterator, List, Type, TypeVar

from mbedtls_dev import build_tree
from mbedtls_dev import test_case
@@ -91,16 +91,31 @@ def create_test_case(self) -> test_case.TestCase:
return tc

@classmethod
def generate_tests(cls):
"""Generate test cases for the target subclasses.

During generation, each class will iterate over any subclasses, calling
this method in each.
In abstract classes, no tests will be generated, as there is no
function to generate tests for.
In classes which do implement a test function, this should be overridden
and a means to use `create_test_case()` should be added.
@abstractmethod
def generate_function_tests(cls) -> Iterator[test_case.TestCase]:
"""Generate test cases for the test function.

This will be called in classes where `test_function` is set.
Implementations should yield TestCase objects, by creating instances
of the class with appropriate input data, and then calling
`create_test_case()` on each.
"""
pass

@classmethod
def generate_tests(cls) -> Iterator[test_case.TestCase]:
"""Generate test cases for the class and its subclasses.

In classes with `test_function` set, `generate_function_tests()` is
used to generate test cases first.
In all classes, this method will iterate over its subclasses, and
yield from `generate_tests()` in each.

Calling this method on a class X will yield test cases from all classes
derived from X.
"""
if cls.test_function:
yield from cls.generate_function_tests()
for subclass in sorted(cls.__subclasses__(), key=lambda c: c.__name__):
yield from subclass.generate_tests()

12 changes: 4 additions & 8 deletions tests/scripts/generate_bignum_tests.py
Original file line number Diff line number Diff line change
@@ -160,14 +160,10 @@ def get_value_pairs(cls) -> Iterator[Tuple[str, ...]]:
yield from cls.input_cases

@classmethod
def generate_tests(cls) -> Iterator[test_case.TestCase]:
if cls.test_function:
# Generate tests for the current class
for l_value, r_value in cls.get_value_pairs():
cur_op = cls(l_value, r_value)
yield cur_op.create_test_case()
# Once current class completed, check descendants
yield from super().generate_tests()
def generate_function_tests(cls) -> Iterator[test_case.TestCase]:
for l_value, r_value in cls.get_value_pairs():
cur_op = cls(l_value, r_value)
yield cur_op.create_test_case()


class BignumCmp(BignumOperation):