Skip to content
This repository has been archived by the owner on Dec 12, 2024. It is now read-only.

Test vectors refactor #11

Merged
merged 5 commits into from
Jan 5, 2024
Merged

Test vectors refactor #11

merged 5 commits into from
Jan 5, 2024

Conversation

amika-sq
Copy link
Contributor

@amika-sq amika-sq commented Jan 5, 2024

Previously, test vectors lived alongside normal unit tests. This caused a few pain points:

  1. Test classes did not follow the naming conventions defined in the sdk-development test vector spec.
    • e.g. Test classes must be prefixed with Web5TestVectors.
  2. It was difficult to understand which tests represented a test vector vs which tests represented a unit test.
    • e.g. Does test_resolve run the resolve test vector, or does it run a unit test on the resolve function? It was impossible to tell without intimate knowledge of the test itself, or the use of rigorous naming conventions of the tests themselves (which isn't possible, because the test names have their own naming convention according to the spec).
  3. The test result output didn't break down each test vector. When one test vector failed, it gave no indication as to which one of the test vectors in the file failed.
    • e.g.: did_jwk/resolve has three separate vectors defined within the resource JSON file. If one vector failed, but the other two succeeded, the test would fail and the output gave no indication as to which of the vectors failed. The developer would need to manually inspect the test with breakpoints in order to determine which one it was. This was time consuming, and just overall a bad experience.

To address these pain points, I have refactored how test vectors work in this package.

Changes made:

  1. All test vectors are now defined in a separate testing target called Web5TestVectors.
  2. All test vectors are now named with the appropriate naming conventions.
  3. All common utility code shared between unit tests & test vectors are defined in a new TestUtilities target.
  4. When running a test vector, each vector is run within its own named XCTActivity, allowing us to know which of the vectors actually failed.

Previously, if a vector failed, we'd have no idea which vector it
was that failed without manually setting breakpoints.

This new structure categorizes each vector within an XCTContext,
named with the vector's description.

This means that when a vector fails, we can look at the Xcode output and
determine exactly which vector failed.
This will produce far better error messages, as well as being less error prone
by removing indexing via strings.
@amika-sq amika-sq merged commit 78d57fa into main Jan 5, 2024
1 check passed
@amika-sq amika-sq deleted the test-vectors-refactor branch January 5, 2024 15:58
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant