Script to validate test cases against the problem specifications.

[glennj]

Identifies test cases from the problem spec canonical data that are unimplemented.

Sample output:

The bash repo, where the scale-generator has no toml file:

$ bin/check_test_cases -a
MISSING: ./exercises/scale-generator/.meta/tests.toml
Exercise scale-generator
  Chromatic scale with sharps = not implemented
  Chromatic scale with flats  = not implemented
  Simple major scale          = not implemented
  Major scale with sharps     = not implemented
  Major scale with flats      = not implemented
  Minor scale with sharps     = not implemented
  Minor scale with flats      = not implemented
  Dorian mode                 = not implemented
  Mixolydian mode             = not implemented
  Lydian mode                 = not implemented
  Phrygian mode               = not implemented
  Locrian mode                = not implemented
  Harmonic minor              = not implemented
  Octatonic                   = not implemented
  Hexatonic                   = not implemented
  Pentatonic                  = not implemented
  Enigmatic                   = not implemented

The Tcl repo, where problem-spec PR exercism/problem-specifications#1612 is in play:

$ bin/check_test_cases -a
Exercise list-ops
  empty list to list = not implemented

---

Reviewer Resources:

Track Policies

[glennj]

I chose gron because I couldn't figure out jq to extract the uuid and description of the test cases within the arbitrarily deeply nested cases objects.

[ErikSchierboom]

Nice! It feels like this is probably something that we should be including in the canonical data syncer. The --check option will output the files with missing test cases, but not the actual test cases. Maybe on higher verbosity levels it should display the actual missing test cases. Could you create an issue in the https://github.com/exercism/canonical-data-syncer/issues repo?

[glennj]

The last commit discarded the long script, replacing it with a version that fetches the canonical-data-syncer and calls it.

[kotp]

[glennj]

While porting the square-root exercise, I tried to use the canonical_data_syncer before I had added the exercise to config.json. the data syncer could not see that square-root as an exercise.

Get the check_test_cases script to look for configlet lint errors first.