moves functionality to Python (#27)

* WIP: moves functionality to Python

Eliminates all but a bash shim file, restructures more in line with the
python-representer and python-anaylzer repos.

To run just `sh bin/run.sh SLUG EXERCISE_DIR OUTPUT_DIR

* update Dockerfile to use new scripts

* Update runner/__init__.py

Co-Authored-By: Corey McCandless <[email protected]>

* Update runner/utils.py

Co-Authored-By: Corey McCandless <[email protected]>

* Refactor and self testing

Significant refactor of the main structure, now working on a test suite
for the runner itself; only problem here is it's not sufficiently
isolated, yet, from the programtically run pytest.

* adds output capture and tests

Updates to add output capturing from stdout. Also many more tests, including some examples of the various stylings available for us to choose from for traceback.

* add tests demonstrating truncation

Now demonstrating truncation of stdout output at 500 chars.

* updates Docker container testing

Updates the Dockerfile to use requirements.txt, just to be consistent
and in case we ever add more dependencies. Moves the run-in-docker
script and eliminates its use in the Github CI action. Adds some
documentation of the current state of things.

* Force Action run

* Fix formatting.

* Fix issue with missing Python... hopefully

* Updates bin/runs.sh to find the correct python

* sanitizes messages more completely

This _should_ make the `message` field more sensible for the students.

.github/workflows/main.yml

@@ -1,18 +1,22 @@
-name: Run Example
+---
+name: CI
 
-on: [push]
+on:
+  push:
+    paths-ignore:
+      - '.gitignore'
+      - 'LICENSE'
+      - '**.md'
 
 jobs:
   build:
+    name: Test Runner
     runs-on: ubuntu-latest
-
     steps:
       - uses: actions/checkout@v1
-      - name: Run python-test-runner on example files
-        run: sh run-in-docker.sh example test/example/ test/output/example
-      - name: Verify that results.json is created
-        run: test -f test/output/example/results.json && cat test/output/example/results.json
-      - name: Verify that test case order in results.json
-        run: bash verify_test_order.sh example test/example test/output/example/
-      - name: Verify exercise with multiple words in name
-        run: sh run-in-docker.sh example-with-longer-name test/example-with-longer-name/ test/output/longer-example
+
+      - name: Build Docker Image
+        run: docker build -f Dockerfile -t python-test-runner .
+
+      - name: Run Tests
+        run: docker run --entrypoint "pytest" python-test-runner

Dockerfile

@@ -1,13 +1,11 @@
 FROM python:3.7-alpine
 
-RUN pip install pytest
+COPY requirements.txt /requirements.txt
 
-WORKDIR /opt/test-runner
+RUN pip install -r /requirements.txt
 
-COPY ./run.sh ./bin/
-COPY ./process_results.py ./
+COPY . /opt/test-runner
 
-# Necessary to apply to tests run in bound directory /solution/
-COPY ./conftest.py /
+WORKDIR /opt/test-runner
 
-ENTRYPOINT [ "sh", "/opt/test-runner/bin/run.sh" ]
+ENTRYPOINT [ "sh", "/bin/run.sh" ]

README.md

@@ -3,14 +3,26 @@
 The Docker image to automatically run tests on Python solutions submitted to [exercism](https://exercism.io).
 
 ## Running the Tests
-To run a solution's tests, do the following:
-1. Open termnial in project's root
-2. Run `./run.sh <exercise-slug> <path/to/solution/folder/> <path/to/output/directory/>`
+To run all of the tests, do the following:
+1. Open a terminal in the project's root
+2. Run `pytest`
 
+These are [golden tests][golden] that compare the `results.json` generated by running the current state of the code against the "known good" `test/<test-name>/results.json`. All files created during the test run itself are discarded.
 
-## Running the Tests in Docker container
-*This script is provided for testing purposes*
+When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new `test/<test-name>/results.json` file.
 
-To run a solution's tests in the Docker container, do the following:
-1. Open terminal in project's root
-2. Run `./run-in-docker.sh <exercise-slug> <./relative/path/to/solution/folder/> <./relative/path/to/output/directory/>`
+## Generating `results.json` for an Exercise
+To run the tests of an arbitrary exercise, do the following:
+1. Open a terminal in the project's root
+2. Run `./bin/run.sh <exercise-slug> <path/to/solution/dir/ </path/to/output/dir/>`
+
+##  Generating `results.json` for an Exercise with Docker
+*This script is provided for testing purposes.*
+
+To run the tests of an arbitrary exercise using the Docker container, do the following:
+1. Open a terminal in the project's root
+2. Run `./bin/run-in-docker.sh <exercise-slug> <./relative/path/to/solution/dir/> <./relative/path/to/output/dir/>`
+
+**Note**: the solution and output directory's **MUST** be relative paths!
+
+  [golden]: https://ro-che.info/articles/2017-12-04-golden-tests

bin/run.py

@@ -0,0 +1,56 @@
+#! /usr/bin/env python3
+"""
+CLI for the test runner for the Python track on Exercism.io.
+"""
+from argparse import ArgumentParser, ArgumentTypeError, REMAINDER
+
+import runner
+import runner.utils
+
+
+def _slug(arg):
+    try:
+        return runner.utils.slug(arg)
+    except ValueError as err:
+        raise ArgumentTypeError(str(err))
+
+
+def _directory(arg):
+    try:
+        return runner.utils.directory(arg)
+    except (FileNotFoundError, PermissionError) as err:
+        raise ArgumentTypeError(str(err))
+
+
+def main():
+    """
+    Parse CLI arguments and run the tests.
+    """
+    parser = ArgumentParser(description="Run the tests of a Python exercise.")
+
+    parser.add_argument(
+        "slug", metavar="SLUG", type=_slug, help="name of the exercise to process",
+    )
+
+    parser.add_argument(
+        "input",
+        metavar="IN",
+        type=_directory,
+        help="directory where the [EXERCISE.py] file is located",
+    )
+
+    parser.add_argument(
+        "output",
+        metavar="OUT",
+        type=_directory,
+        help="directory where the results.json will be written",
+    )
+
+    parser.add_argument("pytest_args", nargs=REMAINDER)
+
+    args = parser.parse_args()
+    runner.run(args.slug, args.input, args.output, args.pytest_args)
+
+
+if __name__ == "__main__":
+    main()

bin/run.sh

@@ -0,0 +1,4 @@
+#! /bin/sh
+root="$( dirname "$( cd "$( dirname "$0" )" >/dev/null 2>&1 && pwd )" )"
+export PYTHONPATH="$root:$PYTHONPATH"
+/usr/bin/env python3 bin/run.py "$@"

pytest.ini

@@ -0,0 +1,2 @@
+[pytest]
+norecursedirs = .git .github example* traceback-styles*

requirements.txt

@@ -0,0 +1 @@
+pytest>=5.3.1