Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Human-readable name of test case does not appear in results.json #21

Open
bryceadam1 opened this issue May 6, 2021 · 1 comment
Open
Labels

Comments

@bryceadam1
Copy link

Description
Naming test cases with a new-line character on the first line causes the name of the function to be placed in the results.json file instead of the human-readable name in the docstring of the testcase.

To Reproduce
Create a testcase and name it with a docstring that has a new-line character on the first line. This file is called tests.py.

import unittest
from gradescope_utils.autograder_utils.decorators import weight, number

class TestAutograder(unittest.TestCase):
    @weight(1.0)
    @number("1")
    def test_prob1(self):
        """
        A human-readable name.
        """
        pass

Running the above with the JSONTestRunner provided in gradescope_utils produces the following results.json:

{
    "tests": [
        {
            "name": "test_prob1 (tests.TestAutograder)",
            "score": 1.0,
            "max_score": 1.0,
            "number": "1"
        },
    ],
    "leaderboard": [],
    "execution_time": "0.00",
    "score": 10.0
}

Expected behavior
Should have produced the following results.json file:

{
    "tests": [
        {
            "name": "A human-readable name.",
            "score": 1.0,
            "max_score": 1.0,
            "number": "1"
        },
    ],
    "leaderboard": [],
    "execution_time": "0.00",
    "score": 10.0
}

Additional context
Removing the new-line character seems to fix the issue. I.e. the following produces the expected result.

import unittest
from gradescope_utils.autograder_utils.decorators import weight, number

class TestAutograder(unittest.TestCase):
    @weight(1.0)
    @number("1")
    def test_prob1(self):
        """A human-readable name."""
        pass

This also seems to only be the case inside of Gradescope's docker containers used for actually grading student submissions. This did not happen when I ran the code on my macbook.

@bryceadam1 bryceadam1 added the bug label May 6, 2021
@ibrahima
Copy link
Contributor

ibrahima commented Aug 15, 2022

Hi @bryceadam1! Sorry for the delay in getting back to you. I think this behavior is from the unittest library itself (we are simply calling test.shortDescription to populate this), so if you're seeing different behavior on your Macbook I wonder if the Python versions are different? We are now defaulting to Python 3 everywhere, and now that we have different Ubuntu base image versions newer versions might have newer versions of Python 3

If you can, please let us know if you are still seeing this issue. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants