Ingest Junit xml report

[jaydeepk]

Hi

Is there a way to integrate the results of a junit test report into PyTest.
The idea would be to point pytest to a junit report and pytest reading it and displaying the results.
Is this possible?

Thanks
Jaydeep

[RonnyPfannschmidt]

its not implemented and i would recommend against it, as junitxml is a very losy format that cannot be used to pretend a pytest session

there is a wish/idea to have reportlog be extended with a "replay" option but that needs manpower

[jaydeepk]

Thanks @RonnyPfannschmidt
I do get that junitxml is a lossy format, but unfortunately in my case, this is something I don't control.
Here's my situation:

In my python test, I forward the call to a proprietary testing framework (which actually executes the test) and it returns a junit xml report.

class TestApis():

    def test_api(self):
        # call propietary test framework which returns junit report

So I was trying to see if there is any way I could injest the junit xml report so that pytest can read and report it.
My strategy was to first parse the junit xml using ElementTree and then somehow try to create a Pytest TestReport for each test in the junit report
I have looked at various hooks like pytest_runtest_makereport, pytest_report_teststatus in the hopes of being able to modify/add to PyTest's report/stats, but I haven't been able to figure out the right hook method to do this.

Any thoughts on this strategy?
Is it possible to create custom TestReports and add them to PyTest stats in any of these hooks ?

[RonnyPfannschmidt]

a starting point could be to just use subtests to trigger reporting of the data

[jaydeepk]

Sorry, could you please elaborate a bit on how subtests could help in this case.
My python test is just these two lines:

class TestApis():

    def test_api(self):
        request = MyRequest()
        PropietaryTestRunner().run(request)

Internally, the PropietaryTestRunner extracts the request data and uses subprocess.Popen to call a java based command line tool to run the test. The command line tool generates a junit test report which I'm hoping that PyTest could somehow read and report.

[RonnyPfannschmidt]

There is nothing in pytest to do that

Given that you don't share the surrounding use case in any comprehensible detail im out of this

[jaydeepk]

Apologies, I should have provided more context.

I'm writing a python wrapper for an api testing too which checks if the api satisfies an Open Api spec.
The api testing tool is available as a java jar which can be executed via command line.
This tool returns test results in a junit xml file.

So lets say I have a python flask app which exposes REST end points.
The app has an open api spec file(in the project root directory) which describes all the rest end points.
I want to validate if the REST end points satisfy the open api spec in a python test (using pytest).

Hence in the pytest, I simply delegate the call to a test runner which calls the java jar.
Here's how the test runner looks:

class TestRunner:
    def __init__(self, host: str = "127.0.0.1", port: int = 5000):
        self.host = host
        self.port = port

    def run(self):
        jar_path = "<path to jar>"
        cmd = [
            "java",
            "-jar",
            jar_path,
            "test",
            "--junitReportDir=<path to junit report>",
            '--host=' + self.host,
            "--port=" + str(self.port)
        ]

        process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
        output, error = process.communicate()

        print(output.decode('utf-8'))

In conftest.py I was looking at picking up the junit test results and somehow piping them to pytest.
The best I could manage so far was to just print them to console:

@pytest.hookimpl(hookwrapper=True)
def pytest_runtestloop(session: Session):
    yield
    junit_report_path = "./junit_report.xml"
    root = ET.parse(junit_report_path).getroot()
    for testcase in root.iter('testcase'):
        scenario = testcase.find('system-out').text.split('Scenario:')[1]
        failure = testcase.find('failure')
        if failure is not None:
            print("FAIL: " + scenario)
            print(failure.get('message'))
            print(failure.text)
        else:
            print("PASS: " + scenario)
    tests = root.get('tests')
    skipped = root.get('skipped')
    failures = root.get('failures')
    passed = int(tests) - int(failures)
    print(f'Tests: {tests},  Passed:{passed},  Skipped: {skipped},  Failed: {failures}')

Please let me know if any more details are required.

[jaydeepk]

Btw thanks for the suggestion to look at subtests.
That led to me to thinking if I could dynamically create test methods based on the junit xml report.
I have kind of got it working with Python's unittest now.
I will update and add the code here when I have get it working with pytest.