Result reporter tool

[jayvdb]

The coala json format provides a stable output.

There are other formats which are useful for reporting tests and failures.

Most commonly used is JUnit format. (which isnt really a specified format; see https://blog.ilias.xyz/unit-test-output-formats-a-state-of-affairs-5734b4c8e577)

Converting the static analysis results into a test results format has been done a few times, such as early PEP8 plugins to Jenkins. The mapping isnt exact, but the benefits of using the test result format is tight integration with various system.

CircleCI can accept this format: https://circleci.com/docs/2.0/collect-test-data/

AppVeyor can receive a test results XML format, in the format output by xunit, presumably the same as JUnit.
https://www.appveyor.com/docs/running-tests/#uploading-xml-test-results

The first project phase could involve converting the JSON to JUnit format and uploading into Circle CI and Appveyor.

The JUnit format should also be tested with a Jenkins install, to ensure it is uploading correctly there. This would mean our Jenkins project is much less important.

Another common format is TAP.

And there is Clover XML format.

Then there are some system, such as Phabricator, which have a custom format they expect.
c.f. facebook/infer#673

The above probably doesnt fill 3 months.

Pulling the coala results from CI using this format could be a way to inform newbies of the style problems with their PRs.

Another way to expand this project is to also push the unit test results as XML to the CI (using combination XML files if necessary)

[jayvdb]

See also https://github.com/Lothiraldan/litf

[jayvdb]

There is also the LSP diagnostics format, which has been worked on at

https://github.com/coala/coala-vs-code/blob/6648e76/coala_langserver/diagnostic.py

[jayvdb]

We have a new repo for it https://gitlab.com/coala/coala-json

[srivama]

@jayvdb Is this project proposal explicitly for GSOC, or can we work and complete it outside of it?

[jayvdb]

See coala-bears> git grep -i json bears for bears using json formats, and look for similar strutures.

@jayvdb Is this project proposal explicitly for GSOC, or can we work and complete it outside

There is at least one projet in this, maybe two. And lots of little related bugs and improvements that are available for anyone to work on.

[AkshJain99]

do we need to develop plugins for each CI and each format mentioned in the project?

[KVGarg]

@jayvdb @li-boxuan Since CI tools will help us in reporting tests and failures, and in that case time factor plays a vital role so won't it be good if we add execution time for each section in the coala output results?

[AkshJain99]

Yes we can add that, and many other result reporter tool as mentioned in example , too shows execution time

…

On Tue, 19 Mar 2019, 11:28 am Keshav Garg, ***@***.***> wrote: @jayvdb <https://github.com/jayvdb> @li-boxuan <https://github.com/li-boxuan> Since CI tools will help us in reporting tests and failures, and in that case *time factor* plays a vital role so won't it be good if we add execution time for each section in the coala output results? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#555 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcS9h66cGg4gztpO4Xj0OxW-oXfqWSwmks5vYHycgaJpZM4TYGAI> .

[li-boxuan]

Since CI tools will help us in reporting tests and failures, and in that case time factor plays a vital role so won't it be good if we add execution time for each section in the coala output results

This sounds like a plan. You can also mention that in your proposal 👍

[chay2199]

Since CI tools will help us in reporting tests and failures, and in that case time factor plays a vital role so won't it be good if we add execution time for each section in the coala output results?

If we observe all the given test result formats carefully then we can find many such attributes that we can add to coala result and thus we can change our coala result probably the one last time before the final mapping to various test results.

[KVGarg]

Yes @chay2199 I've been already analyzing all such attributes that can help us.
Thanks for help 😇 👍

[KVGarg]

@li-boxuan test part of uploading results to CI tools is not to be integrated with the repository, right? that is, we need to just add documentation for developers on how can they upload the results to it, right?

[AkshJain99]

@li-boxuan as we need to create tool which convert json to junit format but also there are existing tool present which can do so cant we integrate them only or make them only to support coala?

[li-boxuan]

as we need to create tool which convert json to junit format but also there are existing tool present which can do so cant we integrate them only or make them only to support coala?

Be specific. Which tool? You can write down in your draft proposal with examples and why you believe they can solve our problem.

[AkshJain99]

okay I am writing down that in my draft proposal and will submit for review today only or at max tomorrow
thanks :)

[Dhiraj240]

@li-boxuan i used csv to json converter like this in my project.

Although this may sound silly do we need to use already available tools like this but its dependency is npm .Also which results are we talking about in phase 1 upload coala results into Circle CI and Appveyor is the the converted json results i.e Junit ? which will then reside in coala-json repo.

Also, jenkins xunit plugin can use xml check this
l found something https://github.com/EduardFazliev/rally_json2junit

Some information about this Rally is ageneric testing tool that automates and unifiesmulti-node OpenStack deployment, verification, testing &profiling.  It can be used as a basic tool for an OpenStack CI/CD systemthat would continuously improve its SLA,performance and stability.

But these may not serve the purpose i think we need to develop the tool our self using the above open sourced available tools.
do we need to create a new repositry in coala for this tool ?
I will making proposal on this project.

[AkshJain99]

@Dhiraj240 r u going to submit the proposal for this idea?

[Dhiraj240]

Yes.I am not getting gut feeling for my ANTLR idea as of now.

[li-boxuan]

Although this may sound silly do we need to use already available tools like this but its dependency is npm

It's not a good idea. First, it's a node package instead of a python package. Second, it might be out of date. Third, it does not have tests. Last, it is not very complex, so why not write a python version?

Also which results are we talking about in phase 1 upload coala results into Circle CI and Appveyor is the the converted json results i.e Junit

Yes. Please also read #704 (review)

do we need to create a new repositry in coala for this tool?

This has been answered in #704