Porting CodeChecker to Windows

[gyorb]

Windows port requirements and limitations

Dependencies

• Python available
• Psycopg2 available
• Pg8000 available
• Python-ldap available
• Python-pam Linux only
• PostgreSQL available
• Thrift compiler available
• Clang/Clang-tidy available
• Curl available (to download other js/python dependencies)
• ...

Compile Command database:

Build command logging is available for

• Linux (CodeChecker built in ld logger or scan-build/intercept-build by @rizsotto)
• OSX (scan-build/intercept-build)
• Windows (scan-build/intercept-build windows version)

You can read more about compilation databases here by @eliben and here by @Sarcasm .

Possible solutions to create compile command database on Windows

• Use scan-build/intercept-build to create compile command database
• CMake projects can export compilation command json which can be processed by CodeChecker.
• analyze the project with clang and generate plist output which can be imported by CodeChecker. (clang-tidy does not provide plist output right now)
• ...

[whisperity]

There isn't seems to be, and theoretically never will be a pam module for Windows, as PAM is a nonexistent concept on Windows machines (user authentication happens over an AD-like system, either to a domain controller (which is an AD provider), or a local loopback).

[rizsotto]

Hi there, [scan-build](https://github.com/rizsotto/scan-build) is works and tested on Windows. See appveyor builds.

[gyorb]

Thanks @whisperity for the comment!

Great news that scan-build works on Windows too, thanks @rizsotto!

I've updated the issue description.

[whisperity]

🙂 You should also add, just for the sake of completeness, that SQLite is also available on Windows. Also various other required (in the full package) Python packages such as portalocker and psutil also work.

[irishrover]

By the way, CodeChecker works in Ubuntu from Windows 10's Linux subsystem!
I've just followed the installation instructions from the CodeChecker homepage.

Wow, even web-server with errors visualization (CodeChecker server) works!

[gyorb]

@irishrover thanks for trying it out, great news!

[lmerynda]

Maybe a good first step would be to enable codechecker cmd to connect with the server from windows. That would already be a good feature on its own. For example someone could connect to the web app and download results in the csv form and import it in the excel to fool around (managers like the excel way!) Or is it already possible to do that from the web app itself (and I don't mean the summary only, but the findings)?

[gyorb]

Hi,

The CodeChecker cmd can connect to a remote server and get the results in various formats (json, csv, simple table) or if you are interested only in the summary you can do that too.
The checker statistics can be exported from the web app.

Under the Windows 10 Linux subsystem you should be able to run the commands, but let us know if there are some problems with it.

[lmerynda]

That is what I'm using on daily basis: codechecker_cmd results <run_name> --url <private_url> -o json, and I'm doing that from my linux terminal and it works great (with filters and other features as well). The point I had is that I can't do that from windows that easily. Ofc one can use windows 10 linux subsystem as you pointed out, but there are still the same limitations for non-technical people.

The use case I was talking about is to make it possible for non-tech to import codechecker results from the server to the excel sheet (excel usually means windows environment).

[irishrover]

That is what I'm using on daily basis: codechecker_cmd results <run_name> --url <private_url> -o json, and I'm doing that from my linux terminal and it works great

Hi. If you have an access to the server, why don't you use a web-interface for watching the results?

[lmerynda]

I do. It's the managers who put things into the excel files, then visualize them in charts to finally copy those charts into power point presentation to show on managers meetings

[lmerynda]

Sorry, my last comment doesn't answer your question. Depends on what I want to get from the results. If I have several products (obviously each one is in a separate run) and each of them is using the same library (let's say it's c++ boost library, which is likely to be used in many projects). And for some sake I'm interested in how many issues do we have in our products that have something to do with the library how do I get this information in other way? What I can do today is to download result from all runs with applied filters (good feature btw!) and then aggregate them myself. Isn't it a valid use? :)

[gyorb]

If I understand your use case right you can try something like this.
If you store all your products (source code) in one product (CodeChecker product) but in different runs you can create a filter where you add all your runs and save a source component filter with the file paths to the same library depending on the usage of that library for that run.
The same library might be stored or included from different locations depending on your source setup.
You could create a boost_uuid_core component filter with multiple file path filters depending on their usage in the products:

*/product1/boost_1_70_0/boost/uuid/*
*/product1/boost_1_70_0/boost/core/*
*/product2/boost/boost/uuid/*
*/product2/boost/boost/core/*

Or a less product specific path filter:

*/boost/uuid/*
*/boost/core/*

[Vexthil]

Sorry for poking but is there any update on this ticket? If I am reading this correctly the only issue is python-pam is only supported on Linux. I don't know anything about Linux myself but this seems to be an authentication module so i'm assuming is only needed for the webpage reports of some sort. As long as these dependencies are managed then ActiveDirectory or even just not supporting accounts on windows should be possible as a starting point.

This is something I am assessing at the moment to see if this is a viable tool for our usage and it looks really good but without a windows version existing we would have to look elsewhere i'm afraid. We don't necessarily need windows source support(though that would be ideal) but maybe as long as a windows package could be made that may make it viable

[whisperity]

@Vexthil Yes, PAM is basically the "user authentication system" for Linux. The pam module in CodeChecker allows you to require user accounts on the server machine (or the PAM stack that is configured on that machine) and let users log in to CodeChecker via the same username and password authentication. It is not needed, it is one of the... IIRC 3 options: you can use either baked-in credentials (in the webserver configuration file, similar to .htpasswd), PAM or LDAP for authentication. You could also run the server without any authentication whatsoever!

Do you need the analysis parts of CodeChecker too, or only the web interface for bug report management? In the latter case, there are pre-built Docker images for the web server part.

[irishrover]

Please look at https://github.com/Ericsson/codechecker/blob/master/tools/report-converter/README.md
May be it's what you need or can somehow be utilized?

[Vexthil]

Thanks for the replies. Its good that you say the pam is not needed and there are other authentication options so that is great to hear. I've only got as far as reading the docs so far so I will get this and give it a go and see what we can find out.

Overall our goal is to get the clang-tidy outputs organised into a status report up on Jenkins but also have the results for local users and so the unique identifier for new issues parts look very interesting. We had started this ourselves but quickly have found issues with the clang-tidy output format and this looks like things you have solved. The report-converter may be the simplest starting point for us and then we can look at the Jenkins/webpage part in the future.

[rasjani]

@Vexthil I've gotten following workflow to work on windows:

1. generate compile_commands.json of my project with cmake
2. process that generated compile_commands.json with my own tooling to generate a clang-tidy report ( https://github.com/rasjani/processcdb )
3. Use CodeChecker's report-converter to generate reports that can be then uploaded to web-ui
4. sent the generated reports from previous step with CodeChecker store into codechecker server running on linux ..

I have one one pull request open @ #3045
And rest of the changes @ rasjani@1359f0b but the commit isn't probably fully working ..

[sriprasannaMG]

Can anyone tell me how the cross translational unit analysis is supposed to work in windows? Reading this issue the concern is mainly on the dashboarding / reporting. In our case we want the CTU analysis to work on a windows system where we have a gcc cross compilation with cmake. We do have compile command json on which clangTidy works, now we want CTU possibilities with codechecker. Any ideas?

[csordasmarton]

What happens if you run the following command on your system: CodeChecker analyze -h. Does it contain the following arguments:

cross translation unit analysis arguments:
  
  These arguments are only available if the Clang Static Analyzer supports
  Cross-TU analysis. By default, no CTU analysis is run when
  'CodeChecker analyze' is called.

  --ctu, --ctu-all      Perform Cross Translation Unit (CTU) analysis, both
                        'collect' and 'analyze' phases. In this mode, the
                        extra files created by 'collect' are cleaned up after
                        the analysis.
.....

If yes, what happens if you run the CodeChecker analyze command with the --ctu and with --verbose debug options? Can you please send us the verbose logs?

[whisperity]

@sriprasannaMG To add a little to what @csordasmarton said. Where do you obtain your Clang from? For CTU analysis, there are two things needed: the Clang binary which contains the static analyser and the necessary additional internal features in it, and the clang-extdef-mapping binary. Please check @csordasmarton's answer, if the help does not contain the quoted part, your distribution does not support CTU analysis.

If you self-compile Clang, make sure to also compile the clang-extdef-mapping target, and install the resulting clang-extdef-mapping (I suppose with .exe at the end?), not just the clang(.exe?) binary.

[whisperity]

Coming over from #3846.

@VenoVeno I think issues related to the logging of compilation databases (or the bad content of compilation databases), and the driving of the analysis, on Windows, we should report that in this ticket to keep everything in one place. @gyorb (our previous project manager) has investigated several ways to obtain a potentially good compilation database on Windows. Namely, LLVM itself ships with a tool called scan-build (and also scan-build-py, which I think is a rewriting of the original tool?), which is commonly used to drive the analysis (of Clang SA, but as far as I know, not of Clang-Tidy) long before CodeChecker existed. If you have the time, please try out these tools (independent of CodeChecker!) to see if your existing build could be passed to Clang to perform an analysis. Maybe we can obtain some information that way as to where our tooling is lacking in the "prepare for an analysis department".

Every other issue related to actually parsing Windows-targeted source files with Clang itself (both for compilation and for analysis...) should be reported directly to the LLVM Project. While many of us developers here are also contributors to LLVM, the two projects are separate, and we still do not use or have Windows computers or projects within the scope of our job (on the CodeChecker project).