-
Notifications
You must be signed in to change notification settings - Fork 21
Developer Notes
To compile Toradocu, run the command: ./gradlew shadowJar
This will create the file build/libs/toradocu-1.0-all.jar.
Notice that building Toradocu requires Java JDK 1.8+.
It takes a long time to compile Toradocu with the command ./gradlew shadowJar, largely because of the time to create the massive final
.jar file. Once you have made the shadowJar file, if you edit the
Toradocu source code, it is much faster to run ./gradlew jar to
create file build/libs/toradocu-1.0.jar.
Using this method, you need to slightly change the way that you run Toradocu:
java -cp toradocu-1.0.jar:toradocu-1.0-all.jar org.toradocu.Toradocu ARGS
You should use this technique if:
- you have already made the shadowJar in the past,
- you have changed Toradocu's source code, and
- you have not changed Toradocu's external dependencies. Whenever you change Toradocu's external dependencies, you need to remake the shadowJar.
To run a single precision/recall test:
./gradlew test --tests org.toradocu.<test suite name>.
For example: ./gradlew test --tests org.toradocu.PrecisionRecallGuava19
To run all the precision/recall tests, use the command:
./gradlew test --tests 'org.toradocu.PrecisionRecall*'
Precision and recall values are printed to standard output during a
precision/recall test. A complete test report with the standard output and
standard error produced by each executed test case can be found in
build/reports/tests/test/index.html.
The precision/recall test suites are always run during ./gradlew build. This
is primarily to find regression errors during continuous integration.
If the tests fail, and you don't know exactly why, then you might want to know how your changes to Toradocu are affecting the output. You can do so by generating output from the original version of Toradocu, save it, and compare that to the current output.
To see the goal and actual output for each test where Toradocu does not produce the goal output:
./gradlew test --tests 'org.toradocu.PrecisionRecall*' | grep "condition\. Comment"
Adding new test cases requires both the source code and a .jar file containing compiled .class files for the system to test.
A way to install these is to update the
{download,extract}{Sources,Binaries} Gradle tasks which will be run before tests.
(Alternately, you can manually
place the .jar file or a directory containing binaries of the class to test in src/test/resources/bin/, and
place the source files of the class to test in src/test/resources/src/.)
The precision/recall test suite works by comparing the actual and goal output from Toradocu's condition translator.
First, create an empty directory
src/test/resources/goal-output/SYSTEM-NAME/, such as src/test/resources/goal-output/plume-lib-1.1.0/.
The condition translator output is formatted as JSON. You will need to create goal output files in the directory you created above. These files can be created using a command like the following:
java -jar build/libs/toradocu-1.0-all.jar \
--class-dir src/test/resources/bin/plume.jar \
--source-dir src/test/resources/src/plume-lib-1.1.0/java/src \
--target-class plume.ArraysMDE \
--condition-translator-output src/test/resources/goal-output/plume-lib-1.1.0/plume.ArraysMDE_goal.json
Then, edit this file to indicate goal output.
To make this editing task easier, when the
--condition-translator-output
option is used, Toradocu also prints (to standard output) the line
numbers of all lines that may have to be modified within the actual output
file in order to form the goal output file.
Empty conditions in goal output do not affect precision and recall. In other words, when a goal condition is empty, the actual output of Toradocu is not checked. This is useful when the goal condition is too hard, or impossible, to derive.
Create a new class in directory toradocu/src/test/java/org/toradocu
that extends AbstractPrecisionRecallTestSuite.
Within the constructor for this class, call the superclass's constructor, passing in the path to the sources, the path to the binaries, and the path to the goal output files. See the Javadoc for AbstractPrecisionRecallTestSuite for more details on the constructor parameters.
Then, for each class that has an goal output file, create a JUnit test case, such as:
@Test
public void testConcurrentHashMultiset() throws Exception {
test("com.google.common.collect.ConcurrentHashMultiset", 0.818, 0.692);
}To create a summary of the Toradocu's precision/recall, run
./stats/precision_recall_summary.sh current
This will produce a CSV file (using comma as separator). The path of the CSV summary is printed on the standard output at the end of the script execution. You can use your favorite spreadsheet program to open the file and read the results.
At the beginning of the script execution you might be asked whether you want to remove the existing (old) summary:
Reply y to continue or stop the execution to avoid the deletion of the
existing summary.
Each line in the summary has the following format:
METHOD,DISTANCE THRESHOLD,REMOVAL COST,CORRECT THROWS CONDITIONS,WRONG THROWS CONDITIONS,MISSING THROWS CONDITIONS,THROWS PRECISION,THROWS RECALL,CORRECT PARAM CONDITIONS,WRONG PARAM CONDITIONS,MISSING PARAM CONDITIONS,PARAMPRECISION,PARAM RECALL,CORRECT RETURN CONDITIONS,WRONG RETURN CONDITIONS,MISSING RETURN CONDITIONS,RETURN PRECISION,RETURN RECALL,CORRECT CONDITIONS,WRONG CONDITIONS,MISSING CONDITIONS,PRECISION,RECALL
At the end of the summary there is an overall report with:
- Number of methods analyzed with Toradocu.
- Number of @param, @return, and @throws comments analyzed with Toradocu.
- Toradocu's precision and recall on @param, @return, and @throws comments.
- Overall Toradocu's precision and recall.
To create a summary of the precision/recall of Toradocu 0.1 (old and discontinued version) run the following commands
git clone http://github.com/albertogoffi/toradocu.git
cd toradocu && git checkout version0.1
./precision_recall_summary.sh
Results are saved in the CSV file results.csv. Each line in the summary has the following format:
METHOD,CORRECT CONDITIONS,WRONG CONDITIONS,MISSING CONDITIONS
Since Toradocu v0.1 was supporting only @throws comment, those numbers in the report file refer only to that kind of comments.
At the bottom of the file there is an overall report with:
- Number of methods analyzed with Toradocu.
- Number of @throws comments (conditions) analyzed with Toradocu.
- Toradocu's precision and recall @throws comments.
It is possible to create a summary of the precision/recall using @tComment as condition translator. To create the summary run the script:
./stats/precision_recall_summary.sh tcomment
This script works in the same way as the standard precision/recall script. Please, refer to the previous section for more detail about the format and the content of the ouput file.
To compare the results of Toradocu with different parameters configuration, run the following commands
./stats/parameters_check.sh config_check_results.csv 0 6
python stats/parameters_check.py config_check_results.csv >config_check_summary.csv
The first command runs Toradocu on the precision/recall test suite, multiple times with different parameters configuration. The script takes as input a file where to store results, lower bound and upper bound of edit distance threshold to use. The values to use for the removal cost are automatically derived from the values of edit distance threshold.
The second command creates a summary of the results for an easy inspection. Open the summary file with your favorite spreadsheet program to see the overall precision and recall obtained with different parameter configurations.
The source code of this project follows the Google Java Style Guide. The style of the code is automatically checked by a Git pre-commit hook. Also, the hook prevents any commits of files containing trailing spaces. The hook is automatically installed as soon as any Gradle task is executed.
If a commit fails due to improper formatting in a source file, you can format your code according to the style guide using the following command:
./gradlew goJF