Home

For developers of pull requests / patches

How to run integration tests?

The directory testsuite contains a number of test cases consisting of a few longer documents and a larger number of small test cases tuned to test treatment of particular special cases. Before submitting pull requests please run these tests by following these steps.

1. Create the benchmark files based on the HEAD master branch:

git checkout master
cd testsuite
verify --reset

(depending on the speed of you machine, this might run a few tens of seconds)

2. Check whether your patch causes any different results:

git checkout my_new_fix
cd testsuite
verify

At the end of the run you will get a summary of the form:

test                 : ok  0.08 s (  -0.01 s)
rapine_et_al         : ok 10.36 s (   0.16 s)
island_obs2004       : ok  7.00 s (   0.48 s)
texdiffsample        : ok  0.01 s (   0.01 s)
...

If all tests report "ok", great, your patch has not caused any side effects on the existing test cases, and is probably good to go.

In case there is a fail, there is a difference in the way the master, and your patched version process the file. The verify script will echo the kdiff3 commands that you can copy/paste in order to visualise the differences (assuming you have kdiff3 installed). Of course, the differences might be neutral, or indeed your patch might have improved the processing of the existing test cases. In this case you must ensure that all the diff files compile and produce visual output at least as good or better than the unpatched latexdiff (verify --run can be used to automate part of this process). If the differences actually lead to an inferior result, you should modify your patch to avoid these negative side effects. In any case, if the test has resulted in different results for any of the test cases mention this in the description of the pull request and describe why you believe these differences actually represent an improvement to the status quo, or at least have not made matter worse.

Please also pay attention to the numbers - the first is the runtime of latexdiff for that test file (which obviously will be system-dependent), and the number in parentheses represents the difference in running time to the reference case. Small deviations are normal due to the different load experienced by the computer, but if a patch results in substantially longer run times, this would be a reason for concern.

3. Finally, if you are feeling particularly helpful you should create a new test-case that shows off the ability of your patch to cope with some new special case (for those type of patches). This requires creating an old and new file. For example, if you have written a patch dealing with some special command in a package called awesome.sty, create the files awesome-old.tex and awesome-new.tex`. Edit verifyand find the commandset testroot= .... Addawesome`` at the end of that list. If you want to, you can also specify options with which latexdiff should be run for that test (it will be obvious from the examples in the verify script how to do this).