Removes Apache Commons Math

[Craigacp]

Description

Add replacement functionality for the things Tribuo used from Apache Commons Math 3.6.1 & migrate all Tribuo uses over to that functionality.

Replaced functionality:

• Gamma function used to compute normalized mutual information & chi squared CDF. The gamma function is ported to Java from fdlibm.
• Cholesky factorization used when sampling from multivariate normal.
• LU factorization used when building sparse linear models.
• Eigenvalue decomposition used when sampling from multivariate normal (to preserve compatibility with Apache Commons Math's multivariate normal sampler).
• Multivariate Gaussian distribution (at the moment only supports sampling, but we will look at adding pdf computation later).

As a result there has been a slight rewrite of the clustering data generators, some methods in the information theory package, and a more drastic rewrite of the sparse linear models code which extensively used ACM's linear algebra package.

The various user facing methods produce approximately the same answers as before, though there are small differences at the level of numerical precision due to different algorithms or the ordering of floating point operations. The sparse linear models code is now much faster as it properly caches the matrix inverse used in several parts of the algorithm.

Note applications which transitively pulled in Apache Commons Math via Tribuo will now need to add an explicit dependency on it.

Motivation

Migrating away from commons math will make our release process simpler, and migrating to our own linear algebra everywhere means when we vectorise it everything in Tribuo will benefit. Also the work to migrate the sparse linear models to Tribuo's linear algebra made it easy to spot that we inverted the same matrix three times, and now we only do it once so it is much faster.

[pogren]

This method adds 30+ lines of code but is never used. Probably harmless given that its in test code - but still a bit suspicious.

[Craigacp]

Ahh, I frequently use that kind of method when I'm comparing implementations, but I kept taking it back out again and then having to remember how to re-implement it (or fake it up using watches in the debugger). So this time I just committed it. I agree it's unused most of the time which is annoying.

[pogren]

This method was formerly a private method in ClusteringMetrics and is now a public method here - seems like it deserves a unit test. Three options seems reasonable - generate some test data using a method from a similar library (suggestions welcome!), generate some regression data with this method and test on that (tautological, I suppose, but at least you will know if the behavior changes), or work out some simple examples by hand. Thoughts?

[Craigacp]

Sure, I can work out some examples to test.

[pogren]

FYI - this method does not work for odd values of degreesOfFreedom.

from scipy.stats import chi2
print(chi2.cdf(3.84,3))

maybe that's why this method is private....

[pogren]

works fine for odd values, btw.

@Test
void testComputeChiSquaredProbability() throws Exception {
// assertEquals(0.9499564787512949, InformationTheory.computeChiSquaredProbability(1, 3.84), 1e-14);
assertEquals(0.8533930378696499, InformationTheory.computeChiSquaredProbability(2, 3.84), 1e-14);
// assertEquals(0.7207323828813903, InformationTheory.computeChiSquaredProbability(3, 3.84), 1e-14);
assertEquals(0.5719076705793775, InformationTheory.computeChiSquaredProbability(4, 3.84), 1e-14);
assertEquals(0.9995692574594243, InformationTheory.computeChiSquaredProbability(2, 15.5), 1e-14);
}

[Craigacp]

Yeah, I need it to compute the GTest but don't want to expose it out as I've not checked that it's valid for other use cases. I'd like to add a fuller stats package to Tribuo at some point, but in the meantime these functions will just appear as private things that we need for various hypothesis tests that we plan to add.

[pogren]

• ClusteringMetrics is a straightforward refactor that uses the same expectedMI method as before except now it is a public method in InformationTheory - added test for adjustedMI which helped uncover a discrepancy in expectedMI which was updated.
• InformationTheory added test for mi and entropy
• ClusteringDataGenerator and GaussianClusterDataSource are straightforward refactors assuming the new MultivariateNormalDistribution is a good/equivalent substitution for the ACM impl.
• SparseLinearModel just changes a couple parameter names and adds a bit of javadoc.
• DenseVector I added a unit test for meanVariance() and the new reduce method that takes a BiFunction parameter.
• Gamma - added unit test for Gamma.gamma using output from scipy and various examples
• DenseMatrix - added additional unit tests for lu factorization, cholensky, and eigendecomp, setColumn, selectColumn,
  • requested change to LUFactorization.l and .u to lower and upper
• DenseSparseMatrix - added simple unit tests for createIdentity, createDiagonal, getColumn
• Matrix interface looks fine.
• The trainers all look like they have been straightforwardly refactored to use the new apis and the tests demonstrate that you can kick them into motion and they do something reasonable.