Addition of a Reproducibility Framework #185

jwons · 2021-10-19T22:12:56Z

Description

Adding a reproducibility framework that uses Tribuo's self-describing models and OLCUT to re-train a given model. It also exposes a configuration manager allowing users to modify the configuration before re-training. This framework also can create a diff between two model provenance objects, for the purpose of comparing a model to its reproduced version.

The framework is a new module within Tribuo that is targeted at Java 16. The module contains a new class, ReproUtil.
A ReproUtil must be instantiated with either a Model or ModelProvenance object. Once instantiated, users can access the OLCUT configuration manager using a getter method. Two methods on the ReproUtil object will re-train a model.
Either reproduceFromProvennace or reproduceFromModel. The reproduceFromModel will only execute if the user passed a Model object upon initial instantiation. Both methods will retrain a model from the configuration stored within a the model's provenance; however, reproducing from a model will additionally compare metadata not stored in the provenance (feature and output domains).

Motivation

This is a new feature that allows automatic reproducibility for any Tribuo-trained model. This framework helps increase the reproducibility of data science conducted using Tribuo, and can act as a basis for an experimental framework since the framework control all aspects of a model's configuration.

…ity framework

…elProvenance or Model object

…en so data paths can be changed

…m prov with csv, from prov with csv that used traintestsplitterm from a model, and from prov where dataPath is changed

…overrideConfigurableProperties, and removed org.tribuo.* import.

… to set the invocationCount within the sync block. Allows for deterministic BaggingTraining even in parallel.

…ire ReproUtility

…cts stored as lists

…ucibility Fixing the merge conflict caused by a import in KMeansTest

Merge upstream datasource, ONNX, and test changes to reproducibility framework

…dels when creating reproducibility object

…ve warnings related to poor use of generics

…tsplitter since splitter is not configurable

…d specialized code for changing some component configurations in repro framework now that ConfigurationManager has a getter, using set operations on provenance objects keys to be more robust to different prov objects, add objects to the diff that are NOT in the interesection of two objects, add reproducibility test for CSVLoader, add test to do a prov diff where one model was trained with a transformTrainer

…ationCount implementations where the rng had to be split. Added tests in Reproducibility for Bagging Trainer. Rolled back OLCUT version requirement so it can be merged in from main branch.

…t changes for reproducibility and changes to LibSVMTrainer and documentation

Merging the conflicts from pulling from oracle/main

…tation

…in the prov are now set, added some future work comments and other documentation, and switched the diff to apply 'original' and 'reproduced' to only primitive values

… not be reproduced

…pendentMultiLabelTrainer based on Adam and Jack's bugfix

Craigacp

Let's clean up the imports, exceptions and variable names. I have some more detailed comments about the ReproUtil class which we should follow up in the threads.

...sification/Core/src/main/java/org/tribuo/classification/baseline/DummyClassifierTrainer.java

Classification/DecisionTree/src/test/java/org/tribuo/classification/dtree/TestCART.java

Common/SGD/src/main/java/org/tribuo/common/sgd/AbstractSGDTrainer.java

Core/src/main/java/org/tribuo/Trainer.java

Reproducibility/src/main/test/java/org.tribuo.reproducibility/ReproUtilTest.java

Craigacp · 2021-10-20T17:34:41Z

Reproducibility/src/main/test/java/org.tribuo.reproducibility/ReproUtilTest.java

+        MutableDataset datasetFromCSV = new MutableDataset<Label>(csvSource);
+
+        LogisticRegressionTrainer trainer = new LogisticRegressionTrainer();
+        LinearSGDModel model = (LinearSGDModel) trainer.train(datasetFromCSV);


I guess we should just fix LogisticRegressionTrainer so LinearSGDTrainer returns a LinearSGDModel.

Reproducibility/src/main/test/java/org.tribuo.reproducibility/ReproUtilTest.java

Craigacp · 2021-10-20T17:41:20Z

So it looks like the tests directly pass on Java 16 which is good, and I'll poke at the CI a little to figure that out.

…nused imports, and more.

…ntout....

…static block, can add pretty printing options now

Craigacp

Few more small things.

Common/SGD/src/main/java/org/tribuo/common/sgd/AbstractSGDTrainer.java

Reproducibility/src/main/java/org/tribuo/reproducibility/ReproUtil.java

Craigacp · 2021-10-22T19:23:16Z

Reproducibility/src/main/java/org/tribuo/reproducibility/ReproUtil.java

-                    val_diff.put(NEW, ((PrimitiveProvenance<?>) provMapB.get(key)).getValue().toString());
-                    report.set(key, val_diff);
+                    ObjectNode valDiff = mapper.createObjectNode();
+                    valDiff.put(OLD, ((PrimitiveProvenance<?>) provMapA.get(key)).getValue().toString());


Pattern match? provenanceA instanceof PrimitiveProvenance<?> primProvA then use primProvA to make the valDiff.put shorter?

Now pattern matches

Reproducibility/src/main/test/java/org.tribuo.reproducibility/ReproUtilTest.java

… licenses in repro module, cleaned up exceptions, fixed imports, and other small code review changes

Craigacp

One small thing on the exception signatures. I'll work on the CI today then we can get that fix in and merge.

Reproducibility/src/main/java/org/tribuo/reproducibility/ReproUtil.java

…lude runtime exceptions. DiffProvenance string now written by ObjectMapper

…ure diff, output diff, and provenance diff

Craigacp

Two small things, then I think we're done. I'll tidy up the javadoc once this has merged, and I'll work on the CI tomorrow morning.

Reproducibility/src/main/java/org/tribuo/reproducibility/ReproUtil.java

Craigacp · 2021-10-27T00:30:44Z

Reproducibility/src/main/java/org/tribuo/reproducibility/ReproUtil.java

@@ -371,14 +435,25 @@ public ConfigurationManager getConfigurationManager(){

        return newModel;
    }
+
+    public record FeatureDiff (Set originalFeatures, Set reproducedFeatures){}
+    public record OutputDiff (Set originalOutput, Set reproducedOutput){}


These sets need a generic type.

I think these are the right types:

public record FeatureDiff (Set<String> originalFeatures, Set<String> reproducedFeatures){} public record OutputDiff<T extends Output>(Set<T> originalOutput, Set<T> reproducedOutput){}

<T extends Output<T>>

Craigacp

Great work Joe.

jwons added 28 commits June 24, 2021 14:41

Added setInvocationCount method to trainers for use by a reproducibil…

d316475

…ity framework

Added a reproducibility utility that can reproduce a model from a Mod…

4b6a7b6

…elProvenance or Model object

added optional parameter to allow a DataSource config to be overwritt…

6a88f8c

…en so data paths can be changed

Refactored ReproUtil to not be static

e2ab826

Added tests for reproducibility utility that test for reproducing fro…

134c98d

…m prov with csv, from prov with csv that used traintestsplitterm from a model, and from prov where dataPath is changed

add configManager getter to ReproUtil, tested that it can be used to …

4da9efa

…overrideConfigurableProperties, and removed org.tribuo.* import.

Added overloaded train method that takes an invocation count using it…

af5bec6

… to set the invocationCount within the sync block. Allows for deterministic BaggingTraining even in parallel.

added diff feature to ReproUtility and added more comments to the ent…

1b0fda0

…ire ReproUtility

refactor and comments on ReproUtil, and diffs will now diff prov obje…

e4c256a

…cts stored as lists

Merge branch 'oracle:main' into reproducibility

30239a5

fixed import merge conflict in KMeansTest

cc8e4e2

git push origin reproducibilityMerge branch 'oracle-main' into reprod…

5f255ce

…ucibility Fixing the merge conflict caused by a import in KMeansTest

Merge pull request #1 from oracle/main

ebed123

Merge upstream datasource, ONNX, and test changes to reproducibility framework

removed unneeded reflection to diff provenance, check for external mo…

5de9dc6

…dels when creating reproducibility object

Checks datasource is configurable before recovering, starting to remo…

0601d14

…ve warnings related to poor use of generics

Check if datasource is configurable after checking if it's a traintes…

245e96b

…tsplitter since splitter is not configurable

Merge branch 'oracle:main' into reproducibility

67f5453

Fixed bug where trainInvocationCounter would not be reset in setInvoc…

a950f37

…ationCount implementations where the rng had to be split. Added tests in Reproducibility for Bagging Trainer. Rolled back OLCUT version requirement so it can be merged in from main branch.

Fixed merge conflicts mostly relating to combining new invocationCoun…

0788f63

…t changes for reproducibility and changes to LibSVMTrainer and documentation

Merge branch 'oracle-main' into reproducibility

b70c94a

Merging the conflicts from pulling from oracle/main

Attempting to finish merge

2477ee7

Merge branch 'oracle-main' into reproducibility

68af149

Some styling changes, fixed final merge issue, and some extra documen…

2f2eb86

…tation

In reproducibility framework, the invocation counts for ALL trainers …

edef8c0

…in the prov are now set, added some future work comments and other documentation, and switched the diff to apply 'original' and 'reproduced' to only primitive values

Fixed bug in reproducibility framework where Transform trainers could…

c560ab9

… not be reproduced

Merging changes

c4705cd

merged upstream tribuo changes, applied setInvocation changes to Inde…

14939f0

…pendentMultiLabelTrainer based on Adam and Jack's bugfix

Craigacp self-assigned this Oct 20, 2021

Craigacp added the OCA signed This PR is from a person/organisation who has signed the OCA label Oct 20, 2021

Craigacp added the squash-commits Squash the commits when merging this PR label Oct 20, 2021

Craigacp requested changes Oct 20, 2021

View reviewed changes

jwons added 5 commits October 20, 2021 15:43

initial round of feedback, fixing generics, exceptions, print outs, u…

32b8148

…nused imports, and more.

Made some exceptions more explicit

dde0074

Added exception I thought I did yesterday that turned out to be a pri…

65b234d

…ntout....

removed more snake_case

fcb5015

Brought object mapper up to a static variable and initialize it in a …

c0afdc8

…static block, can add pretty printing options now

Craigacp requested changes Oct 22, 2021

View reviewed changes

jwons added 3 commits October 22, 2021 16:07

simplified default train method with invocationCount parameter, fixed…

6f503da

… licenses in repro module, cleaned up exceptions, fixed imports, and other small code review changes

added unneeded check for dataset provenance, and removed completed todos

13c0473

Diff will now add lists that only appear in one model

08586d3

Craigacp requested changes Oct 26, 2021

View reviewed changes

Reproducibility/src/main/java/org/tribuo/reproducibility/ReproUtil.java Outdated Show resolved Hide resolved

jwons added 3 commits October 26, 2021 14:25

Cleaned up exceptions, begin filling out javadocs particularly to inc…

13a8a91

…lude runtime exceptions. DiffProvenance string now written by ObjectMapper

Moved test resources

de98f3a

Changed reproduceFromModel return type to record that has model, feat…

3a7d19e

…ure diff, output diff, and provenance diff

Craigacp requested changes Oct 27, 2021

View reviewed changes

jwons added 3 commits October 27, 2021 12:43

removed rogue import and specified Set types in diff records

f0760fc

only build reproducibility module with java 16

0383ba4

fixed generic in diff record

0e7cd6d

Craigacp marked this pull request as ready for review October 27, 2021 21:13

Craigacp approved these changes Oct 27, 2021

View reviewed changes

Craigacp merged commit 57c4c1e into oracle:main Oct 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addition of a Reproducibility Framework #185

Addition of a Reproducibility Framework #185

jwons commented Oct 19, 2021

Craigacp left a comment

Craigacp Oct 20, 2021

Craigacp commented Oct 20, 2021

Craigacp left a comment

Craigacp Oct 22, 2021

jwons Oct 22, 2021

Craigacp left a comment

Craigacp left a comment

Craigacp Oct 27, 2021

jwons Oct 27, 2021

Craigacp Oct 27, 2021

jwons Oct 27, 2021

Craigacp left a comment

Addition of a Reproducibility Framework #185

Addition of a Reproducibility Framework #185

Conversation

jwons commented Oct 19, 2021

Description

Motivation

Craigacp left a comment

Choose a reason for hiding this comment

Craigacp Oct 20, 2021

Choose a reason for hiding this comment

Craigacp commented Oct 20, 2021

Craigacp left a comment

Choose a reason for hiding this comment

Craigacp Oct 22, 2021

Choose a reason for hiding this comment

jwons Oct 22, 2021

Choose a reason for hiding this comment

Craigacp left a comment

Choose a reason for hiding this comment

Craigacp left a comment

Choose a reason for hiding this comment

Craigacp Oct 27, 2021

Choose a reason for hiding this comment

jwons Oct 27, 2021

Choose a reason for hiding this comment

Craigacp Oct 27, 2021

Choose a reason for hiding this comment

jwons Oct 27, 2021

Choose a reason for hiding this comment

Craigacp left a comment

Choose a reason for hiding this comment