Adds a multi-label linear sgd classifier #106
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…to make the tests pass.
- Adds the gradient and loss to BinaryCrossEntropy (formerly Sigmoid). - Adds the loss to Hinge, and refactors the gradient computation. - Adds a reduction method to the SGDVector interface.
…ns when performing operations with dense arguments.
… in preparation for sharing code between the trainers.
… class. Note this commit changes the serialization format for all the models. Compatibility with the 4.0 serialised models will be restored later.
…earSGDModel. Adding a test for 4.0 models to the linear SGD package in classification and regression.
…bclasses. Suppressing the unchecked array creation warning, we know it's safe.
Classification/SGD/src/main/java/org/tribuo/classification/sgd/objectives/Hinge.java
Outdated
Show resolved
Hide resolved
eelstretching
requested changes
Dec 16, 2020
eelstretching
requested changes
Dec 17, 2020
Common/SGD/src/main/java/org/tribuo/common/sgd/AbstractLinearSGDTrainer.java
Outdated
Show resolved
Hide resolved
| /** | ||
| * Copies the supplied matrix. | ||
| * @param other The matrix to copy. | ||
| */ |
There was a problem hiding this comment.
Just curious why we're not using arraycopy in this constructor?
There was a problem hiding this comment.
Because it's not final and ShrinkingMatrix and AdaGradRDAMatrix both subclass DenseMatrix and mess with the get method to apply a transformation as part of the regularisation during training. I guess I could check if it's only a DenseMatrix without any other classes and then do an arraycopy, falling back to this code otherwise, but it doesn't seem worth it at the moment.
MultiLabel/SGD/src/main/java/org/tribuo/multilabel/sgd/linear/LinearSGDTrainer.java
Show resolved
Hide resolved
Common/SGD/src/main/java/org/tribuo/common/sgd/AbstractLinearSGDTrainer.java
Outdated
Show resolved
Hide resolved
|
@eelstretching any thoughts on using the Pair vs the concrete class to return the gradient and the loss? |
eelstretching
previously approved these changes
Dec 17, 2020
|
I removed the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a native multi-label linear SGD model, refactors all the linear sgd models to share a base class, and adds a bunch of additional support to the math library for dense vectors (in preparation for further optimizations to the linear sgd models). It also switches over the vector normalizers to have an in place normalization method to reduce allocations in the inner loop of training and inference.
Motivation
The multi-label package includes a wrapper to convert any classifier into a multi-label classifier, but this wrapper is too slow for large applications. The multi-label linear sgd classifier in this PR is an order of magnitude or so faster when ran on a large text corpus. It also preps the linear models package for further work introducing vectorisation, l2 regularisation and other improvements.