-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Random seed not implemented for Orthogonal Initializer #7376
Labels
type:feature
New feature or request
Comments
cosminc98
pushed a commit
to cosminc98/tfjs-seeded-orthogonal-initializer
that referenced
this issue
Feb 14, 2023
…d use QR decomposition method instead of gramSchmidt for consistency with the python implementation
cosminc98
added a commit
to cosminc98/tfjs-seeded-orthogonal-initializer
that referenced
this issue
Feb 14, 2023
…d use QR decomposition method instead of gramSchmidt for consistency with the python implementation
A similar issue I have just discovered: #5955 |
cosminc98
pushed a commit
to cosminc98/tfjs-seeded-orthogonal-initializer
that referenced
this issue
Feb 15, 2023
… variables in Orthogonal Initializer apply method
cosminc98
added a commit
to cosminc98/tfjs-seeded-orthogonal-initializer
that referenced
this issue
Feb 15, 2023
… variables in Orthogonal Initializer apply method
mattsoulanille
pushed a commit
that referenced
this issue
Feb 16, 2023
…gramSchmidt (#7377) Allow seeding the Orthogonal Initializer and add test. Use QR instead of gramSchmidt for consistency with the python implementation.
Fixed by #7377. Thanks for the PR! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System information
Describe the feature and the current behavior/state.
For reference, if you try to use the Orthogonal initializer with a seed you would get the following error:
Random seed is not implemented for Orthogonal Initializer yet
Current implementations:
The current Orthogonal initializer in TFJS does not use a seed (which is needed to reproduce results across multiple training sessions) to initialize the RNG. Also, the TFJS version uses tf.linalg.gramSchmidt instead of tf.linalg.qr like the Python version (this is mostly a matter of consistency). There are two main modifications to be made:
Will this change the current api? How?
The current API does not change. The signature of the apply method stays the same. Only the implementation itself and some tests need to change.
Who will benefit with this feature?
People training models with weights initialized by the Orthogonal initializer as the method of generating the values matches the one from python (it should thus be easier to replicate results from python) and is reproducible given a fixed seed (should allow for less variability between training sessions by starting with the same weights which is useful for research / debugging).
The text was updated successfully, but these errors were encountered: