Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix default for intermediate quadratic layers in ICNN #587

Merged
merged 1 commit into from
Oct 9, 2024
Merged

Conversation

marcocuturi
Copy link
Contributor

@marcocuturi marcocuturi commented Oct 9, 2024

The current diag-quadratic part of ICNN at intermediate layers is not producing expected effect. Essentially all diag terms are initialized and stay at 0.

This simple fix changes two things:

  • set initializer value for all elements of diag to -2.
  • set rectifier_fun to jax.nn.softplus to have differentiability everywhere. jax.nn.softplus(-2.0) is roughly 0.1 as well as its derivative

Copy link

codecov bot commented Oct 9, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.00%. Comparing base (7309c76) to head (f58fa93).
Report is 18 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #587   +/-   ##
=======================================
  Coverage   88.00%   88.00%           
=======================================
  Files          73       73           
  Lines        7820     7820           
  Branches      567      567           
=======================================
  Hits         6882     6882           
  Misses        789      789           
  Partials      149      149           
Files with missing lines Coverage Δ
src/ott/neural/networks/icnn.py 94.54% <ø> (ø)

@michalk8 michalk8 self-requested a review October 9, 2024 17:40
@michalk8 michalk8 added the bug Something isn't working label Oct 9, 2024
@michalk8 michalk8 merged commit 0219671 into main Oct 9, 2024
11 of 12 checks passed
@michalk8 michalk8 deleted the fix-ICNN branch October 9, 2024 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants