Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add: leaky_relu non-linearity #72

Merged
merged 8 commits into from
Dec 29, 2022
Merged

add: leaky_relu non-linearity #72

merged 8 commits into from
Dec 29, 2022

Conversation

dcbuild3r
Copy link
Contributor

Add Leaky ReLU by rebasing from main

if scale_diff > 0 {
let mult = scale_to_multiplier(scale_diff);
mn.opkind = OpKind::LeakyReLU(mult as usize); // now the input will be scaled down to match
mn.output_max = input_node.output_max / mult;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also want to read in the leaky relu slope somewhere here; from the Onnx node.

@alexander-camuto
Copy link
Collaborator

alexander-camuto commented Dec 5, 2022

Hey @dcbuild3r thanks for this its looking really good.

In terms of next steps:

  • ensure non-linearity behaves as expected by generating an Onnx graph / input file for a graph with a single LeakyReLU op (with slope 0.05) and then running ezkl on it
  • once it is working you can add that example to this repo and add the example to the tests/integration_tests.rs file.
  • more broadly we probably want to think about how to generalize ingesting the slope parameter into the LeakyReLU function. It could just be item 2 of the scale array or something.
  • concurrently we also need a way to load that parameter from the Onnx node. I've added a comment where it should probably happen in the code. Onnx / tract can be kind of fiddly and have inconsistent interfaces for loading parameters so lmk if you need help with that.

@alexander-camuto alexander-camuto changed the title add(feat/leaky_relu) add: leaky_relu non-linearity Dec 9, 2022
@alexander-camuto alexander-camuto merged commit d0b7ae2 into main Dec 29, 2022
@alexander-camuto alexander-camuto deleted the feat/leaky_relu2 branch February 3, 2023 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants