added support for sigmoid activation fucntion #21424

mosesdaudu001 · 2023-08-07T16:12:51Z

The PR is not fully ready yet, as I still need to implement for 7 more functions.

As at now, I have only done for the sigmoid activation function

github-actions · 2023-08-07T16:13:43Z

Thanks for contributing to Ivy! 😊👏
Here are some of the important points from our Contributing Guidelines 📝:
1. Feel free to ignore the run_tests (1), run_tests (2), … jobs, and only look at the display_test_results job. 👀 It contains the following two sections:
- Combined Test Results: This shows the results of all the ivy tests that ran on the PR. ✔️
- New Failures Introduced: This lists the tests that are passing on master, but fail on the PR Fork. Please try to make sure that there are no such tests. 💪
2. The lint / Check formatting / check-formatting tests check for the formatting of your code. 📜 If it fails, please check the exact error message in the logs and fix the same. ⚠️🔧
3. Finally, the test-docstrings / run-docstring-tests check for the changes made in docstrings of the functions. This may be skipped, as well. 📚
Happy coding! 🎉👨‍💻

ivy-leaves · 2023-08-07T16:30:04Z

If you are working on an open task, please edit the PR description to link to the issue you've created.

For more information, please check ToDo List Issues Guide.

Thank you 🤗

jshepherd01 · 2023-08-08T12:01:37Z

I think you need to add the complex_mode to the container class's _static_hardswish and _static_sigmoid functions as well

mosesdaudu001 · 2023-08-08T13:37:02Z

I think you need to add the complex_mode to the container class's _static_hardswish and _static_sigmoid functions as well

Sure I will add that

jshepherd01

There's some fiddly parts of the testing which will need changing related to the dtypes. I also think you've missed out relu6 in the jax frontend.

About testing: it seems like there's currently a lot of test failures, mostly for paddle and torch, and also complex values never seem to be generated with the tensorflow backend (and therefore are never generated for the functional API tests, since tensorflow is the ground truth for those)

ivy_tests/test_ivy/test_frontends/test_jax/test_nn/test_non_linear_activations.py

ivy_tests/test_ivy/test_functional/test_experimental/test_nn/test_activations.py

jshepherd01 · 2023-08-16T13:57:43Z

Looks like tests are still failing. If you run the tests for the jax frontend, you'll see that some backends aren't handling complex values correctly, and if you print the input_dtype for each test you'll see that hypothesis doesn't generate any complex inputs when tensorflow is the chosen backend. Those will both need fixing.

Another thing is that as of yesterday (specifically as of #21902) the stateful API classes no longer need to take parameters like complex_mode in the _forward method, so they'll need to be removed here

jshepherd01

I've done some testing of each of the functions you've modified and attached the feedback to each one separately to make it easier to read through

ivy_tests/test_ivy/test_frontends/test_jax/test_nn/test_non_linear_activations.py

jshepherd01 · 2023-08-24T16:33:53Z

ivy/functional/ivy/activations.py

@@ -376,8 +376,13 @@ def relu(
 @to_native_arrays_and_back
 @handle_array_function
 @handle_device_shifting
+@handle_complex_input
 def sigmoid(


Complex numbers are marked as unsupported for sigmoid in the tf and paddle backends

To fix this, I have to add a jax-like function for sigmoid. I just did in the latest commit

Are you sure that's needed? I just tested on the main branch and ivy.sigmoid(3+4j) works just fine with both paddle and tensorflow backends

Running the test without the sigmoid jax-like function gives errors in tensorflow and paddle.

[cpu-paddle-False-False] Failed: [undefined]AssertionError: the results from backend paddle and ground truth framework jax do not match (0.2689414322376251+0j)!=(0.21795842936625076-0.20194822765801285j)

[cpu-tensorflow-False-False] Failed: [undefined]AssertionError: the results from backend tensorflow and ground truth framework jax do not match (1+0j)!=-0j

Do you still have the test cases that returned those errors?

For Paddle:
`Falsifying example: test_jax_sigmoid(
on_device='cpu',
frontend='jax',
backend_fw='paddle',
dtype_and_x=(['complex128'], [array(-1.-1.j)]),
test_flags=FrontendFunctionTestFlags(
num_positional_args=0,
with_out=False,
inplace=False,
as_variable=[False],
native_arrays=[False],
test_compile=False,
generate_frontend_arrays=False,
),
fn_tree='ivy.functional.frontends.jax.nn.sigmoid',
)

You can reproduce this example by temporarily adding @reproduce_failure('6.82.5', b'AXicY2QAAkYGCGBEYzMwAAAAfgAG') as a decorator on your test case`

For Tensorflow:
`Falsifying example: test_jax_sigmoid(
on_device='cpu',
frontend='jax',
backend_fw='tensorflow',
dtype_and_x=(['complex128'], [array(-709.-1.j)]),
test_flags=FrontendFunctionTestFlags(
num_positional_args=0,
with_out=False,
inplace=False,
as_variable=[False],
native_arrays=[False],
test_compile=False,
generate_frontend_arrays=False,
),
fn_tree='ivy.functional.frontends.jax.nn.sigmoid',
)

You can reproduce this example by temporarily adding @reproduce_failure('6.82.5', b'AXicY2QAAkYGMGA6imBDaQAObADM') as a decorator on your test case
@handle_frontend_test(`

ivy/functional/ivy/activations.py

ivy/functional/ivy/experimental/activations.py

jshepherd01 · 2023-08-24T16:42:35Z

ivy/functional/ivy/experimental/activations.py

@@ -350,8 +410,13 @@ def selu(
 @to_native_arrays_and_back
 @handle_array_function
 @handle_device_shifting
+@handle_complex_input
 def silu(


Complex numbers are marked as unsupported for silu in the tf, torch, and paddle backends. The implementation in the paddle backend throws an error:

@with_unsupported_device_and_dtypes( {"2.5.1 and below": {"cpu": ("bfloat16",)}}, backend_version, ) def silu(x: paddle.Tensor, /, *, out: Optional[paddle.Tensor] = None) -> paddle.Tensor: if x.dtype in [paddle.float32, paddle.float64]: return F.silu(x) if paddle.is_complex(x): > return x * (1.0 / (1.0 + paddle_backend.exp(-x))) E ValueError: (InvalidArgument) __mul__(): argument (position 1) must be int, float, bool or Tensor, but got Array (at /paddle/paddle/fluid/pybind/eager_utils.cc:1435)

I don't think this should be done using a jax_like function, it would be better to modify the paddle backend implementation to work properly with it

Without the silu jax-like function, there are erros in tf and paddle during testing

./ivy_tests/test_ivy/test_frontends/test_jax/test_nn/test_non_linear_activations.py::test_jax_silu[cpu-tensorflow-False-False] Failed: [undefined]AssertionError: the results from backend tensorflow and ground truth framework jax do not match (-88-1j)!=(-0+0j)

./ivy_tests/test_ivy/test_frontends/test_jax/test_nn/test_non_linear_activations.py::test_jax_silu[cpu-paddle-False-False] Failed: [undefined]AssertionError: the results from backend paddle and ground truth framework jax do not match (-0.2689414322376251-0.2689414322376251j)!=(-0.4199066758155823-0.016010195016860962j)

Do you have the test cases for these handy?

Falsifying example: test_jax_silu(
on_device='cpu',
frontend='jax',
backend_fw='tensorflow',
dtype_and_x=(['complex64'], [array(-88.-1.j, dtype=complex64)]),
fn_tree='ivy.functional.frontends.jax.nn.silu',
test_flags=FrontendFunctionTestFlags(
num_positional_args=0,
with_out=False,
inplace=False,
as_variable=[False],
native_arrays=[False],
test_compile=False,
generate_frontend_arrays=False,
),
)

You can reproduce this example by temporarily adding @reproduce_failure('6.82.5', b'AXicY2AAAkYGCIhAYkNoAAZ9AFw=') as a decorator on your test case

Falsifying example: test_jax_silu(
on_device='cpu',
frontend='jax',
backend_fw='paddle',
dtype_and_x=(['complex64'], [array(-1.-1.j, dtype=complex64)]),
fn_tree='ivy.functional.frontends.jax.nn.silu',
test_flags=FrontendFunctionTestFlags(
num_positional_args=0,
with_out=False,
inplace=False,
as_variable=[False],
native_arrays=[False],
test_compile=False,
generate_frontend_arrays=False,
),
)

You can reproduce this example by temporarily adding @reproduce_failure('6.82.5', b'AXicY2AAAkYGCGBEYzMwAAAAXwAF') as a decorator on your test case

ivy/functional/ivy/experimental/activations.py

ivy/functional/backends/tensorflow/activations.py

ivy/functional/backends/torch/activations.py

jshepherd01 · 2023-08-25T16:00:54Z

I've added more comments to some of the threads I started on the last review. I'll do a proper code review early next week, but if you could address those in the meantime that would be great. Thanks :)

…term into its own variable

And a couple other bugs that were causing things to break, such as removing duplicated atol from `test_jax_hard_swish` and adding complex dtype to some tests. These functions now pass all tests in the jax frontend, except sigmoid and silu which fail for tf backend due to a known issue

Update supported dtypes for certain functions, and change the safety factor for logsigmoid. All tests now pass locally, except problems caused by tensorflow's sigmoid and a problem with the test for elu (which seems to feed in the wrong values).

jshepherd01 · 2023-09-06T13:12:10Z

I started experimenting with fixing the errors and ended up just writing up a fix myself so I thought I might as well push it. There remain some test failures caused by an issue with tensorflow's sigmoid (reported here: tensorflow/tensorflow#61800) and some caused by bfloat16 problems with numpy, as well as an issue caused by an apparent bug in the test for elu which still needs to be fixed.

ivy-seed · 2023-10-14T05:53:20Z

This PR has been labelled as stale because it has been inactive for more than 7 days. If you would like to continue working on this PR, then please add another comment or this PR will be closed in 7 days.

github-actions · 2023-10-14T11:56:25Z

Thank you for this PR, here is the CI results:

This pull request does not result in any additional test failures. Congratulations!

added support for sigmoid activation fucntion

6b90a8a

mosesdaudu001 requested review from Ishticode and jshepherd01 August 7, 2023 16:12

ivy-seed assigned tomatillos Aug 7, 2023

ivy-leaves added Array API Conform to the Array API Standard, created by The Consortium for Python Data API Standards JAX Frontend Developing the JAX Frontend, checklist triggered by commenting add_frontend_checklist Ivy Functional API labels Aug 7, 2023

applied complex_input decorator to hardswish activation fucntion

a9879a1

mosesdaudu001 added 10 commits August 8, 2023 16:15

added argument to static sigmoid and hardswish container funcs

16c42d4

applied complex_input decorator to silu activation fucntion

a9a9a53

resolved conflicts

04e0002

Merge branch 'master' into complex_input_activation_functions

a101978

applied complex_input decorator to selu activation fucntion

63eeb4c

Merge branch 'unifyai:master' into complex_input_activation_functions

6e97b30

applied complex_input decorator to logsigmoid activation function

0636cfb

applied complex_input decorator to elu activation function

01c3e41

applied complex_input decorator to relu6 activation function

83a94aa

added float_complex to stateful test

54eb859

jshepherd01 suggested changes Aug 15, 2023

View reviewed changes

made changes as suggested by joe

cfe6cdc

mosesdaudu001 requested a review from jshepherd01 August 15, 2023 14:12

mosesdaudu001 and others added 4 commits August 18, 2023 09:43

fixed testing issues for sigmoid

1c00385

Bring docstrings and call signatures in line with standards

d4d70b1

fixed testing issues for silu

5a0956e

fixed testing issues for selu

5bb2dab

jshepherd01 suggested changes Aug 24, 2023

View reviewed changes

mosesdaudu001 added 4 commits August 25, 2023 10:00

added jax-like function for hardswish

32ce676

added support for complex dtpes in hardwish tf and torch

86e2a88

added sigmoid jax-like function

6a92760

removed complex from unsupported for sigmoid

c0a70ba

jshepherd01 suggested changes Aug 25, 2023

View reviewed changes

ivy/functional/backends/tensorflow/activations.py Outdated Show resolved Hide resolved

ivy/functional/backends/tensorflow/activations.py Outdated Show resolved Hide resolved

ivy/functional/backends/torch/activations.py Outdated Show resolved Hide resolved

mosesdaudu001 added 4 commits August 25, 2023 14:49

removed complex from unsupported for selu and updated jax-like function

e1163ea

removed complex from unsupported for silu

47d0231

removed complex from unsupported for elu

f472575

updated some backends as requested by Joe

c445b97

mosesdaudu001 requested a review from jshepherd01 August 25, 2023 15:20

mosesdaudu001 and others added 10 commits August 26, 2023 09:44

Merge branch 'main' into complex_input_activation_functions

d48fe89

created custom implementation for tensorflow logsigmoid

6747831

added support for complex in tensorflow elu and selu

e35a406

made changes to elu-jax-like function so as to extract the condition …

43e1013

…term into its own variable

Merge branch 'main' into complex_input_activation_functions

5719e7e

updated files

a4df5a9

removed duplicated file

b5fd995

Merge branch 'master' into moses-complex

47abd9a

fix: update supported dtypes

4bb0291

Update supported dtypes for certain functions, and change the safety factor for logsigmoid. All tests now pass locally, except problems caused by tensorflow's sigmoid and a problem with the test for elu (which seems to feed in the wrong values).

style: update docstrings and function signatures for complex_mode

b7a0daf

jshepherd01 removed their assignment Sep 19, 2023

ivy-seed assigned tomatillos Sep 19, 2023

ivy-seed added the Stale label Oct 14, 2023

NripeshN closed this Nov 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added support for sigmoid activation fucntion #21424

added support for sigmoid activation fucntion #21424

mosesdaudu001 commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

ivy-leaves commented Aug 7, 2023

jshepherd01 commented Aug 8, 2023

mosesdaudu001 commented Aug 8, 2023

jshepherd01 left a comment

jshepherd01 commented Aug 16, 2023

jshepherd01 left a comment

jshepherd01 Aug 24, 2023

mosesdaudu001 Aug 25, 2023

jshepherd01 Aug 25, 2023

mosesdaudu001 Aug 26, 2023

jshepherd01 Aug 29, 2023

mosesdaudu001 Aug 29, 2023

jshepherd01 Aug 24, 2023

mosesdaudu001 Aug 25, 2023

jshepherd01 Aug 25, 2023

mosesdaudu001 Aug 26, 2023

jshepherd01 Aug 29, 2023

mosesdaudu001 Aug 29, 2023

jshepherd01 commented Aug 25, 2023

jshepherd01 commented Sep 6, 2023

ivy-seed commented Oct 14, 2023

github-actions bot commented Oct 14, 2023

added support for sigmoid activation fucntion #21424

added support for sigmoid activation fucntion #21424

Conversation

mosesdaudu001 commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

ivy-leaves commented Aug 7, 2023

jshepherd01 commented Aug 8, 2023

mosesdaudu001 commented Aug 8, 2023

jshepherd01 left a comment

Choose a reason for hiding this comment

jshepherd01 commented Aug 16, 2023

jshepherd01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jshepherd01 commented Aug 25, 2023

jshepherd01 commented Sep 6, 2023

ivy-seed commented Oct 14, 2023

github-actions bot commented Oct 14, 2023