Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow gelu approximations #1911

Merged
merged 5 commits into from
Jul 14, 2023
Merged

Allow gelu approximations #1911

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d31ac74
Allow gelu approximations.
pcuenca Jul 11, 2023
3741477
Add test for `tanh` gelu approximation.
pcuenca Jul 12, 2023
dd9a028
Remove `sigmoid` GELU in frontend
pcuenca Jul 12, 2023
044a096
Assert gelu approximation is supported.
pcuenca Jul 12, 2023
f13ffec
gelu tests cover the default initialization.
pcuenca Jul 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions coremltools/converters/mil/frontend/torch/ops.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3898,8 +3898,15 @@ def gelu(context, node):
assert len(inputs) in (1, 2)
if len(inputs) == 2:
approximate = inputs[1].val
assert approximate == 'none'
res = mb.gelu(x=inputs[0], name=node.name)
if approximate == "tanh":
approximate = "TANH_APPROXIMATION"
elif approximate == "sigmoid":
approximate = "SIGMOID_APPROXIMATION"
elif approximate == "none":
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If pytorch only support two modes right now,
it will be better to do:

else:
    assert approximate == "none"

to deal with the future possible change in the torch frontend (like they add more support for different mode)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't do it because an unsupported approximation would still fail in the mb.gelu implementation. But you are right that this is a better place to signal where the conversion needs to happen, changing it now!

approximate = "EXACT"
else:
approximate = None
res = mb.gelu(x=inputs[0], mode=approximate, name=node.name)
context.add(res)

pcuenca marked this conversation as resolved.
Show resolved Hide resolved

Expand Down