-
Notifications
You must be signed in to change notification settings - Fork 937
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BERT implementation #43
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is really nicely done!! I love it. I left some mostly minor comments.
Please address and then we can merge it into the examples.
Also could you add a requirements.txt
for dependenies?
Outside of anything that requires a change to MLX core, I think I've made the changes. I'll put off removing the |
Submitted and tested a PR for the bias change in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, just two more comments, then we can merge this and follow up after the change to core multiheadedattention.
…and fixing code type
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Brilliant, thank you for putting this together!
BERT implementation
Added and documented the BERT implementation. It currently uses the HF tokenizer, because that handles a lot of nice things (like padding, masking, and token_type_ids).
It's tested against the HuggingFace implementation at: batch sizes >= 1, different models (
bert-base-uncased
,bert-base-cased
,bert-large-uncased
,bert-large-cased
), etc.