This is a draft implementation of the formula softmax(QK^T/sqrt(d_k))V.
The reference paper is "Attention is All You Need" (https://arxiv.org/abs/1706.03762).
-
Notifications
You must be signed in to change notification settings - Fork 2
IrishCoffee/cudnnMultiHeadAttention
About
multi-head attention implementation
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published