Flash Attention and Fused Attention #18121

Unanswered

agemagician asked this question in Q&A

agemagician
Oct 14, 2023

Hello,

I have seen in pallas an implementation for both Flash Attention and Fused Attention.

Couple of questions here:

What is the difference between Flash Attention and Fused Attention ?
Is it tested and can be used in training ?
Are both of them support CPU, GPU and TPU ?
When you will integerate bias, mask and dropout to the current implementation ?

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment