attention masks in each layer

[jiaosiyu1999]

Thank you for your work.
I noticed that you didn't seem to add a lower triangular matrix as attention mask in the mamba block to ensure that the current token only attends to the previous tokens. This appears to be inconsistent with "next-token prediction"
Could you please explain the reason for this?

[hp-l33]

Hi! Regarding your mention of the attention mask, Mamba does not require an explicit attention mask (similar to RNNs). Mamba achieves the masking of past tokens through causal 1D convolution, ensuring that the current token can only attend to previous tokens. This design aligns with the goal of "next-token prediction." For more details, please refer to the original Mamba paper.

[jiaosiyu1999]

Thanks a lot for the help.

[maxin-cn]

Hi! Regarding your mention of the attention mask, Mamba does not require an explicit attention mask (similar to RNNs). Mamba achieves the masking of past tokens through causal 1D convolution, ensuring that the current token can only attend to previous tokens. This design aligns with the goal of "next-token prediction." For more details, please refer to the original Mamba paper.

Hi @hp-l33 , If I don't want the mamba model of AiM to predict the current token and only see the previous tokens, but want it to see all tokens, how should the mamba model be set up?

[hp-l33]

Hi! Regarding your mention of the attention mask, Mamba does not require an explicit attention mask (similar to RNNs). Mamba achieves the masking of past tokens through causal 1D convolution, ensuring that the current token can only attend to previous tokens. This design aligns with the goal of "next-token prediction." For more details, please refer to the original Mamba paper.

Hi @hp-l33 , If I don't want the mamba model of AiM to predict the current token and only see the previous tokens, but want it to see all tokens, how should the mamba model be set up?

Unfortunately, the naive Mamba used in AiM cannot attend to tokens bidirectionally. You might consider alternative approaches.

[maxin-cn]

Hi! Regarding your mention of the attention mask, Mamba does not require an explicit attention mask (similar to RNNs). Mamba achieves the masking of past tokens through causal 1D convolution, ensuring that the current token can only attend to previous tokens. This design aligns with the goal of "next-token prediction." For more details, please refer to the original Mamba paper.

Hi @hp-l33 , If I don't want the mamba model of AiM to predict the current token and only see the previous tokens, but want it to see all tokens, how should the mamba model be set up?

Unfortunately, the naive Mamba used in AiM cannot attend to tokens bidirectionally. You might consider alternative approaches.

Thanks~