Fix SDPA sliding window compatibility

[fxmarty]

As per title, fixes #28980

Supersedes #29220 #29407 as the implementation ends up being different (added you as co-author here @ehuaa).

This bug dates back to #26572 where sliding_window was not properly accounted for in the _prepare_4d_causal_attention_mask_for_sdpa method. Since then, SDPA support was added to models that use sliding window, but this bug was not yet fixed.

[ehuaa]

Hi, @fxmarty ,thanks for your great work! I think you can add some tests i mentioned in #29407, to check if the result with sliding window in SDPA is the same as flashattention2.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[fxmarty]

@ehuaa Thank you, for sure will do!

Runing mistral, mixtral, starcoder2 tests, those fail but are already failing on main:

FAILED tests/models/mistral/test_modeling_mistral.py::MistralIntegrationTest::test_speculative_generation - AssertionError: 'My f[19 chars]is 100% Sriracha. I love the heat, the tang and the fact costs' != 'My f[19 chars]is 100% Sriracha. I love the heat, the ...
FAILED tests/models/mixtral/test_modeling_mixtral.py::MixtralIntegrationTest::test_small_model_logits - AssertionError: The values for attribute 'dtype' do not match: torch.float16 != torch.float32.
FAILED tests/models/mixtral/test_modeling_mixtral.py::MixtralIntegrationTest::test_small_model_logits_batched - AssertionError: The values for attribute 'device' do not match: cuda:0 != cpu.
FAILED tests/models/starcoder2/test_modeling_starcoder2.py::Starcoder2IntegrationTest::test_starcoder2_batched_generation_4bit - AssertionError: Lists differ: ['Hel[110 chars]t is related to the topic of "How to make a ga[179 chars]ute'] != ['Hel[110 chars]t is aimed at creating a...
FAILED tests/models/starcoder2/test_modeling_starcoder2.py::Starcoder2IntegrationTest::test_starcoder2_batched_generation_eager - AssertionError: Lists differ: ['Hel[181 chars]I am currently working on', "def hello_world()[114 chars]app"] != ['Hel[181 chars]I am looking for a', "de...

[ArthurZucker]

Don't know if I mentioned it offline, we'll refactor this to a single function without inheritance similar to update_causal_mask. See Recurrent Gemma, as it supports sliding window!

Thanks for re-enabling sliding window.

[ArthurZucker]

There are basically 2 cases:

1. You ignore the causal mask
2. You don't ignore it.
   Code is really not super clear but we will refactor this soon anyways.