Window attention with huge window size(such as 65k) encountered shared mem error

[jesonxiang]

System Info

A800
trtllm v0.8

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. build engine with large seqlen, eg. 256k input
2. use huge window attention size, eg. 65k window
3. inference with 256k prompt

Expected behavior

can infer normally by using multi-block decoding.

actual behavior

additional notes

some informations may help:

1. 65k window will also trigger multi-block mode
2. the calculation of params.timesteps_per_block in function multi_block_grid_setup also use the max input length, but not the real windowed block size.
3. shared mem usage in function smem_size_in_bytes leads the wrong space, core.

[byshiue]

Sorry for late response. It is a known issue about computing the share memory size when we use sliding window. We will fix it soon.

[PerkzZheng]

the fix will be included in next week's update (Tuesday). Feel free to give it a try.