Sliding-Window Attention

[czoli1976]

Hi @kali , I have noticed that there is now no support for it.

WDYT? Would it be a nice addition ?

[kali]

Mmm... I jsut did some form of it for nemotron, we could do more, which variant do you have in mind ?

[czoli1976]

LLM decode case — causal sliding-window like Mistral

[kali]

ok, how do the model encode this in the wild ? onnx with an explicit mask ? It will be interesting to see if the recent work i did for nemotron generalizes at all :)

[czoli1976]

Dug into it — neither common path uses an explicit mask, which surprised me:

ORT-GenAI / Olive / AMD on-device exports: the window is a local_window_size=4096 attribute on the fused GroupQueryAttention op (checked a real Mistral-7B-v0.1 export — all 32 layers, no Trilu/banded mask, just a 2D padding mask).
optimum / transformers.js decomposed exports: the window is dropped — I exported a tiny Mistral with sliding_window=64 and it comes out as plain full-causal attention; the window value isn't in the graph at all.
So there's no banded mask to reuse the nemotron machinery on — in the fused path it's an op attribute (which tract's GroupQueryAttention currently rejects), and in the decomposed path it's gone.

so ...: step 1 is accepting local_window_size on GQA, step 2 is the bounded ring-buffer cache (the real win, encoding-independent).

Happy to take the decode-cache piece — it builds on the in-place cache in #2321.

[czoli1976]

@kali see #2327 about this