The paper behind it was presented in December 2025 Neurips

Release thread: https://xcancel.com/p_nawrot/status/2014770473289019709

Slides and audio presentation: https://neurips.cc/virtual/2025/loc/san-diego/poster/119605

I'd be interested to hear some use cases people have for large contexts on an 8B model. Other than sentiment analysis or summarization (this release implies agentic use). My experience with the general intelligence of agentic interactions is that everything is unusable before 32B for any context greater than 4k tokens.

I made a smaller sized version https://huggingface.co/g023/Qwen3-8B-DMS-8x-4bit-NF4