Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

83•snehesht•1h ago

Comments

snehesht•1h ago

I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.

https://huggingface.co/Qwen/Qwen3.8-Flash-Next

proc0•1h ago

Do you know how it compares to Qwen 3.8 27B? I really want to compare the distilled ones with harness versus the full MoE versions.

incognito124•43m ago

Qwen 3.8 flash next is way better than 27B. It's so good I dont even use claude anymore

snehesht•40m ago

Yeah I agree, I'm running it with Pi didn't notice much difference compared to lower tier models and the speed, of course.

nicce•23m ago

I am running 27B with Deepseek Harness these days and somehow just by using it, without any parameter changes, the model feels even more intelligent.

mickeyp•37m ago

I have not tried Flash Next yet; but 27B is a cracking, little model. It is the first small model that I, as someone with 30 years of experience, can finally say is good enough to hand off small and mid-sized tasks and expect a pretty good result.

It is also a competent tool caller when quantised to NVFP4 for use with ninfer; my own harness only reports the occasional hiccup and it is only because the model will sometimes emit tool calling tokens in its reasoning loop.

snehesht•24m ago

This is interesting, thanks. - https://github.com/Neroued/ninfer

thatsabadlook•19m ago

Significantly better for both performance and real world use case. 3.8 27b is a good small model. This is a good model.

geye1234•13m ago

I find 27B more accurate -- maybe because I'm running at FP8 instead of NVFP4? Flash Next starts making spelling mistakes when I get to 150K context or so. Also it sometimes ignores .md file instructions. Not sure if others have found that.

thatsabadlook•21m ago

Why is this surprisingly well? It's 2.5x faster than anthropic models, you have data sovereignty, privacy,and that's a strong model. Sounds like a best case scenario to me

esafak•51m ago

Has anyone calculated the effective intelligence of these quantized models?

mkl•43m ago

There's some info in the README, including:

> Coder: a coding version with half of the experts removed. It reaches 91% of the full model's SWE-bench Verified score (measured by its authors) and fits 32 GB of RAM.

https://github.com/Niko1221/Strata#which-model-should-i-pick

javier2•38m ago

ok that is getting interesting!

nicce•26m ago

I wonder how this Coder compares to Qwen 3.8 27B. Can it be really better since they are competitive for same memory requirements?

nsagent•12m ago

See this recent paper: Quantization Degradation in Large Language Models: A Signal–Noise Perspective [1].

  We observe that such degradation varies substantially across these factors: 4-bit quantization usually preserves performance, 2-bit often causes broad degradation

This repo uses 2-bit quantization and removes some of the experts for its smallest fastest model. Make of that what you will.

[1]: https://arxiv.org/abs/2608.08188

quietFalcon•50m ago

Nice, though generation speed is the easy half for MoE offload, what's your prompt processing look like at say 16k context?

snehesht•38m ago

They have some community benchmarks published https://github.com/Niko1221/Strata/tree/main/bench/results

merbanan•11m ago

Q2_0 does 33 tok/s decode and ~600t/s prompt processing at 128k context on RTX2060 8GB VRAM.

ISTA IQ3_XXS does ~21 tok/s decode and ~240t/s prompt processing

gdevenyi•41m ago

I had this working with the FreeToken inference engine a month ago when they launched.

https://github.com/FlashML-org/FreeToken

deadbunny•33m ago

> Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.

And I thought piping to bash was bad

snehesht•32m ago

Yeah, I was surprised at first then had to dig through setup.py and setup.sh files to figure out.

gchamonlive•20m ago

Piping to bash is definitely worse because there is no plan mode in bash. Agents also normally don't execute anything transparently, at worst you'll see it doing something weird in the logs.

prettyblocks•27m ago

I've been playing with this on a 3090 and it FLIES. Does a pretty good job too on the tasks I've thrown at it (php code base security audits).

hypfer•27m ago

Is these another one of those repos where it turns out that claude decided to quant the KV cache to q4 or smaller?

The Readme doesn't say, but it's all AI generated, so..

panny•17m ago

I'm far less interested in how good a big expensive model is on hardware 99% of people can't afford and would rather see what runs best on a chromebook or mobile phone with 8GB of RAM.

MrDrMcCoy•13m ago

Ternary Bonsai 2 might be for you.

0xbadcafebee•15m ago

Lol, sure, if you quant it to hell (Q2) it'll go real fast...

Tepix•11m ago

Q2 quantization. Not interested.

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

Glashütte Trash Clock – A 30-minute pendulum clock made from trash

VGHF Digital Archive passes 5000 magazines. Here's what's next

Tell HN: Bob Cringely has died

Why don't more developers “use the platform”?

Show HN: AI search for every photo and every frame of video on macOS

The work by Valve's Timur Kristóf on improving old AMD GPUs on Linux

gpuvis: GPU Trace Visualizer

Rejection Sensitivity in Gifted and Twice-Exceptional Children

Agents don't need memory, they need documentation

Treachery in the Rodin Museum 3D scan verdict

Emitting metadata early makes building/checking Rust up to twice as fast

Hole Punch: Sling your spaceship around gravitational fields

We're working on a new RuneScape MMO

So you think you could be an electrician?

LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents

Celebrating the 100th birthday of the kidney donated to him as a teenager

The Heilbronn Problem

We're going to need default hard budget caps on pretty much everything

Reasons I didn't become an EMT, ranked

We want you to build the next Git platform on Cloudflare

Religious scholars met with Anthropic

I quit OpenAI because its culture is broken

Math's pedagogical curse – Grant Sanderson [video]

Your body of work thinks back at you

What Meta got right with Muse

How to hack time, with C2PA

Dirty Optimization Secrets (C for Playdate)

Surely you have ultra-wideband radios on your bins too?

FTL: A new operating system for clouds

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

Comments

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

Glashütte Trash Clock – A 30-minute pendulum clock made from trash

VGHF Digital Archive passes 5000 magazines. Here's what's next

Tell HN: Bob Cringely has died

Why don't more developers “use the platform”?

Show HN: AI search for every photo and every frame of video on macOS

The work by Valve's Timur Kristóf on improving old AMD GPUs on Linux

gpuvis: GPU Trace Visualizer

Rejection Sensitivity in Gifted and Twice-Exceptional Children

Agents don't need memory, they need documentation

Treachery in the Rodin Museum 3D scan verdict

Emitting metadata early makes building/checking Rust up to twice as fast

Hole Punch: Sling your spaceship around gravitational fields

We're working on a new RuneScape MMO

So you think you could be an electrician?

LeCun has "zero concerns" about AI wiping out humanity, recent "rogue" incidents

Celebrating the 100th birthday of the kidney donated to him as a teenager

The Heilbronn Problem

We're going to need default hard budget caps on pretty much everything

Reasons I didn't become an EMT, ranked

We want you to build the next Git platform on Cloudflare

Religious scholars met with Anthropic

I quit OpenAI because its culture is broken

Math's pedagogical curse – Grant Sanderson [video]

Your body of work thinks back at you

What Meta got right with Muse

How to hack time, with C2PA

Dirty Optimization Secrets (C for Playdate)

Surely you have ultra-wideband radios on your bins too?

FTL: A new operating system for clouds