It’s kind of like general aviation where you can go buy a Cessna but it’s only going to realistically get you somewhere you could drive anyways but do you really wanna spend that mush cash to get road trip distance at slightly better than road trip speeds? You really need a 5 million dollar jet and that’s just not practical. That’s sort of how I feel about this device.
cmrdporcupine•53m ago
My Spark can do Qwen3.6 MoE A3B at 60 to 70-ish token/second and that's really good, but there's limits the usefulness of that model. It's not useful for coding, in any case.
Once people can run something like GLM 5.2 at lower quants (512GB could do a passable job), then I think the story changes.
Whether we ever see DRAM as cheap as it was ever again, I don't know.
science4sail•37m ago
That doesn't mean that local models are useless though! If Mythos/Sol is an ASI that threatens to take your job and turn you into paperclips, then Qwen/Gemma is an old-fashioned office secretary that loyally helps you with tasks but doesn't have a good grasp of details. Every white-collar worker 50 years ago would have killed to have a hard-working personal secretary.
whaleofatw2022•30m ago
Yeah it is slower than real RAM by a good amount for latency, but you can get similar bandwidth and the cost was history about half of the same size DDR.
science4sail•23m ago
cmrdporcupine•23m ago
Not anymore cost effective, I guess, but gets you the ability to work over very large model sizes maybe. But the problem is that tensor matmul etc hardware wouldn't work effectively with it.
Useful for KVCache though.