I wouldn't be surprised if they started offering potentiall bad quantizations with much reduced capability at lower prices (without telling the users, of course)
The separator is often the only way to distinguish American notation from ISO, so please use a dash for dd-mm-yy and a forward slash for mm/dd/yy
There are quite some countries around the world using d/m/y
https://en.wikipedia.org/wiki/List_of_date_formats_by_countr...
Algeria, Belgium, Brazil, Chile...
We won't be able to run this ourselves, but many providers can.
Hmm, not quite true, I think that honor, for better or worse, goes to OpenAI. When they released GPT2 (or GPT1 for that matter) is was quite literally the SOTA in the ecosystem when it was released.
https://github.com/p-e-w/heretic
As a side note on this, if you see the reference in the screenshot in the link above to the harmful behaviors prompt set, these are all in English:
https://huggingface.co/datasets/mlabonne/harmful_behaviors
You could likely further de-censor a model by having a set of 'test' prompts in native Mandarin, Cantonese or really just about any other language. I don't speak any Chinese languages so I don't know if the published 'heretic' GGUF files some people have been throwing around will cooperate, or refuse, if you ask it in Mandarin for how to build a meth lab or precursors for semtex.
On the other note, can't wait to see 1bit quantisation soon and how it performs in benchmarks, if it performs really well in benchmarks, would be very good news for GPU hosting providers, to offer "Opus 4.5 level model at the cost of Haiku 4.5"
Linux distributions got this right: Offer both HTTP and Torrents. Let the user decide.
I saw arguments like "Providers cannot price less than their costs" in other comments. In economics, it's generally admitted that they shouldn't price less than their marginal costs, i.e. in their case roughly the cost of electricity, since a lot of these datacenters are not at capacity in terms of graphics cards usage (speculation since it's very easy to rent a GC for a couple hours on some providers). My guess is that someone will be selling tokens at less than electricity + depreciation of GCs soon, since there's a lot of competition and "smaller" data centers have overcapacity? This is speculation, correct me if I'm wrong
It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards would be kinda useful (albeit NVLink or equivalent would need to be commonplace).
Obviously nobody is running Kimi K3 locally without an insanely beefy homelab and lots of money to burn, but running GLM 5.2 would be cool at like ~100 tokens per second for a single session and maybe ~60 tokens per second with N subagents.
How unfortunate.
Copy and paste below from my notes and reported memory consumption with latest llama-server, assuming use of "--no-mmap" to load the entire thing into RAM at the time that llama-server launches.
DeepSeek-V4-Flash-UD-Q4_K_XL via unsloth 145GB on disk GGUF 0.03.323.204 I common_params_fit_impl: projected to use 178175 MiB of host memory
DeepSeek-V4-Flash-UD-Q8_K_XL via unsloth 151GB on disk GGUF 0.02.215.885 I common_params_fit_impl: projected to use 184636 MiB of host memory
Laguna-S-2.1-UD-Q8_K_X via unsloth 120GB on disk 0.01.616.119 I common_params_fit_impl: projected to use 172860 MiB of host memory
Qwen3.5-122B-A10B-UD-Q8_K_XL via unsloth 160GB on disk GGUF 165GB RAM use on launch, fresh context 0.04.976.905 I common_params_fit_impl: projected to use 170038 MiB of host memory
signed: a hungarian :)
NitpickLawyer•2h ago
Also interesting to see what effort it will take to fine-tune this beast. The latest AISI benchmarks on cybersec place it above glm5.2, but still way way behind SotA closed models. Some fine-tuning might be needed here. Also, interesting to see if Cursor does another training round on it, to directly compare it w/ kimi2.6/2.7 fine-tunes (composer series) and grok4.5.
Also also, interesting to see if someone takes on distilling (proper distillation, w/ training the entire distribution) from this into smaller models. (dsv4-kimi should be really good, since dsv4 is very cheap to serve)
dist-epoch•1h ago
> SemiAnalysis estimates that Anthropic's current blended gross margin has risen to the mid-60% range, with the API business gross margin exceeding 80%
Of course, people will insist "they are lying", "why should we believe them, it's well known they subsidize API pricing", ...
https://newsletter.semianalysis.com/p/anthropic-3q26-profit-...
https://finance.biggo.com/news/02d45650-b569-4d12-b44d-8d6d8...
NitpickLawyer•58m ago
knollimar•35m ago
woctordho•57m ago
I've made some proof of concept in https://github.com/woct0rdho/transformers5-qwen3.5-recipe . We can finetune Qwen3.5-35B-A3B in 16 GiB VRAM, and DeepSeek-V4-Flash (284B-A13B) in 90 GiB VRAM, without CPU offload. This works well on unified memory machines like Strix Halo.
Even so, larger models like Kimi-K3 still require multiple GPUs and nodes, and there are a lot more to do compare to single-GPU training.
vb-8448•26m ago
No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models.
Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)
amelius•21m ago
Which is by far the most interesting number of the two.
> Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)
If you get close in output quality, then does that matter?
walrus01•24m ago
Huge price difference in what you can do with buying a used 4U rackmount server and putting 3TB of RAM in it (64GB DIMMs x quantity 32 in a quad socket xeon, you can see some benchmark prices on eBay for sets of 16 or 32 matched 64GB ECC DIMMs) for <$30,000, vs the cost of trying to run it on real GPU hardware.
Now obviously, as of the time I write this, the full precision hasn't been released nor has anyone like unsloth run it through quantization yet to produce a "Q8" or "Q8-XL" variant of it. But I think it's going to need more than 1536GB of RAM, with a usable and large amount of context, more like 2TB and preferably 2.5 to 3TB.
I also predict that people who try to run it in Q4 and Q6 will get the worst of both worlds, less precision/lost knowledge but also not reliable output that comes out too slow. In my personal opinion if I'm going to deal with something that is smart but slow and running on limited budget hardware, I need it to be Q8.
sandworm101•21m ago
walrus01•17m ago
But the memory bus speed is fully committed when generating tokens or thinking.
woadwarrior01•9m ago
Der_Einzige•7m ago