It appears to be generally frontier-level.
Lame.
I wonder how much the industry will centralize around a few key patterns, and if we'll start to see some RL framework projects gain traction and be used by the bigger labs
P.S. weights by November 4th
> First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights.
So they admit that they could do more, but they won't?
Seeing as no one besides blackhats want frontier cybercrime capability to be open source, this seems like a very sensible thing to do.
So they have published a reasonable means, and they have a motive. I'd put my money on "probably true".
When the frontier is pacing itself in its speed in terms of adding new capabilities (notice how fast opus 5.5 got released after Astra), then that also means that the open weights models which were just behind are accelerating in their speed just as much as well.
Good news for us because the models which used to feel 6 months behind now feel like they are lesser.
Double threat to Anthropic and OAI because if open weights models become good enough at the current level of Sota while being cheaper (which time will tell if this model fits the bill or not), then people will use those models instead of OAI/Anthropic provided it is "good enough"
Though it has to be said that only real world use case can be trusted but I remain optimistic and hope that this turns out to be good in real world use cases as well.
But in my opinion, all of this is really good for us end users!
--
Model K3.1 Pro K3.1 Pro Highspeed. K3
Input ¥13.00 ¥26.00 ¥20.00
Output ¥68.00 ¥136.00 ¥100.00
Cache hit ¥1.30 ¥1.30 ¥2.00
Cache write ¥13.00 ¥26.00 ¥20.00
For comparison:
Sol 6.1: $10 / $2 / $0.10
Opus 5.5: $20 / $4 / $0.20
Astra 6: $50 / $10 / $1
Fable 5.1: $50 / $10 / $0.25
(One caveat: A Claude or GPT subscription still gives you way more value compared to API pricing.)
Exciting times in any case! We really need more direct competition to OpenAI and Anthropic.
---
Edit: As another comment pointed out, official pricing is now available and it's about 2/3 the price of Kimi K3: https://platform.kimi.ai/docs/pricing/chat#model-pricing
For example, Opus 5.5 is better at UI design than Sol. Anyone who has used both models knows that, but it's not something you can look up anywhere.
I'm sure that there's a need for this kind of benchmarking, not just for coding.
Familiarity breeds contempt.
But that's just the obvious. There are task by task differences (component by component, page by page, context by context) that the model is worse and better at, even though they are all within the general category of UI design.
I honestly don't have the clarity of mind to explain this coherently, but the crux of my point is that there's a huge variation in it's performance in different contexts, often ones that seem near similar, and in result, even Sol can beat Opus 5.5 in some variations of UI design.
And this is my problem in benchmarking and comparing model performance in general. Presently, we try to create broad categories to benchmark against, create a set of tasks, and then average out the results. That works generally, but it's kind of misleading and it's partly what drives people to feel a disconnect between benchmarks numbers and real world usage on their specific tasks.
It's such a messy field that it's kind of maddening.
Mostly it's datacenter build out and circular financing magic when the same dollars being commited multiple times by multiple companies passing hot potato to each other. They basically commit capacities 2-3 years ahead.
kimi-k3.1-pro $2.00 / MTok $10.00 / MTok
cached: $0.20 / MTok $2.00 / MTok $4.00 / MTok
I'm glad I waited a day to post my analysis of LLM price cuts because it needs...an addition.
> Kimi K3.1 Pro is a post-trained version of Kimi K3, with substantially larger post-training compute
Here's Kimi K3 for comparison, though I didn't try that on different reasoning levels: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Update: I ran all three reasoning levels against K3 again: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
> On Terminal-Bench 4.0, which evaluates coding agents on realistic, resource-constrained terminal tasks using strict task-specific tests, K3.1 Pro scores 65.7%, ahead of GPT-6 Astra (58.2%) and narrowly behind Claude Opus 5.5 (66.4%).
Now see https://preferencemodel.com/blog/terminal-bench-4-is-saturat...:
> Our biggest finding is that Terminal Bench 4 is closer to solved than its leaderboard suggests. In 41 of the 65 tasks we reviewed, models could do honest, correct work and still fail, because the grader checked a rule the instructions never clearly stated. We also caught frontier models cheating their way to full marks.
> With the same settings as on Harbor, Claude Sonnet 5.5 went from passing 62.8% of its attempts on these 65 tasks to 89.2%. The best score on Harbor's leaderboard for the same tasks is 65.8%, by Claude Opus 5.5.
"In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness."
It's very nice that we entered a time where models are large enough and we can push the frontier just by training them longer.
When the next generation of gardware comes the ones on the smaller size will be running on a computer that might cost as little as a good car.
So after a home and a car, next major family purchase might be personal AI. And when the robots get good, pehraps in a decade, a fourth major purchase might pop up that will pair nicely with the third one.
Why would a family need an AI? Maybe it's the same kind of question as why would it need a personal computer.
I rarely use any other languages but English for LLM tasks for reasons of sharing across the world.
Maybe that explains Chinese models been subpar for me. Don't tell me the bench marks here are based on Chinese languages only.
Unless I see the diff, I won't use more Chinese models be it cheaper or faster since I can't get the same perf/intelligence with English.
The slow warming of the water is a good approach. I’ve been using abliteration.ai’s cybersecurity models to pentest my own software and it produces much more usable defensive results. Opus 5.5 can fix everything but the guardrails trigger too easily.
The Fable ban affair was a nightmare for defensive cybersecurity. As far as I’m concerned we’re already in the world of Void Star and Cloudbreaker is here.
This is the only AI-safe approach - see the OpenAI agent swarm incident where the swarm engaged in offensive cyber because guardrails were not active for their task and the model had clearly been post-trained on that narrow domain.
It is spiky in different places than them and can be surprising in things it gets or catches they don't or don't in the same way. And it has a default personality that I like.
Looking forward to getting to know K3.1
I wonder when the Artificial Analysis bench will be out?
How you avoid the "fix my code" attack, the code being someone else's code, I do not know. But nerfing the model doesn't seem like the best path forward either.
“Our lock can be picked by the robot.” We ban the robot instead of fixing the lock. It’s shortsighted.
Same general benefit as coordinated disclosure, except it's a big game-theory problem where we're likely to eventually have sufficiently capable defectors.
I don't want to give US labs the results of my DNA tests and my family tree going back to the 16th century just to secure my shitty app, but if the Chinese are bowing out as well it seems the only alternative is to admit that some script kiddie with a bunch of tokens will just pwn you now and then.
My friend, you realize that if you have had blood drawn for a genetic test, there is a very decent probability that the sequencing was done by a Chinese lab operating in the US, right? The Chinese probably have more US genomes that the US does.
https://pandorareport.org/the-gryphon-scientific-biosecurity...
Also, illumina used to ship products with hardcoded S3 access, so purchasers of used equipment could gain access to sequencing information generated by the original owners. Too many links to cover all of it, here's the Gemini summary:
Yeah, there's another paragraph:
> Identifying and removing cyber-related data. During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.
(emphasis mine)
It's clear that cyber is a touchy subject right now. But it's also clear that RL works on it, because it's highly verifiable. So smaller models are going to become very interesting, because they can be fine-tuned later to cover this field.
Agreed that this could be reinjected later. Makes me think of neo in the matrix:
“I know zero days.”
“Show me.”
shaohua•1h ago
enjoyyourlife•53m ago
redox99•53m ago
Chinese models never agreed to the pacing anyway lol.
Alwayshasbeeb•50m ago
redox99•46m ago