Google's model release is pure PR puffery. It's no coincidence that it was released on the day of its earnings call.

I got it via the "Pro" plan, exclusively for personal task usage. So no agentic programming.

For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown. Secondly, the best part is the integration with Google search and Maps. Makes it seamless for trip planning. Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.

Either way, I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.

No, you can't use it. You can't even see it, and I won't provide you any information. But you must trust me that I am the genius who made the best model in the whole world.

* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.

* Expensive.

* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.

Especially with the recent addition of graphic visualization widgets

It manages to strike the perfect balance between depth and style of prose that other Frontier models just can't quite seem to match

I take this opportunity to give feedback on Argon. The limitations are a problem. I have a Pro subscription and Argon already stops working after 3-4 prompts with long delays afterwards. It's virtually useless right now even when you pay monthly for it. It is better for some tasks than 3.1 Pro, though. Not much in my experience but it can sometimes fix errors that 3.1 has problems with. I never use Flash models since they often give abysmal results.

So I'm stuck with using Gemini Pro 3.1 for now.

Is this API-only or US-only for now?

Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.

Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.

I used to defend Google's models months ago, but switched to Luna for less intelligent models.

But with this bs of not making it even public, pushing off release of newer models late, I lost faith in Google.

- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.

- Today the newest model announced from a big lab tops the charts.

I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.

It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.

Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.

My hunch is that Google will end up being best for non coding tasks. Google can concede coding to other vendors who pay Google for infra.

Let’s see, maybe they’ll surprise this time.

Until it got lobotomised. Then it lost most of it's capability.

Similar situation happened to Anthropic's Fable. It was an amazing model until Opus 5.5 came out.

So now, I'm not even looking at a "new" model until it is a month or two old. By then we will know if they will destroy ability to make it cheaper to host.

Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get a few weeks later is the same thing as what has been benchmarked here. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.

  [1] https://news.ycombinator.com/item?id=49901736

So here's a meta-benchmark: at what point does the act of choosing an LLM product for a reasoning task become less reliable (in the sense of "predictive of future success") than hiring a human being for that same task? Have we already cross that threshold? I think we might have.

And once we do... what happens then? Will the truly elite models be hidden in a sea of mediocrities simply because the goofball "managers" doing the hiring are too dumb to recognize their genius?