Anecdotal, opinion:

Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.

My anecdotal evidence says its still as blind as any other model, it has no taste, no attention to any sort of detail.

For 2 weeks I've been trying to get Codex to "outpaint" a wonderful image it generated as placeholder art for a level background.

After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.

An average pixel-artist could have recreated the image and more within 2-3 days.

In the third vision bench result, Sol is 100% correct but the expected has 1 error. Seems like an oversight.

In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.

Penny sample shown looks like failed EXIF orientation registered by the model/harness. The coins are correctly marked, it's rotated 90 degrees.

I've decided it's "good enough" after I saw it properly quote a string of text that was very roughly highlighted within a nested visual context. It also identified the context correctly (modal inside webapp inside screenshot of user desktop).

I agree. It did very well on an extremely challenging task.

I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.

In addition, the poster itself also happened to contain similar clothing.

You can see the reference images and its output in my writeup here: https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-reco...

While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.

I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right?

In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.

All of your use cases are very advanced.

I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.

I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.

I will add that all frontier models were fine with such tasks from the early 2024's.

I didn't expect Gemini 3.5 Flash to top basically every metric in this article.

Where are the Qwen benchmarks in this? I would be more interesting to see how Qwen performs.