Indeed, captcha vs captcha bot solvers has been an ongoing war for a long time. Considering all the cybercrime and ubiquitous online fraud today, it's pretty impressive that captchas have held the line as long as they have.

You could definitely do better than we do here - this was just a test of how well these general-purpose systems are out-of-the-box

Makes sense, what do you think it was trained on?

After watching hundreds of these runs, Gemini was by far the least frustrating model to observe.

That's not due to accuracy, you're getting tarpitted for not looking human enough.

Didn't look a lot into this but I think the fact that humans are willing to do this in the "cents per thousand" or something range means that it's really hard to get much interest in automating it

There's a browser extension to solve them. Buster.

While running this I looked at hundreds and hundreds of captchas. And I still get rejected on like 20% of them when I do them. I truly don't understand their algorithm lol

Testing those same captcha on Google Chrome improved my accuracy by at least an order of magnitude.

Either that or it was never about the buses and fire hydrants.

Not sure it is your case but I think I sometimes had to solve many of them when I am in my daily task rush. My hypothesis is that I solve them too fast for "average human resolving duration" recaptcha seems to expect (I think solving it too fast triggers bot fingerprint). More recently when I fall on a recaptcha to solve, I consciently do not rush it and feel have no more to solve more than one anymore. I don't think I have super powers, but as tech guy I do a lot a computing things mechanically.

Pro tip, select a section you know is wrong, then de select it before submitting. Seems to help prove you are not a bot.

Same! As we talk about in the article, the failures were less from raw model intelligence/ability than from challenges with timing and dynamic interfaces

yes

We have an example of a failed cross-tile result in the article - the models seem like they're much better at detecting whether something is in an image vs. identifying the boundaries of those items. This probably has to do with how they're trained - if you train on descriptions/image pairs, I'm not sure how well that does at learning boundaries.

Reload are challenging because of how the agent-action loop works. But the models were pretty good at identifying when a tile contained an item.