>An interesting observation we made was that even though the 3 optimizers were given tens of thousands of dollars in compute, they found the best solution very early on
Honestly, this confirms some bias I already have from coding — models can get to the best solution pretty fast, but if the solution is something obscure, they usually don’t find it.
jeremytian•39m ago
From my experience, models that drive optimizers in a loop often get stuck in local minima. ofc, this is just correlation, and not truly indicative, but if you look at the first graph, analyzing precision vs cost, all three model-driven optimizers just bounce back and forth between a precision range, not really expanding the horizon or getting too much worse than the orignal. from our analysis on this data, model-driven optimizers typically don't start from scratch again, they just continue down their optimization path.
andy0liang•52m ago
It would be interesting to see how a human researcher would do on the same task and same budget. 2 weeks of salary + a Claude Code max plan might outperform all of these auto-optimizers in a loop.
jeremytian•37m ago
this would be an interesting experiment but i think one of the main benefits of having at least some automated pipeline is that human iteration is quite slow since the information intake speed is a few magnitudes slower. probably some combination would be optimal, like ensuring alignment of objectives at the beginning and tuning along the way
AbhinavX•53m ago
Honestly, this confirms some bias I already have from coding — models can get to the best solution pretty fast, but if the solution is something obscure, they usually don’t find it.
jeremytian•39m ago