What's your testing framework like? Do you guess at the best model, then send the same request +/-1 and compare?
mfkhalil•4m ago
It’s a mix. On benchmarks, yeah, we route a request, then run the same request on models within a range of capability and compare the results.
We also have our entire team using it for their day-to-day work, and they’re devs who are pumping out tens of PRs a day, so they’re a pretty unforgiving test group and are always giving feedback.
We also work with companies that have it either running on real traffic or running shadow evals in the background (which do a version of the +/-1 method you’re describing), and they’re a huge source of feedback.
At the end of the day, most of our focus is on how it performs on production workloads. As useful as benchmarks can be for initial testing, I think it’s dangerous to over-optimize for those kinds of tasks (and most frontier models have already learned how to solve those problems optimally during training, so results can be misleading there).
stAInley•17m ago
mfkhalil•4m ago
We also have our entire team using it for their day-to-day work, and they’re devs who are pumping out tens of PRs a day, so they’re a pretty unforgiving test group and are always giving feedback.
We also work with companies that have it either running on real traffic or running shadow evals in the background (which do a version of the +/-1 method you’re describing), and they’re a huge source of feedback.
At the end of the day, most of our focus is on how it performs on production workloads. As useful as benchmarks can be for initial testing, I think it’s dangerous to over-optimize for those kinds of tasks (and most frontier models have already learned how to solve those problems optimally during training, so results can be misleading there).