They are all different problems for the different languages. I was hoping this was a benchmark that attempted to see which languages were more efficient to use with which models.
dia80•37m ago
Why test Fable high effort vs Sol medium? Especially when Sol comes out 4-5x cheaper in their tests at those effort levels.
cbg0•25m ago
I think you just answered your own question.
Edit: In DeepSWE Sol High scores the same as Fable High for ~1/3 of the cost.
spullara•53m ago