But pretty quickly I realized they can really suck at this without a lot of guidance in unintuitive ways. So I built a fake grocery store to grade it.
EvalCart is a set of ~2k grocery items and, after ordering, gives the agent a receipt of its execution. Right now it includes a score and feedback on completeness, budget, taste, waste, efficiency, and a few others. The brands are synthetic so if you're testing explicit rules like "Must buy Lucky Charms", that's a gap.
The leaderboard only has some of my own test runs and baselines so far. If you build shopping agents, run one and tell me what's missing.
You don't need to sign up to use it, only if you want to store a history of runs there.