If you've built a benchmark, whether it succeeded or not, I'd love to learn about a few things in particular (any other advice or common pitfalls is also appreciated):
Credibility: what made people trust a benchmark built by the company selling the product. Did publishing it help? Did you bring in outside partners, academic partners or customers to make it more independent?
Data: whether synthetic examples ever convinced anyone, or whether it always came down to real customer data. If it was real data, how did you anonymise it, get customers to agree to share it, and keep it realistic after scrubbing out sensitive details?
Grading: how you proved to others grader was valid, especially if you used an LLM as the judge. Additionally, how did you keep updating as the latest models came out.
For reference we're building a general benchmark for systematically comparing swarm architectures. We're starting by reporting on when a swarm beats a single agent and when it doesn't. Our goal is to show where we are better with quantification at realistic tasks which are similar to customer needs. This matters especially since people ask us why they need 1000s of agents, and just saying it will be faster or cheaper can feel a bit arbitrary without numbers to back it up.