Are there any objective metrics/ benchmarks that people test harnesses by?

There are just so many now that it's hard to personally test them all or just trust the vibes.

Very fascinating, super interesting engineering. Although i do find it very funny how they just bypass a massive vulnerability, basically zero data isolation (even between good actors, let alone bad ones) with 3 sentences. Only in the llm space you can slap a massive limitation like this in the middle of the article and continue like nothing happened

> Whatever anyone tells Audel becomes part of the single experience that every other conversation draws on. In practice, Audel is bad at keeping secrets. Ask it what it’s been working on with someone else and it will often just tell you, even though we’ve asked it not to. We also haven’t studied what happens when two people give conflicting instructions. For now, we assume anything you tell Audel is shared with everyone on the team.

> "Headlong is a complete agent harness with a core of less than 10K lines of Bash..."

Wow. So, be nice or I'll replace you with a very large shell script?

It buries the lede. Prime Agent sounds like a very cool project.