Started this weekend project because I was thinking about whether software engineers’ performance should be tied to business impact like other engineers (as opposed to having PMs focus on that and devs executing) and then started thinking about Dev KPIs.

Developer work is unusually public. Aside from git diffs, you can see PR comments, Linear threads, and get a sense of both the complexity of the work and how people collaborate.

I tried a little adversarial experiment: - Take recent commits and have an LLM guess the "spec" (simulating a ticket on Linear, without building this step) - Ask Claude Code to implement the same thing - Use another LLM to compare the two solutions blindly - If the LLM version is worse than the human version, keep giving it hints until it matches or exceeds the human contribution - More elaborate hints = higher complexity score - Evaluating comments is even simpler. I didn’t try an adversarial approach, but there’s no reason it wouldn’t work.

This turned into a small library I hacked together. You can score devs on repos for fun.

I wonder if managers use numbers simply because they can’t hold all the context of a person’s contributions, and so lose out on nuance. What if LLMs could hold all of the context of your work and give a fairer evaluation? Could we move away from PMs deciding the “what” and engineers deciding the “how”, to Engineers deciding both?

PRs welcome!