I've spoken to several engineering leaders who want to better understand their team's AI usage, but don't know where to start. I've also spoken to several engineers who are stuck and don't know how to level up. There's a lot of FOMO building up and being vulnerable about your AI skills is not easy.
Adoption metrics like DAU/WAU and token counts tell you who has access and who uses AI tools. While an ok proxy for AI proficiency and productivity, they say very little about what happens inside a session and how engineers and teams actually use AI.
Which models get used and were they the right fit for the work? Are we spending consciously? Do tests run before a commit? Do outputs get questioned or accepted as is? Do outputs need a lot of repair or do they just get refined? Are repos missing out on scaffolding opportunities (skills, context files)? How's my team doing in AI competencies like context management, evals, and orchestration?
Pheebs captures that and more. It records the shape of sessions and gives engineers a private view of their own work, separate from the team reports. To make sense of the data, Pheebs proposes an AI Proficiency Model, which looks at two things:
1- Repertoire: which harness capabilities show up in an engineer's work across six competencies: models, artifacts, MCP, evals, context management, and orchestration.
2- Judgement (inspired by the Discernment competency from Anthropic's AI Fluency Index): what happens to AI output before a PR is created and merged. Does it get verified, challenged, refined? And was the model used appropriate for the task? Most teams run the largest model for everything. Pheebs shows how much they could save by running smaller models where they fit.
Pheebs is open source so give it a try and send us some feedback!
pratesd•1h ago
Adoption metrics like DAU/WAU and token counts tell you who has access and who uses AI tools. While an ok proxy for AI proficiency and productivity, they say very little about what happens inside a session and how engineers and teams actually use AI.
Which models get used and were they the right fit for the work? Are we spending consciously? Do tests run before a commit? Do outputs get questioned or accepted as is? Do outputs need a lot of repair or do they just get refined? Are repos missing out on scaffolding opportunities (skills, context files)? How's my team doing in AI competencies like context management, evals, and orchestration?
Pheebs captures that and more. It records the shape of sessions and gives engineers a private view of their own work, separate from the team reports. To make sense of the data, Pheebs proposes an AI Proficiency Model, which looks at two things:
1- Repertoire: which harness capabilities show up in an engineer's work across six competencies: models, artifacts, MCP, evals, context management, and orchestration.
2- Judgement (inspired by the Discernment competency from Anthropic's AI Fluency Index): what happens to AI output before a PR is created and merged. Does it get verified, challenged, refined? And was the model used appropriate for the task? Most teams run the largest model for everything. Pheebs shows how much they could save by running smaller models where they fit.
Pheebs is open source so give it a try and send us some feedback!