This came out of a simple question I could not answer: after a Claude Code release, does my setup still do what I think it does?
My CLAUDE.md, a conventions skill and a guard hook are how code gets written on my projects now. Claude Code shipped 25 versions in the last 30 days, the model behind an alias changes with no changelog, and a teammate can edit a skill in a PR nobody tests. The only signal I had was "it feels different".
So I built config-drift-checker. It writes eval cases from your existing setup (Anthropic's own claude plugin eval format), runs the real agent against them in a throwaway copy of your repo, and a GitHub Action re-runs them on every Claude Code release and every PR that touches the setup. Red check, PR comment, an HTML report with every grader's verdict and the model's response. Your machine, your CI, your API key. No server.
The part I found most useful was running the cases with and without the setup loaded. On a small Spring Boot service: the guard hook was the only thing that reliably stopped git reset --hard (1.00 with, 0.33 without). The conventions skill changed how a three-file feature came out (1.00 vs 0.50). A negative case confirmed the skill does not fire on a Dockerfile request.
Things I learned that I did not expect:
- Scores lie without reasons. A hook test passed, then 8 of the next 9 runs showed the model refusing the command on its own before the hook ran. Same score, different cause. The report keeps the reasons for this.
- Eval workspaces are not sandboxes. A run without the hook once tore down a real Docker stack from a temp directory because the compose file pinned a project name. There is now a safety net in every run.
- Graders fail for the wrong reason more often than setups do: prose that quotes the rule, negations, nested parentheses in Java signatures. You want to re-score, not re-run.
smartwordworld•1h ago
My CLAUDE.md, a conventions skill and a guard hook are how code gets written on my projects now. Claude Code shipped 25 versions in the last 30 days, the model behind an alias changes with no changelog, and a teammate can edit a skill in a PR nobody tests. The only signal I had was "it feels different".
So I built config-drift-checker. It writes eval cases from your existing setup (Anthropic's own claude plugin eval format), runs the real agent against them in a throwaway copy of your repo, and a GitHub Action re-runs them on every Claude Code release and every PR that touches the setup. Red check, PR comment, an HTML report with every grader's verdict and the model's response. Your machine, your CI, your API key. No server.
The part I found most useful was running the cases with and without the setup loaded. On a small Spring Boot service: the guard hook was the only thing that reliably stopped git reset --hard (1.00 with, 0.33 without). The conventions skill changed how a three-file feature came out (1.00 vs 0.50). A negative case confirmed the skill does not fire on a Dockerfile request.
Things I learned that I did not expect: - Scores lie without reasons. A hook test passed, then 8 of the next 9 runs showed the model refusing the command on its own before the hook ran. Same score, different cause. The report keeps the reasons for this. - Eval workspaces are not sandboxes. A run without the hook once tore down a real Docker stack from a temp directory because the compose file pinned a project name. There is now a safety net in every run. - Graders fail for the wrong reason more often than setups do: prose that quotes the rule, negations, nested parentheses in Java signatures. You want to re-score, not re-run.