The developer here. My background is in networking/systems and got interested in MechInterp for various reasons, and one of the first things I noticed was many methods that have been built to monitor internal AI state are generally evaluated on how they perform as a 'classifier' of sorts; the idea is to look at the ROC curve or the AUROC on a test set etc. However, being a good 'classifier' doesn't really tell you if the monitor helps you take better decisions, specially under operational constraints. For instance, there is always a budget (time/compute etc) that is imposed on inspecting every operation an agent proposes, which means mistakes will be made and not all mistakes are the same.
Take a simple example; an AI agent/assistant wants to read a private log (it's authorized) and deleting a backup. Your audit budget lets you review only one. An internal monitor inspecting the activation of the model may give a 99% chance of violation for reading the log, but a 75% violation chance for backup deletion. If your controller focuses on the higher score, and if you can monitor only one action, you'll miss the more serious deletion action.
This is where ObserverBench comes in. Borrowing from control theory, an observer is a component that estimates the hidden state of a system so a controller can act on the estimate. ObserverBench does this for internal AI monitors: know whether an estimate is useful for a particular decision, by testing it against the actions the decision produces. So instead of just using a loss function, we also use a controller and a budget to evaluate monitors/observers.
There are multiple results on the page, but to highlight one set of results on testing monitors on Gemma 2-9B-it checking code for backdoors, we see:
* A prompted monitor (asking Gemma to review the code itself) beats an internal Gemma scope SAE feature probe on ranking (aurora 0.893 vs 0.87)
* However, on simulated examples (2000), with 1% backdoor prevalence and 2% review budget, the internal probe left fewer backdoors unreviewed.
* Resampling makes the ranking uncertain, but this shows the core issue- a higher test set score (auroc) does not mean you catch more violations under a budget.
ObserverBench fixes the task, actions, budget and decision rule so that one can compare different observers on raw estimates/scores and their downstream decisions.
Grateful to get any feedback, curious to know if others running safety monitors have seen this mismatch between eval metrics and budget constrained decisions.
kwisatzh•20m ago
The developer here. My background is in networking/systems and got interested in MechInterp for various reasons, and one of the first things I noticed was many methods that have been built to monitor internal AI state are generally evaluated on how they perform as a 'classifier' of sorts; the idea is to look at the ROC curve or the AUROC on a test set etc. However, being a good 'classifier' doesn't really tell you if the monitor helps you take better decisions, specially under operational constraints. For instance, there is always a budget (time/compute etc) that is imposed on inspecting every operation an agent proposes, which means mistakes will be made and not all mistakes are the same. Take a simple example; an AI agent/assistant wants to read a private log (it's authorized) and deleting a backup. Your audit budget lets you review only one. An internal monitor inspecting the activation of the model may give a 99% chance of violation for reading the log, but a 75% violation chance for backup deletion. If your controller focuses on the higher score, and if you can monitor only one action, you'll miss the more serious deletion action.
This is where ObserverBench comes in. Borrowing from control theory, an observer is a component that estimates the hidden state of a system so a controller can act on the estimate. ObserverBench does this for internal AI monitors: know whether an estimate is useful for a particular decision, by testing it against the actions the decision produces. So instead of just using a loss function, we also use a controller and a budget to evaluate monitors/observers.
There are multiple results on the page, but to highlight one set of results on testing monitors on Gemma 2-9B-it checking code for backdoors, we see: * A prompted monitor (asking Gemma to review the code itself) beats an internal Gemma scope SAE feature probe on ranking (aurora 0.893 vs 0.87) * However, on simulated examples (2000), with 1% backdoor prevalence and 2% review budget, the internal probe left fewer backdoors unreviewed. * Resampling makes the ranking uncertain, but this shows the core issue- a higher test set score (auroc) does not mean you catch more violations under a budget.
ObserverBench fixes the task, actions, budget and decision rule so that one can compare different observers on raw estimates/scores and their downstream decisions.
Two ways to check it out:
* Interactive walk thru (browser, no login): https://kwisatzh.github.io/observerbench/try/ A simple example in a sandbox * Local runner (no GPU needed or model downloads): https://github.com/kwisatzh/observerbench#sixty-second-demo Uses saved measurements from Qwen2.5-7B base to test which internal model edit to choose.
Grateful to get any feedback, curious to know if others running safety monitors have seen this mismatch between eval metrics and budget constrained decisions.