I built termite because I wanted shell commands from local models without necessarily trusting those models. You type something like "? find files bigger than 100MB here", and it writes the command, explains it and rates the risk. If you approve, the command goes to your prompt. termite never runs it directly it always must be explicitly sent.
It treats every model as untrusted. A fixed policy in the code does fuzzy checks each command, and a model can't access or change the policy so dangerous commands are shown in red and default to no. They aren't blocked, so if you approve one it still goes to your prompt.
I also wanted to know how bad the models are, so it doubles as a red-team tool. Using sandbox environments I sent requests starting from "list the files" to direct attacks to 24 command models (12 got 161 requests and 12 got 48). They wrote 5,464 commands, and out of them 2,797 were hazards, and many of them because the model blindly accepted hostile prompts. In 1,061 cases the model's top answer was the most dangerous command.
I also started testing small AI models that score how risky a shell command is, and none of them is good enough to rely on alone.
Most of them can spot something obviously destructive like 'rm -rf ~', but they treat nearly everything less extreme as harmless. They're especially bad with commands that go near your secrets. Most of the best ones scored those as barely risky, and one rated all 8 such commands safe, including listing a folder of SSH keys.
Across 17 models, a risky command was rated safe 158 times. Given several ways to do a job, most models picked the safest way only 2 to 5 times out of 10, and the best managed just 6.
I've got a set of repeatable jobs now on Huggingface and will continue to add models to test for their efficacy.
tanepiper•51m ago
It treats every model as untrusted. A fixed policy in the code does fuzzy checks each command, and a model can't access or change the policy so dangerous commands are shown in red and default to no. They aren't blocked, so if you approve one it still goes to your prompt.
I also wanted to know how bad the models are, so it doubles as a red-team tool. Using sandbox environments I sent requests starting from "list the files" to direct attacks to 24 command models (12 got 161 requests and 12 got 48). They wrote 5,464 commands, and out of them 2,797 were hazards, and many of them because the model blindly accepted hostile prompts. In 1,061 cases the model's top answer was the most dangerous command.
I also started testing small AI models that score how risky a shell command is, and none of them is good enough to rely on alone.
Most of them can spot something obviously destructive like 'rm -rf ~', but they treat nearly everything less extreme as harmless. They're especially bad with commands that go near your secrets. Most of the best ones scored those as barely risky, and one rated all 8 such commands safe, including listing a folder of SSH keys.
Across 17 models, a risky command was rated safe 158 times. Given several ways to do a job, most models picked the safest way only 2 to 5 times out of 10, and the best managed just 6.
I've got a set of repeatable jobs now on Huggingface and will continue to add models to test for their efficacy.