Well after probably a day of training it seems like its come up with something that actually does detect AI text pretty well... maybe too well considering my original plan was to train this type of model to be able to break it. And this is how naive my thinking was for ways to break the detector "something something embedding space" , "use actual randomness to break the predictable pattern of AI text, like doing python's random() to select from a list of words that could be replaced in a paragraph while making the rest of the text fit". Both of which ended up instantly failing, and I told claude to figure out some ways autonomously. Turns out "red-teaming" is a scary topic for Anthropic so it just started to block its different approaches after some time, and the best it got to was "rewrite in the style of [actual human text]" which was 15/19 detected and adding random known human text where it made sense which was also similarly ranked but ofc there's the issue of Pangram like detectors having sentence level scans.
All that to say while my initial thing I built this for ended up not working, at least I have kind of my own pangram that I can test things out on later on, and it also proved how good claude is at autonomously training if you give it a hill to climb on.
PS: this text itself isn't claude written and you can verify it by putting it in the demo :p .