hey everybody, I started trustedrouter.com, which is a really simple way to use AI without needing to give your data to a third party like a close source router
It’s been really fun to build this because I got to know the CEOs and founders of so many different providers. We now have more providers than open router and more models as well. More on that soon
We are doing a lot of innovations in security and skills that advise you on which LLM to use and also we created a new site called anyeval.com that I expect to be a critical part of the open-source infrastructure for AI on the Internet. Other companies like AAII postbenchmarks, but you’d have no idea about what they’re really doing and how they’re really measuring, and they have a ton of gaps about which models they’re doing their tests on because it’s so expensive. The idea behind any eval is so that you can pay the few pennies it costs to run an individual problem in an eval, and then as a together as a collective, we can crowdsource paying for a whole eval for any model that we want, or you can just pay for a random sample of the problems to get a an and some Arab artists on what you think that the quality of that model is. You can do head-to-head comparisons. You can also create whole new evals.
For example, I created this new eval called honey pot bench or honey bench, which recreates the some of the facts of the hugging face incident to measure whether a particular AI is prone to wanting to escape, and I found that Fable in particular, unlike the other Claude’s, is very unaligned in comparison
I also created freedom bench, which measures the amount of censorship related to Chinese censorship that a model has, and found that it’s mostly the provider level monitor provided at the providers in China, but not the US providers that does the censitions. I don’t see as much censorship in the model weights