I think the more people looking at this the better. I have a feeling there will be some breakthroughs in identifying important circuits and being able to make more efficient model architectures that are bootstrapped from some identified primitives.

https://open.spotify.com/episode/3H46XEWBlUeTY1c1mHolqh?si=L...

https://www.dwarkesh.com/p/sholto-trenton-2 -- search the transcript for "circuit" for the quick bits.

Eg, "If you look at the circuit, you can see that it's not actually doing any of the math, it's paying attention to that you think the answer's four and then it's reasoning backwards about how it can manipulate the intermediate computation to give you an answer of four."

https://transformer-circuits.pub/

[1]: https://transformer-circuits.pub/2021/garcon/index.html

Have fun

https://gist.github.com/jexp/8d991d1e543c5a576a3f1ee70132ce7...