I have been studying modern LLM architectures and started implementing them from scratch in PyTorch to better understand the design choices behind each model.

OpenArch is a collection of these implementations, including Llama, Qwen, DeepSeek, Gemma, Kimi, GPT-OSS and others.

The goal is to keep the code readable and useful as a reference when going from the paper to an actual implementation.

Would be interested in feedback from people working on model architecture and training.

Hey anuj

This is excellent for understanding. I'm having some trouble to get into understanding - pytorch is for me the RL which is used as gym/training. There I can chose ppo, dnq and other agents to perform some predefined actions in a predefined gym/world.

The repo you are showing - I really have problems to get it into RL understanding of mine. What's the gym? What are the agents. Can it be used to train that models with pytorch?

Sorry for the noob question. Papers are overwhelming my noob brain.