Ask HN: Relatively SoTA LLM Agents from Scratch?

4•solsane•1mo ago

As we know, OpenAI is not so open.

In 2023, I was playing with transformers, RNNs and I had an understanding how it worked from top to bottom (e.g. made my own keras, could whiteboard small nets) and I can throw things together in keras or tf pretty quick

I got a job and never touched that again. Data and compute notwithstanding, how hard would it be to make a pet project foundation model using the latest techniques? I’ve heard about MoE, things like that and I figure we’re not just throwing a bunch of layers and dropout in Keras anymore.

Comments

huevosabio•1mo ago

The Olmo team is AFAIK the only SOTA-ish model that has fully open source code and data. Their report is fantastic: https://www.datocms-assets.com/64837/1763662397-1763646865-o...

It should give you an idea of how hard it is to do a SOTA model from scratch!

If you relax the SOTA aspect, Karpathy's nanochat has you covered: https://github.com/karpathy/nanochat

walpurginacht•1mo ago

I'd suggest you take a read on HuggingFace's writeup when they trained smolLM3

https://huggingface.co/spaces/HuggingFaceTB/smol-training-pl...

rare detailed insight on the entire process

bjourne•1mo ago

Read this article: https://dl.acm.org/doi/10.1145/3712285.3759827 Training algorithms are relatively simple (base training, fine-tuning, RL), but the scale is critical. I.e., the engineering infrastructure. The authors recommend a 128 GPU cluster minimum and many petabytes of training data.

Ask HN: Non AI-obsessed tech forums

Ask HN: Anyone Using a Mac Studio for Local AI/LLM?

Ask HN: Ideas for small ways to make the world a better place

AI Regex Scientist: A self-improving regex solver

Ask HN: 10 months since the Llama-4 release: what happened to Meta AI?

Ask HN: Who wants to be hired? (February 2026)

Ask HN: Any International Job Boards for International Workers?

Ask HN: Who is hiring? (February 2026)

Ask HN: Why LLM providers sell access instead of consulting services?

Tell HN: Another round of Zendesk email spam

Ask HN: What is the most complicated Algorithm you came up with yourself?

Ask HN: Is Connecting via SSH Risky?

Ask HN: Has your whole engineering team gone big into AI coding? How's it going?

Ask HN: How does ChatGPT decide which websites to recommend?

Ask HN: Is it just me or are most businesses insane?

Ask HN: Mem0 stores memories, but doesn't learn user patterns

Ask HN: Anyone Seeing YT ads related to chats on ChatGPT?

Ask HN: Does global decoupling from the USA signal comeback of the desktop app?

Ask HN: Is there anyone here who still uses slide rules?

Kernighan on Programming

We built a serverless GPU inference platform with predictable latency

Ask HN: How Did You Validate?

Test management tools for automation heavy teams

Ask HN: Cheap laptop for Linux without GUI (for writing)

Ask HN: Have you been fired because of AI?

Ask HN: Does a good "read it later" app exist?

Ask HN: OpenClaw users, what is your token spend?

Ask HN: Anyone have a "sovereign" solution for phone calls?

How do you deal with SEO nowadays?

Ask HN: Has anybody moved their local community off of Facebook groups?