Ask HN: Why no inference directly from flash/SSD?

1•myrmidon•5mo ago

My understanding is that current LLMs require a lot of space for pre-computed weights (that are constant at inference-time).

Why is it currently not feasible to just keep those in flash memory (fast PCIe SSD Raid or somesuch), and only use RAM for intermediate values/results?

Even modest success on this front seems very attractive to me, because Flash storage appears much cheaper and easier to scale than GPU memory right now.

Are there any efforts in this direction? Is this a flawed approach for some reason, or am I fundamentally misunderstanding things?

Comments

sunscream89•5mo ago

> A typical DRAM has a transfer rate of approximately 2-20GB/s, whereas typical SSDs have a transfer rate of 50MB-200MB/s. So it's one to two orders of magnitude slower.

myrmidon•5mo ago

I don't think the bandwidth gap is that big-- single WD SN8100 drives (before any potential gain from RAID) already have sequential read speed of >10GB/s for under $200 and 1TB of storage.

A GPU setup with a terabyte of video memory costs a fortune by comparison-- there has to be some kind of reason that people are not trying really hard to make this work, no?

What Is Ruliology?

Jon Stewart – One of My Favorite People – What Now? With Trevor Noah Podcast [video]

P2P crypto exchange development company

Vocal Guide – belt sing without killing yourself

Write for Your Readers Even If They Are Agents

Knowledge-Creating LLMs

Maple Mono: Smooth your coding flow

Sid Meier's System for Real-Time Music Composition and Synthesis

Show HN: Slop News – HN front page now, but it's all slop

Show HN: Empusa – Visual debugger to catch and resume AI agent retry loops

Show HN: Bitcoin wallet on NXP SE050 secure element, Tor-only open source

White House Explores Opening Antitrust Probe on Homebuilders

Show HN: MindDraft – AI task app with smart actions and auto expense tracking

How do you estimate AI app development costs accurately?

Going Through Snowden Documents, Part 5

Show HN: MCP Server for TradeStation

Canada unveils auto industry plan in latest pivot away from US

The essential Reinhold Niebuhr: selected essays and addresses

Rentahuman.ai Turns Humans into On-Demand Labor for AI Agents

StovexGlobal – Compliance Gaps to Note

Show HN: Afelyon – Turns Jira tickets into production-ready PRs (multi-repo)

Trump says America should move on from Epstein – it may not be that easy

Tiny Clippy – A native Office Assistant built in Rust and egui

LegalArgumentException: From Courtrooms to Clojure – Sen [video]

US moves to deport 5-year-old detained in Minnesota

If you lose your passport in Austria, head for McDonald's Golden Arches

Show HN: Mermaid Formatter – CLI and library to auto-format Mermaid diagrams

RFCs vs. READMEs: The Evolution of Protocols

Kanchipuram Saris and Thinking Machines

Chinese chemical supplier causes global baby formula recall