Ask HN: Hardware for 1k RPS?

5•gsky•1d ago

I ran an uncensored model on a CPU server. as expected its dead slow (min or two per query).

What kinda hardware (GPU) do i need to serve 1k RPS?

I could not find APIs for uncensored models that kinda forced me to run locally

Comments

barnabee•1d ago

https://venice.ai claim to offer uncensored models (I’ve not tested that claim)

gsky•1d ago

Thanks, I give it a try.

eddythompson80•1d ago

Depends on your model size and how many of it can fit in memory. Multiply the size by 1k and divide by the memory capacity of the hardware for a rough ballpark.

Gaia Is a Tough Bitch

Passage of Senate Bill 29 Positions Texas as a Leading State for Incorporations

Rudyard Kipling, Kim (1901)

Generators for Certain Alternating Groups with Applications to Crypto (1975)

Portal 2 Becomes an Impressively Capable Web Server

Show HN: Modmenu for mlbb that will make you undefeated

Violating memory safety with Haskell's value restriction

The Math Behind That Dick Joke in HBO's "Silicon Valley" (2019)

Mary Meeker Artificial Intelligence Report [pdf]

Countries on the edge of the Sahara desert are reversing desertification

Show HN: Pokémon Blue remade in React (playable in browser)

AI Speeds Up Dev, Cloning Comes Easy Too

Jokes from the Jar – Powered by AI

Ask HN: User-scoped AI tokens for safe client-side AI logic?

I'm Open-Sourcing My Custom Benchmark GUI

Show HN: Built FLUX.1 Kontext image editor – context-aware AI editing

Show HN: Ephe – A Minimalist Open-Source Markdown Paper for Today

VimGolf – Real Vim ninjas count every keystroke

Processing Deep Space Astrophotos with Free Software

Playary – a fast, cross-platform music and podcast streaming

ICE Paid Palantir Millions for 'Complete Target Analysis of Known Populations'

JD Vance has changed his name multiple times over the years

Blasting Past iOS 18

A Beginner's Guide to Prisma ORM

DevEco Studio: Essential Features for HarmonyOS Development

Muscular Christianity

Builder.ai's AI Companion Natasha Was Indian Workers, Now Bankrupt

Entry-level tech hiring down 25% in 2024 despite market recovery

FreeBSD 14.3-RC1 Now Available

Why DeepSeek is cheap at scale but expensive to run locally