I'm not, yet, looking into training models, but more about inference, while staying relatively on a budget. I have about €2-3k to spend on hardware, for a server I can run at home. I know that one can rent an AI inference server, but from my research, they tend to cost around €200-300/mo, which means that my hardware will pay for itself in ~10 months.
However, I'm quiet confused with what hardware to chose. I saw recommendations for Ryzen AI Max+ 395, in SFF PCs, but some people suggest instead to build a "regular" PC with Nvidia 3090 or 4090.
People who run small servers (as opposed to €10-15k beasts), what's your experience and recommendations?
lesserknowndan•1w ago
I have been running some local models on LMStudio using an AMD 5950XT with 128 GB Ram (until recently with no GPU offloading). My two cents is that I keep reaching for my free ChatCPT account because I find the local models fairly unreliable.
Some just seem to get into a thinking loop where they go over the same ground again and again. Others can just output pure garbage.
Would anyone here suggest any specific models. Especially for Swift/SwiftUI. PHP. And MySQL Stored Procedures.