Ask HN: Thoughts on AMD Ryzen AI MAX+ 395 for Local AI?
2•mempirate•4h ago
Wondering if anyone has played around with the Strix Halo based systems with 128GB of unified memory for local AI, and what their experience has been so far in running different models:
- Practical limit on number of parameters x quantization
- What you've been using it for
- Which models and inference engines combine well
Comments
Iolaum•4h ago
Own a framework desktop 128gb.
GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
mempirate•1h ago
Thanks for the detailed response. My goal is to deploy small to medium-sized LLMs on it for things that I'd rather keep private and local, i.e. automation around smart home, but also tracking finances and portfolio, aggregating & summarizing news feeds and blog posts and so on. For these purposes, medium intelligence at low TPS seems good enough to me.
But a large part of it is also wanting to learn more about serving these models, and potentially some small-scale experiments with training and fine tuning.
Iolaum•4h ago
GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
mempirate•1h ago
But a large part of it is also wanting to learn more about serving these models, and potentially some small-scale experiments with training and fine tuning.