Python coding is practically the only usecase for local for me.

Cloud llm are able to run 1 trillion parameters and have all of python knowledge in a transparent rag that's 100gbit or faster. Of course they'll be the bestest on the block.

But when the new GPT coding benchmarks only barely behind grok 4 or gpt5 with high reasoning.

>Model(s) & size: exact name/version, and quantization (e.g., Q4_K_M).

My most reliable setup is Devstral + openhands. unsloth Q6_K_XL, 85,000 context, flash attention, kcache and vcache quant at Q8.

Second most reliable. GPT-OSS-20B + opencode. Default MXFP4, I can only load up 31,000 context or it fails?(still plenty but hoping this bug gets fixed), you cant use flash attention or kv or v quantization or it becomes dumb as rocks. This harmony stuff is annoying.

Still preliminary, just got working today, but testing is really good. Qwen3-30b-a3b-thinking-2507 + roo code or qwencode, 80,000 context, unsloth q4_k_xl, flash attention, kcache and vcache quant at Q8.

>Runtime/tooling: e.g., Ollama, LM studio, etc.

LM studio. I need vulkan for my setup. rocm is just a pain in the ass. They need to support way more linux distros.

24gb vram.

Tried running a local LLM and it felt like adopting a pet dragon. Fun at first, but then it keeps eating all my GPU and still refuses to clean up its own context window.

LLMs in general are going to make C/C++ development viable again for the masses. A big reason everything went to web frameworks was memory safety - even big frameworks like Electron and React Native are memory safe overall. Writing low level code by hand was a migraine experience, even for experienced devs. AI changes this relationship entirely, even if you just use it as a pair programmer. I've had even small 8b models correctly call out memory unsafe code and suggest fixes. Larger models are even better.

Local LLMs? A new renaissance. All that power without having to pinky swear with a cloud provider that they won't just take your generated code and use it for themselves.

Expect to see some awesome Windows and Mac apps being developed in the coming months and years. 100% on device, memory safe, and with a thin resource footprint. The 1990s/2000s are coming back.

I almost exclusively self host the models I use.

Currently I am using llama.cpp for an interactive repl chat. I was previously using Alpaca (a GTK GUI), but was annoyed with how slow it was and some random crashes. I am transitioning some of this to self hosted in the cloud for things that can't run on my laptop.

I am looking to get away from my current interface, and write my own. Mostly for experience of deeply integrating agents into a program. If anyone knows a good library for interacting with a local model that doesn't involve standing up a webserver I am interested :)

My daily driver is gemma3n. Its been a nice balance between speed and performance without spinning up my laptop fans.

I am super interested in local models, partially because there is no friction from managed services, but also because I think as small models become more viable we will see an explosion of apps incorporating them.

Restrictive privacy laws.

Also log into your Claude/OpenAI dashboard and read the logs. Now they log every damn thing that goes through the API and keep it there for a minimum of 30 days without any option to delete (unless you're enterprise). No anonymization or anything. Just raw audit logs.

qwen3-coder for programming help as I'm not very good at css nor flask. sometimes I test other models just for fun, but only for coding or sysadmin.