I would like to share my latest open source local LLM inference tool implemented in C#. It supports models like Gemma4, Qwen3.6 with multi-modal (image, vision, audio), reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability. The API is completely compatible with OpenAI and Ollama interface.
By my brief benchmark, it has > 80% decode performance of llama.cpp on Gemma4 model, and I will keep optimizing it. The entire benchmark report will sent out in the repo very soon.
Really appreciated if you can try it and give me some feedback. If you like it, it will be a big thank you if you can star it. Thank you very much!
zhongkaifu•1h ago
By my brief benchmark, it has > 80% decode performance of llama.cpp on Gemma4 model, and I will keep optimizing it. The entire benchmark report will sent out in the repo very soon.
Really appreciated if you can try it and give me some feedback. If you like it, it will be a big thank you if you can star it. Thank you very much!