I think that if we're realistic with ourselves, AI will become exponentially more expensive to train, but without additional high quality data (not you, synthetic data), we're back to 1980s era AI (expert systems), just with enhanced fossil fuel usage to keep up with the TPUs. What's old is new again, I suppose!
I sincerely hope to be proven wrong, of course, but I think recent AI innovation has stagnated in terms of new things it can do. It's a great tool, when you use it to leverage that distribution (eg, semantic search), but it might not fundamentally be the approach to AGI (unless your goal is to replicate what we can, but less spikey)
In other words way forward seems to be to put models in loops. Which includes internal 'thinking' and external feedback. Make them use generated and acquired new data. Lossy compress the data periodically. And we have another race of algorithms.
Any benefit on a Mac with apple silicon? Any experiences someone could share?
1. Because the support in llama.cpp is horizontal integrated within ggml ecosystem, we can optimize it to run even faster than ollama.
For example, pixtral/mistral small 3.1 model has some 2D-RoPE trick that use less memory than ollama's implementation. Same for flash attention (which will be added very soon), it will allow vision encoder to run faster while using less memory.
2. llama.cpp simply support more models than ollama. For example, ollama does not support either pixtral or smolvlm
You'll have to compile llama.cpp from source, and you should get a llama-mtmd-cli program.
I made some quants with vision support - literally run:
./llama.cpp/llama-mtmd-cli -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL -ngl -1
./llama.cpp/llama-mtmd-cli -hf unsloth/gemma-3-12b-it-GGUF:Q4_K_XL -ngl -1
./llama.cpp/llama-mtmd-cli -hf unsloth/gemma-3-27b-it-GGUF:Q4_K_XL -ngl -1
./llama.cpp/llama-mtmd-cli -hf unsloth/unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_XL -ngl -1
Then load the image with /image image.png inside the chat, and chat away!
EDIT: -ngl -1 is not needed anymore for Metal backends (CUDA still yes) (llama.cpp will auto offload to the GPU by default!). -1 means all GPU layers offloaded to the GPU.
Very nice for something that's self hosted.
If you want to see, here it is:
https://gist.github.com/Q726kbXuN/f300149131c008798411aa3246...
Here's an example of the kind of detail it built up for me for one image:
It's wrapped up in a bunch of POC code around talking to LLMs, so it's very very messy, but it does work. Probably will even work for someone that's not me.
I'm sure there's a context limit if you have enough images, where you need to start map-reducing things, but even that wouldn't be too hard.
On macOS I downloaded the llama-b5332-bin-macos-arm64.zip file and then had to run this to get it to work:
unzip llama-b5332-bin-macos-arm64.zip
cd build/bin
sudo xattr -rd com.apple.quarantine llama-server llama-mtmd-cli *.dylib
./llama-mtmd-cli -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL -ngl 99
./llama-server -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL -ngl 99
Btw, the brew version will be updated in the next few hours, so after that you will be able to simply "brew upgrade llama.cpp" and you will be good to go!
Edit: sorry this is only true on Metal. For CUDA or other GPU backends, you still need to manually specify -ngl
I have no idea how to specify custom layer specs with multi GPU, but that is interesting!
(See the code in side llama_model_default_params())
Llama-server allowing vision support is definitely super cool - was waiting for it for a while!
This is perfect for real-time home video surveillance system. That's one of the ideas for my next hobby project!
llama-server -hf ggml-org/SmolVLM-Instruct-GGUF
llama-server -hf ggml-org/SmolVLM-256M-Instruct-GGUF
llama-server -hf ggml-org/SmolVLM-500M-Instruct-GGUF
llama-server -hf ggml-org/SmolVLM2-2.2B-Instruct-GGUF
llama-server -hf ggml-org/SmolVLM2-256M-Video-Instruct-GGUF
llama-server -hf ggml-org/SmolVLM2-500M-Video-Instruct-GGUF 25t/s prompt processing
63t/s token generation
Steps to reproduce:
git clone https://github.com/ggml-org/llama.cpp.git
cmake -B build
cmake --build build --config Release -j 12 --clean-first
# download model and mmproj files...
build/bin/llama-server \
--model gemma-3-4b-it-Q4_K_M.gguf \
--mmproj mmproj-model-f16.gguf
Note: if you are not using -hf, you must include the --mmproj switch or otherwise the web interface gives an error message that multimodal is not supported by the model.
I have used the official ggml-org/gemma-3-4b-it-GGUF quants, I expect the unsloth quants from danielhanchen to be a bit faster.
just trying to understand, awesome work so far.
simonw•3h ago