didn't Ollama vendor Llama cpp too?

Most projects typically start with llama.cpp and then move away to proprietary kernels

Thank you! it goes without saying that the field is rapidly developing, so the use cases range from private AI assistant/companion apps to internet connectivity-independent copilots to powering private wearables, etc.

We're currently working with a few projects in the space.

For a demo of a familiar chat interface, download https://apps.apple.com/gb/app/cactus-chat/id6744444212 or https://play.google.com/store/apps/details?id=com.rshemetsub...

For other applications, join the discord and stay tuned! :)

Cactus is a framework - not the app itself. If you're looking for an Android demo, you can go to

https://play.google.com/store/apps/details?id=com.rshemetsub...

Otherwise, it's easy to build any of the example apps from the repo:

cd react/example && yarn && npx expo run:android

or

cd flutter/example && flutter pub get && flutter run

great stuff. (good timing for a post given all the comet news too :) )

DM me on BF - let's talk!

appreciate the feedback! Made the demo links more prominent on the README.

Some Android models won't support GPU hardware; we'll be addressing that as we move to our own kernels.

The app itself is just a demonstration of Cactus performance. The underlying framework gives you the tools to build any local mobile AI experience you'd like.

you can plug in a vector DB and run Cactus embeddings for retrieval. Assuming you're using React Native, here's an example:

https://github.com/cactus-compute/cactus/tree/main/react#emb...

(Flutter works the same way)

What are you building?

The models themselves live in an isolated sandbox. On top of that, each mobile app has its own sandbox - isolated from the phone's data or tools.

Both the model and the app only have access to the tools or data that you choose to give it. If you choose to give the model access to web search - sure, it'll have (read-only) access to internet data.

So far, our focus is on supporting models with fully open-sourced weights. Providers who are sensitive about their weights typically lock those weights up in their cloud and don't run their models locally on consumer devices anyway.

I believe there are some frameworks pioneering model encryption, but i think we're a few steps away from wide adoption.

Simple answer is they won't send the model to the end user if they don't want it used outside their app.

This isn't really anything novel to LLMs of AI models. Part of the reason for many previously desktop applications being cloud or requiring cloud access is keeping their sensitive IP off the end users' device.

Thanks for the feedback. You're right to point out that Google AI Edge is cross-platform and more flexible than our phrasing suggested.

The core distinction is in the ecosystem: Google AI Edge runs tflite models, whereas Cactus is built for GGUF. This is a critical difference for developers who want to use the latest open-source models.

One major outcome of this is model availability. New open source models are released in GGUF format almost immediately. Finding or reliably converting them to tflite is often a pain. With Cactus, you can run new GGUF models on the day they drop on Huggingface.

Quantization level also plays a role. GGUF has mature support for quantization far below 8-bit. This is effectively essential for mobile. Sub-8-bit support in TFLite is still highly experimental and not broadly applicable.

Last, Cactus excels at CPU inference. While tflite is great, its peak performance often relies on specific hardware accelerators (GPUs, DSPs). GGUF is designed for exceptional performance on standard CPUs, offering a more consistent baseline across the wide variety of devices that app developers have to support.

I would say that while Google's MediaPipe can technically run any tflite model, it turned out to be a lot more difficult to do in practice with third-party models compared to the "officially supported" models like Gemma-3n. I was trying to set up a VLM inference pipeline using a SmolVLM model. Even after converting it to a tfilte-compatible binary, I struggled to get it working and then once it did work, it was super slow and was obviously missing some hardware acceleration.

I have not looked at OP's work yet, but if it makes the task easier, I would opt for that instead of Google's "MediaPipe" API.

Does Google AI Edge have React Native support? Doesn't seem like it? Cactus does though.

> as you are for sure aware

> Why lie?

Whoa—that's way too aggressive for this forum and definitely against the site guidelines. Could you please review them (https://news.ycombinator.com/newsguidelines.html) and take the spirit of this site more to heart? We'd appreciate it. You can always make your substantive points while doing that.

Note this one: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

True - but Cactus is not just an app.

We are a dev toolkit to run LLMs cross-platform locally in any app you like.

Didn't know that. Thanks

Very good point - we've heard this before.

We're restructuring the model initialization API to point to a local file & exposing a separate abstracted download function that takes in a URL.

wrt downloading post-install: based on our feedback, this is indeed a preferred pattern (as opposed to bundling in large files).

We'll update the download API, thanks again.

love this. So many layers deep, we just had a good laugh.

reminds me of

- "You are, undoubtedly, the worst pirate i have ever heard of" - "Ah, but you have heard of me"

Yes, we are indeed a young project. Not two weeks, but a couple of months. Welcome to AI, most projects are young :)

Yes, we are wrapping llama.cpp. For now. Ollama too began wrapping llama.cpp. That is the mission of open-source software - to enable the community to build on each others' progress.

We're enabling the first cross-platform in-app inference experience for GGUF models and we're soon shipping our own inference kernels fully optimized for mobile to speed up the performance. Stay tuned.

PS - we're up to good (source: trust us)

Thanks for the comment, but:

1) The commit history goes back to April.

2) LlaMa.cpp licence is included in the Repo where necessary like Ollama, until it is deprecated.

3) Flutter isolates behave like servers, and Cactus codes use that.

thank you! We're continue to add performance metrics as more data comes in.

A Qwen 2.5 500M will get you to ≈45tok/sec on an iPhone 13. Inference speeds are somewhat linearly inversely proportional to model sizes.

Yes, speeds are consistent across frameworks, although (and don't quote me on this), I believe React Native is slightly slower because it interfaces with the C++ engine through a set of bridges.

in the app you mean?

Adding shortly!

hot off the press in our latest feature release :)

we support cloud fallback as an add-on feature. This lets us support vision and audio in addition to text.

We are writing our own backend, but tflite (now called LiteRT) was not faster than GGML when we tested and GGML is already well supported. But we are moving away completely anyway.

Probably they mean new apps. Since kotlin multiplatform on android is just native android and android share is like 70% if devices it already at least 50% market share of mobile apps. If you add flutter and react native there is not much left: only games like unity and unreal. I see much less iOS jobs these days.

Thanks!

You don't need to guess: https://github.com/cactus-compute/cactus/tree/main/cpp

no, good observation - not hidden; we don't have a "clear conversation" button.

to your previous point - Cactus fully supports tool calling (for models that have been instruction-trained accordingly, e.g. Qwen 1.7B)

for "turning your old phones into local llm servers", Cactus is likely not the best tool. We'd recommend something like actual Ollama or Exo

say more about "community tools"?

thank you! Very kind feedback, and we'll add your feedback to our to-dos.

re: "question would get stuck on the last phrase and keep repeating it without end." - that's a limitation of the model i'm afraid. Smaller models tend to do that sometimes.

that's our mission! if you are passionate about the space, we look forward to your contributions!

looking forward to your feedback!