Hi HN! Sipp is an open-source AI inference library for running local models in browsers with up to 3x faster decode speeds than alternative libraries.

My background is in HCI (human-computer interaction) and graphics programming. Me along with my co-founder have been experimenting and thinking a lot about what the next user experience will look like when tokens are commodified to the point of being essentially “free.” A motivation for us was to try to move beyond the chat app and information retrieval use cases that are dominant now, and figure out how AI could instead act as a continuous and silent hand that helps the user indirectly, subtly monitoring their intent and dynamically generating or shifting the UI to meet their needs.

In our explorations, we ran into two pain points: 1) when running AI in the browser, performance wasn’t good enough for real-time applications, and model loading and caching were issues; 2) when trying to run locally on desktop, there weren’t any good solutions for embedding a model for use through platforms like Electron or Tauri.

So a few months ago, we started contributing to the llama.cpp project to explore browser inference. At the time, it didn’t have great WebGPU support. We helped get the backend working across most modern GPUs (16 bit fp is required) with fairly good ops coverage. Simultaneously, we started building up Sipp from the ground up using Rust and C++ to create a unified library around how local and cloud inference can be used together in real applications and use cases.

The result, Sipp achieves up to a 3x speedup in overall token decoding compared to alternative libs. But we believe there is more room for improvement here. Our goal is to investigate more bespoke inference pipelines per model architecture, which will allow us to further optimize for both compute and memory constraints more effectively.

Sipp uses a single, unified client API. A large concern for us was how to create a library that effectively bridges both local and cloud inference in a simple way. We wanted developers to start by running a small local model in the browser via WebGPU (or via other backends), and then scale the exact same code path to a self-hosted gateway (CUDA/Vulkan/Metal) or a cloud provider just by changing the endpoint. This enables the benefits of running locally for small tasks while letting you offload to a provider or cloud for tasks that require more intelligence.

Currently, our primary focus has been optimizing the browser experience, but we are actively working on creating additional backends for running an LLM locally via our client API, which works across Node, Rust, and Python right now.

What we are most excited about next is pushing our existing backends even further. As anyone working with a high-performance system knows, raw algorithms and efficient matmul are only half the battle. A significant portion of bottlenecks in real-time systems comes from inefficient memory management and VRAM<>RAM transfer costs.

We believe there is a massive opportunity to extract even more performance through aggressive kernel fusion. By creating “bespoke” kernels tailored to specific model architectures, we can drastically minimize intermediate memory copies. Our next goal is to see exactly how close we can push local inference to the theoretically “perfect” decode and prefill speeds for consumer hardware.

We have a live, simple chat demo running entirely on-device on our site, as well as a benchmarking tool if you want to test or verify the performance differences on your own hardware.

We’d love for you to tear into the code, run the benchmarks, and tell us where we can improve. I’ll be here all day to answer any questions about the architecture, kernel fusion, or HCI in the age of local LLMs.

More in depth tech info is here: https://dev.to/constant_chen_/sipp-a-local-first-runtime-for...

Scattered Spider Hackers Plead Guilty on Day 1 of Trial