Basically zero using a 7700S in Chrome on Ubuntu though. Must be not working at all, as the performance difference can't be that large.
> I want to wash my car. The car wash is 50 meters away. Should I walk or drive?
Edit: this is running with MLX, and not with WebGPU on the browser, also it is a fail.
"""
You should *walk*.
Here’s why:
- *Distance*: 50 meters is very short — about 0.3 miles or 0.5 kilometers.
- *Time*: Walking takes roughly 2–3 minutes. Driving would take longer due to parking, starting the engine, and maneuvering.
- *Effort*: Walking is light exercise and avoids parking hassle.
- *Safety*: Less risk of misjudging parking space or damaging the car.
Unless you have a disability that makes walking difficult, walking is the most practical and efficient choice.
"""
Can't see any real value in using 1 bit llms except and apart for the sake of running it for demo or feel good factor of running a higher param model locally and in limited hardware!
Since use as it is very limited, could someone throw some light on any actual use case?
Locally, it's definitely not the full 3.6 27b, but for ~6G with the context, it's quite impressive, and I like what this implies for larger models. Speeds on the 1255u are abysmal, though (~1TPS or less) - granted, CPU-only.
I'm getting ~0.5 tps from [qwen], and ~10 tps from [gemma] - roughly the same size, same quant, same hardware (8-core CPU) same software (llama.cpp).
[qwen] Qwen3.6-27B-Q4_K_M.gguf
[gemma] gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf
``` Could not load.
Error: failed to call OrtRun(). ERROR_CODE: 1, ERROR_MESSAGE: Non-zero status code returned while running GroupQueryAttention node. Name:'/model/layers.0/attn/GroupQueryAttention' Status Message: Failed to create a WebGPU compute pipeline: A valid external Instance reference no longer exists. ```
Me:
> Describe the process of pasteurisation
Response:
> pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the the, and and and, and process, and and the, past, and and and past, and the, the, and the past, the the, process, process, and past, the past, past, the and, the past, and and and and and and and and and the, and and and and and and and, the the, the the, the or and and, the the, the and the, which the, and the, the the, past, and n the process, and and and and, past, and, and and, the past, and, the the, past,, and the, the the, the is the, past, and and and and and, and and and past, the and the, the the, the the, are and past, and which the, the and n, n the, the n past, past, n the, and n, the the process, which past, the the, the n, the the, the is past, the the, is, past, the the, past, and past, process, the the, the the, the and and the, and which past, the
(and that basically just goes on and on like that)
But yes, this will happen. 1.7B is already a tiny model, running that at 1-bit is going to lead to some behavioral issues and will need tuning of temperature/topP/repeatPenalty/etc, and probably a software fallback that detects pathological cases like this and simply retries. Just like we all had to do in the times of GPT 2 and GPT 3
My demo uses 2 bit quantization to run llama3 models on any device with enough ram.
They are here to route things, do tool calls, ask experts.
Imagine a prompt:
```
INSTRUCTION: You're a customer service agent that routes customer service requests.
CUSTOMER MESSAGE: I haven't received my order and want a refund.
CONTEXT: A signature is required at delivery. No one was available to sign.
You can route to one of:
* Billing & refunds
* Shipping & tracking
* Order cancelation
* Level II customer service
```
The dumb model, e.g. 1.7B, wrongly routes to Billing. The smart model, e.g. GPT-5.6-Medium, correctly routes to Shipping. It matters. In the real world, the CONTEXT will even be 10-100x larger and noisier.
I didn't see any further work in this direction however. The paper seems to have been mostly overlooked.
Much more important is the total amount of memory needed by an LLM, which can be decreased either by reducing the number of bits per parameter or the number of parameters. Arithmetic circuits are already cheap so for the inference cost the amount of memory transfers is more important.
It is not clear yet which number of bits per parameter allows the minimization of the total memory requirement, at a given LLM quality.
stfurkan•16h ago
1. https://github.com/stfurkan/bitgpu
2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine
mgoetzke•16h ago
Guessing there is some work to do still
``` calculate({"expression":"g"}) → error: only numbers, + - * / and parentheses are allowed error: No user query found in messages. What is 8+5? Use the calculator get_time({}) → Mon Jul 20 2026 10:36:28 GMT+0200 (Central European Summer Time) error: No user query found in messages. hi Hello! 2 tokens · 6.2 tok/s · prefill 4563 ms · confidence 87% which tools do you know about ? I'myepic know about the following are two tools I am aware of course, I currently i know about I know about me know I am aware of course I know I know I am aware of the following
calculate({"expression":" I know I am aware of know I am aware of I am aware I know I know know "}) → error: only numbers, + - * / and parentheses are allowed error: No user query found in messages.```
stfurkan•16h ago
alex7o•16h ago
stfurkan•16h ago
luca-3•13h ago
Also, do you have data on the technical requirements for this? If someone uses an old phone, e.g., does it still work?
stfurkan•12h ago
It needs webgpu support but I haven't had a chance to test with different devices.
EGreg•12h ago
Want to connect and discuss it? https://calendly.com/safebotsai
stfurkan•12h ago
espetro•11h ago
Besides it, how are you dealing with the harness the model can use on web? I'm currently building https://github.com/bolojs/bolo to extend the base harness agents can use on web and I'm interested in hearing from your experience on crafting tools for bitgpu
stfurkan•7h ago