9.7 tokens/sec actually seems like a lot! That’s fun!
althea_tx•20m ago
This is a really cool project. Thanks for sharing!
rao-v•9m ago
This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
chrishynes•6m ago
Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
cr125rider•32m ago