frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

https://github.com/RunanywhereAI/trnrt
9•sanchitmonga22•1h ago
Hi HN, Sanchit here from RunAnywhere (YC W26). We just open-sourced TrnRT: Trainium RunTime, our inference engine for AWS Trainium. We started 3 weeks back, and currently we’re able to serve and run GLM 5.3 flash (FP8) on AWS trn2.48xlarge, and will bring more models support soon. It’s open to contributions :)

What it does today:

- Serves 2 instances of GLM-5.3 flash on a single box, 8 trn2 with a cache aware router.

- Can process 3.57M TPM (tokens per minute) on SemiAnalysis's AgentX benchmarks.

- Runs decode as one NKI kernel per layer half, or for the 11 sparse-attention layers fuses both blocks into a single kernel, for faster performance.

We started working on Trn2 because we wanted to explore it and see if we can come close to beating it’s performance in terms of cost efficiency compared to H100/H200. A big MoE with 1M context window such as GLM-5.3 flash requires a lot of memory, and memory/tb on a trn2.48xlarge costs less than half compared to 8x h200 on AWS, and hence we wanted to experiment with it and now we’re sharing some early results.

But AWS's own stack didn't support the model: vLLM Neuron's latest release did not register any GLM model, and NxDI, AWS's own distributed inference framework, is no longer under active development. We also saw the drop of SGLang Neuron implementation yesterday, but there is no good support for Trn2, which is the only chip available for public use right now.

We ran semiAnalysis's AgentX benchmarks, which replays real coding agent sessions:

- 1 box served 28.9 RPM(requests per minute) with 32 agents at batch 12 (also known as concurrency)

-The P50 TTFT with a cached prompt at 16 agents is ~1.24 seconds with a decode of 35.9 tps.

Initially, the recurrence in one kda (linear attention) layer was ~2% of the layer's attention time at batch 1, and the rest was projections, transposes and XLA glue between small kernels.

So we wrote the megakernel so that it covers the whole layer in 2 kernel dispatch, which made decode 25% faster at batch 8.

The sparse attention kernel keep a layer’s working weights in SBUF, distributing weighted streaming across the cores and handling the interchip communication within

There’s an approximation in the sparse-attention prefill, affecting about 1.2% of block selections.

You can refer to the write-up for the profiling results:

https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-mega...

Please give it a try yourself on a trn2.48xlarge and reproduce the AgentX results by following the steps in docs/RESULTS.md. Happy to collaborate and hear any feedback.

Deploying high performant dedicated inference at lab like performance and half the price for open models such as the GLM-5.3 family, Kimi K3, Deepseek flash etc. is how we make money. You can talk to us regarding the same at founders@runanywhere.ai.

We hope you find this useful. If you are working with Trainium or have experience with NKI kernels, we'd love to get your honest feedback. Feel free to share the models you want support for next, and we welcome any contributions to the repository.

Comments

bookernath•11m ago
Tokens per minute is a strange metric, I usually see tok/s. A per-second figure would be more accessible in comparing this to alternatives.

Show HN: Alloy CSS, dialogs, menus and tabs that work with JavaScript off

https://alloyhtml.com/example.html
89•prafandy•2h ago•12 comments

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

https://github.com/RunanywhereAI/trnrt
10•sanchitmonga22•1h ago•1 comments

Show HN: VennDB: append-only in-memory Rust DB for rows queried using bit flags

https://github.com/plabayo/venndb
30•gdcbe•1d ago•2 comments

Show HN: Hue MD – Markdown that shows status at a glance and stays plain text

https://spingras.github.io/hue-md/
2•spingras•32m ago•0 comments

Show HN: Introduction to GPU Inference, a short course with Interactive diagrams

https://athrael.net/courses/gpu-inference/
2•athrael-soju•36m ago•0 comments

Show HN: Zigzag, a native Mac browser for people leaving Arc

https://github.com/ZigaoWang/zigzag
98•ZigaoWang•3d ago•61 comments

Show HN: Soccer-Bench – Watch teams of LLM agents play 5v5 soccer

https://darksinian.github.io/soccer-bench/
3•Darksinian•50m ago•0 comments

Show HN: Moonswing – a Linux desktop environment built with Flutter

https://github.com/miracle-wm-org/moonswing
4•mkosarek•1h ago•1 comments

Show HN: Atli – MCP server that lets AI agents buy and install travel eSIMs

https://heyatli.com/mcp
4•peterhoang•2h ago•0 comments

Show HN: Agav – an open-source terminal agent designed to waste fewer tokens

https://github.com/prapaa-ai/agav
16•GeradeMoleculee•3h ago•7 comments

Show HN: An open virtual office to cowork with humans while your agents work

https://flat.social/virtual-office-platform
11•pawelwentpawel•3h ago•3 comments

Show HN: Sutra-OS – A self-hosted ontology platform

https://github.com/0nelight/sutra-os
64•0nelight•18h ago•34 comments

Show HN: Belay – formal verification for SQLite migrations and queries in Lean 4

https://github.com/vihren-dev/belay
3•neprotivo•4h ago•0 comments

Show HN: Orchard – Turn a Linux server into your own cloud

https://orchard.hackclub.com
4•leafd•4h ago•1 comments

Show HN: Show HN: PWT Today – A Daily Pokémon World Tournament Browser Run

https://pwt-today.com/
2•bigpapikite•4h ago•0 comments

Show HN: Pulsar – cross-harness agent to agent communication and messaging

https://github.com/yocontra/pulsar
19•contrahax•14h ago•3 comments

Show HN: Open-source, real-time object tracking for live video feeds

https://github.com/aryamandarda/tamserve
34•aryaman_darda•20h ago•2 comments

Show HN: Shoptimist – Amazon but Curated

https://shoptimist.co/
5•MisterJoe•5h ago•2 comments

Show HN: MeshLive – Offline Video calls and livestreams over Bluetooth mesh

https://meshlive.github.io/
2•gptgautam1•8h ago•0 comments

Show HN: Fakemetrics.com – "build in public" like a superstar

https://fakemetrics.com/
49•tigranbs•23h ago•21 comments

Show HN: Toolbelt – Free developer utilities that run in the browser

https://gettoolbelt.dev/
6•gprakasha234•14h ago•1 comments

Show HN: ThingStudio, Node-RED style programming for microcontrollers

https://thingstudio.net/
39•mkarliner•6d ago•8 comments

Show HN: Birds as Code

https://erl-j.github.io/fun-zone/birds.html
18•abdljasser2•22h ago•10 comments

Show HN: Free weekly backups for Supabase Free projects (pg_dump to R2)

https://github.com/liulewis/supabase-free-backup-lite
10•agricalc•16h ago•0 comments

Show HN: A terminal-native idle factory game, written in Rust with ratatui

https://gitlab.com/snipedbywifi/terminactory
172•snipedbywifi•3d ago•44 comments

Show HN: Deslop skill based on corpus-linguistics and psycholinguistics research

https://github.com/etodanik/deslop
16•dannypovolotski•1d ago•4 comments

Show HN: The Last Complication, a Zachtronics-style watchmaking puzzle game

https://lastcomplication.com/
86•tdeangelis•1d ago•18 comments

Show HN: Carton Clock – A working pendulum clock made from empty drink cartons

https://www.niklasroy.com/carton-clock/
87•r0r0•4d ago•2 comments

Show HN: db-branch – database branches in seconds for AI agents and CI, via ZFS

https://github.com/db-branch/db-branch
45•nu_nu11•2d ago•16 comments

Show HN: NOS x86-64 hypervisor, written in Rust

https://nos-os.tech/en/demo
4•irqlevel•19h ago•0 comments