frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

https://github.com/RunanywhereAI/trnrt
7•sanchitmonga22•1h ago
Hi HN, Sanchit here from RunAnywhere (YC W26). We just open-sourced TrnRT: Trainium RunTime, our inference engine for AWS Trainium. We started 3 weeks back, and currently we’re able to serve and run GLM 5.3 flash (FP8) on AWS trn2.48xlarge, and will bring more models support soon. It’s open to contributions :)

What it does today:

- Serves 2 instances of GLM-5.3 flash on a single box, 8 trn2 with a cache aware router.

- Can process 3.57M TPM (tokens per minute) on SemiAnalysis's AgentX benchmarks.

- Runs decode as one NKI kernel per layer half, or for the 11 sparse-attention layers fuses both blocks into a single kernel, for faster performance.

We started working on Trn2 because we wanted to explore it and see if we can come close to beating it’s performance in terms of cost efficiency compared to H100/H200. A big MoE with 1M context window such as GLM-5.3 flash requires a lot of memory, and memory/tb on a trn2.48xlarge costs less than half compared to 8x h200 on AWS, and hence we wanted to experiment with it and now we’re sharing some early results.

But AWS's own stack didn't support the model: vLLM Neuron's latest release did not register any GLM model, and NxDI, AWS's own distributed inference framework, is no longer under active development. We also saw the drop of SGLang Neuron implementation yesterday, but there is no good support for Trn2, which is the only chip available for public use right now.

We ran semiAnalysis's AgentX benchmarks, which replays real coding agent sessions:

- 1 box served 28.9 RPM(requests per minute) with 32 agents at batch 12 (also known as concurrency)

-The P50 TTFT with a cached prompt at 16 agents is ~1.24 seconds with a decode of 35.9 tps.

Initially, the recurrence in one kda (linear attention) layer was ~2% of the layer's attention time at batch 1, and the rest was projections, transposes and XLA glue between small kernels.

So we wrote the megakernel so that it covers the whole layer in 2 kernel dispatch, which made decode 25% faster at batch 8.

The sparse attention kernel keep a layer’s working weights in SBUF, distributing weighted streaming across the cores and handling the interchip communication within

There’s an approximation in the sparse-attention prefill, affecting about 1.2% of block selections.

You can refer to the write-up for the profiling results:

https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-mega...

Please give it a try yourself on a trn2.48xlarge and reproduce the AgentX results by following the steps in docs/RESULTS.md. Happy to collaborate and hear any feedback.

Deploying high performant dedicated inference at lab like performance and half the price for open models such as the GLM-5.3 family, Kimi K3, Deepseek flash etc. is how we make money. You can talk to us regarding the same at founders@runanywhere.ai.

We hope you find this useful. If you are working with Trainium or have experience with NKI kernels, we'd love to get your honest feedback. Feel free to share the models you want support for next, and we welcome any contributions to the repository.

What is the best harness for each model?

https://runta.com/blog/introducing-frontierharness-eval-v1-1/
151•Edward40•2h ago•67 comments

Show HN: Alloy CSS, dialogs, menus and tabs that work with JavaScript off

https://alloyhtml.com/example.html
83•prafandy•2h ago•12 comments

New Evidence Shows School Phone Bans Are Gradually Improving Test Scores

https://www.nytimes.com/2026/11/17/upshot/school-phone-bans-study-scores.html
801•donohoe•5h ago•417 comments

Carnegie Mellon is putting computer science courses on YouTube for free

https://www.cs.cmu.edu/learn/
2102•pcompeau•20h ago•282 comments

First Photos of the Poles of a Rod-Shaped Bacterium

https://nautil.us/check-out-the-first-photos-of-the-poles-of-a-rod-shaped-bacterium-1286664
72•dustfinger•1d ago•16 comments

NSA Internal Network Down

https://www.reuters.com/technology/highly-classified-us-national-security-communications-network-...
111•nodoodles•1h ago•52 comments

YC wrote software to run our internship expo – and an 8-bit SIM to show it off

1•ryankicks•34m ago

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

https://github.com/RunanywhereAI/trnrt
9•sanchitmonga22•1h ago•1 comments

VisualBasic.World: A gallery of Planet Source Code programs from 1997 to 2012

https://visualbasic.world/
170•broast•1d ago•62 comments

Optimizely compromised, served malware on Zoom, GitLab

https://blog.report-uri.com/worlds-largest-brands-infected-with-malware-via-optimizely/
573•kristjansson•18h ago•228 comments

How the Claude Code team thinks about verification

https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/
161•craigmart•7h ago•148 comments

Rows vs. Columns: How different are they, really?

https://spiraldb.com/blog/rows-vs-columns
110•eatonphil•5d ago•40 comments

Exploiting Replay Protected Monotonic Counters to Brute-Force Windows Hello PINs

https://arxiv.org/abs/2611.18126
10•cwerling•1d ago•1 comments

Antfly learned to stop worrying and love Apache 2.0

https://antfly.io/thoughts/how-antfly-learned-to-stop-worrying-and-love-apache-2
15•drewlanenga•2h ago•3 comments

Turbo Pascal Compiler Written in Turbo Pascal

https://turbo51.com/compiler-design/tpc16-turbo-pascal-compiler-written-in-turbo-pascal
77•andsoitis•1d ago•30 comments

BugMeNot has closed

https://www.bugmenot.com/
684•lode•20h ago•312 comments

Elephant-Goldfish Model

https://egm.dev/
10•_tk_•1d ago•9 comments

Optimizely hack infected CNN, Fox News, BBC, Salesforce sites with malware

https://www.bleepingcomputer.com/news/security/optimizely-supply-chain-attack-delivers-malware-to...
12•balderdash•21m ago•1 comments

Charles Schwab website goes down

https://www.businessinsider.com/charles-scwab-website-outage-reports-2026-11
30•tedd4u•1h ago•6 comments

We don't need more apps. We need better open source

https://jordankrueger.com/blog/better-open-source-not-more-apps/
266•pandemicsoul•2h ago•164 comments

Nimony: A New Compiler for Nim

https://nimony.org/
151•Fervicus•1d ago•73 comments

If an App Asks for Full Disk Access or System and App Protection Override

https://support.apple.com/en-us/149699
209•jshier•15h ago•111 comments

Why 3dfx 1 cards had such a washed out color output?

https://twitter.com/id_aa_carmack/status/2122478083319992513
251•mfiguiere•17h ago•90 comments

Show HN: VennDB: append-only in-memory Rust DB for rows queried using bit flags

https://github.com/plabayo/venndb
30•gdcbe•1d ago•2 comments

Highly classified US national security communications network goes down

https://www.cnn.com/2026/11/17/politics/highly-classified-national-security-communications-networ...
185•monkeydust•1h ago•171 comments

Chemistry is Actually Geometry

https://chillphysicsenjoyer.substack.com/p/chemistry-is-actually-geometry
309•sebg•21h ago•130 comments

Show HN: Zigzag, a native Mac browser for people leaving Arc

https://github.com/ZigaoWang/zigzag
97•ZigaoWang•3d ago•61 comments

Comcast's "fiber-powered Wi-Fi" is cable, AT&T says in false advertising case

https://arstechnica.com/tech-policy/2026/11/comcast-falsely-advertises-cable-internet-as-fiber-po...
230•jaredwiener•20h ago•202 comments

Frame – 31.5" color ePaper display

https://trmnl.com/blog/introducing-frame
510•geerlingguy•1d ago•376 comments

Microsoft blames monster email outage on software defect introduced in July fix

https://www.theregister.com/saas/2026/11/16/microsoft-blames-monster-email-outage-on-software-def...
429•Bender•1d ago•317 comments