frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

https://github.com/RunanywhereAI/trnrt
1•sanchitmonga22•15m ago
Hi HN, Sanchit here from RunAnywhere (YC W26). We just open-sourced TrnRT: Trainium RunTime, our inference engine for AWS Trainium. We started 3 weeks back, and currently we’re able to serve and run GLM 5.3 flash (FP8) on AWS trn2.48xlarge, and will bring more models support soon. It’s open to contributions :)

What it does today:

- Serves 2 instances of GLM-5.3 flash on a single box, 8 trn2 with a cache aware router.

- Can process 3.57M TPM (tokens per minute) on SemiAnalysis's AgentX benchmarks.

- Runs decode as one NKI kernel per layer half, or for the 11 sparse-attention layers fuses both blocks into a single kernel, for faster performance.

We started working on Trn2 because we wanted to explore it and see if we can come close to beating it’s performance in terms of cost efficiency compared to H100/H200. A big MoE with 1M context window such as GLM-5.3 flash requires a lot of memory, and memory/tb on a trn2.48xlarge costs less than half compared to 8x h200 on AWS, and hence we wanted to experiment with it and now we’re sharing some early results.

But AWS's own stack didn't support the model: vLLM Neuron's latest release did not register any GLM model, and NxDI, AWS's own distributed inference framework, is no longer under active development. We also saw the drop of SGLang Neuron implementation yesterday, but there is no good support for Trn2, which is the only chip available for public use right now.

We ran semiAnalysis's AgentX benchmarks, which replays real coding agent sessions:

- 1 box served 28.9 RPM(requests per minute) with 32 agents at batch 12 (also known as concurrency)

-The P50 TTFT with a cached prompt at 16 agents is ~1.24 seconds with a decode of 35.9 tps.

Initially, the recurrence in one kda (linear attention) layer was ~2% of the layer's attention time at batch 1, and the rest was projections, transposes and XLA glue between small kernels.

So we wrote the megakernel so that it covers the whole layer in 2 kernel dispatch, which made decode 25% faster at batch 8.

The sparse attention kernel keep a layer’s working weights in SBUF, distributing weighted streaming across the cores and handling the interchip communication within

There’s an approximation in the sparse-attention prefill, affecting about 1.2% of block selections.

You can refer to the write-up for the profiling results:

https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-mega...

Please give it a try yourself on a trn2.48xlarge and reproduce the AgentX results by following the steps in docs/RESULTS.md. Happy to collaborate and hear any feedback.

Deploying high performant dedicated inference at lab like performance and half the price for open models such as the GLM-5.3 family, Kimi K3, Deepseek flash etc. is how we make money. You can talk to us regarding the same at founders@runanywhere.ai.

We hope you find this useful. If you are working with Trainium or have experience with NKI kernels, we'd love to get your honest feedback. Feel free to share the models you want support for next, and we welcome any contributions to the repository.

The Benefit of Art – George Eliot, the Natural History of German Life [pdf]

https://www.laits.utexas.edu/farrell/documents/NaturalHistory.pdf
1•tylerdane•5s ago•1 comments

Mistral offers three months of Vibe Pro free in Europe

https://xitter.cf/MistralAI/status/2122713540666806397
1•DavCreator•53s ago•1 comments

Local first, OSS meeting notetaker

https://merkwerk.dev/
1•arzelaascoli•1m ago•0 comments

DNA Damage and Lifespan

https://www.science.org/content/blog-post/dna-damage-and-lifespan?referrer=https%3A%2F%2Fnews.goo...
1•bookofjoe•1m ago•0 comments

Show HN: A live globe of the gamedev GitHub pushes

https://worldspace.crankyfuse.io/
1•crankyfuse•1m ago•0 comments

Show HN: ChineseHuangli: A modern spin of traditional Chinese almanac calendar

https://chinesehuangli.com/
1•solous•1m ago•0 comments

Show HN: Hopkey, one ⌘T command bar for Safari, Chrome, Brave, Edge and Vivaldi

https://hopkey.app/
1•en3sis•1m ago•0 comments

Horizon Labs

https://horizon-labs.ai/
1•pretext•2m ago•0 comments

AMD ends confusion during DDR5 memory training, thanks to this new BIOS feature

https://www.club386.com/amd-asus-memory-training-debug-bios-update/
1•robin_reala•2m ago•0 comments

Disney plans $3B in cuts and more layoffs even as its profit doubles

https://www.martincid.com/business/disney-3-billion-cost-cuts-more-layoffs-profit-doubles/
1•thunderbong•3m ago•0 comments

Vulnerability Garden – A list of named vulnerabilities, attacks, and exploits

https://vulnerability.garden/
1•OuterVale•4m ago•0 comments

Bolt's Scooters Are Becoming Accessibility Sensors

https://geoawesome.com/bolt-scooters-accessibility-mapping-wheels4wheels/
1•chippy•4m ago•0 comments

Show HN: An AI app controlled from your phone's Picture in Picture

https://barnacle.qa
2•billychasen•4m ago•0 comments

Built a free resume edit tool thats landing offers

https://v3.edit2offer.com/
1•jacksonwhole•4m ago•0 comments

Optimizely Cyberattack Exposes Visitors to Malware on 178 Organisations

https://fish4news.co.uk/optimizely-malware-attack-178-organisations-websites/
1•fairytalemtg•4m ago•1 comments

Giving AI Agents a Better Way to Debug CSS

https://www.amitmerchant.com/giving-ai-agents-a-better-way-to-debug-css/
1•speckx•5m ago•0 comments

Break My Guardrails – AI sandbox escape challenge

https://playground.haikeilabs.com/
1•nhumrich•5m ago•0 comments

Ask HN: Are open-weight models like Qwen used in production?

1•k-thimmaraju•6m ago•0 comments

Strengthening NATO's Cyber Defenses

https://openai.com/index/strengthening-nato-allied-cyber-defences-ai/
1•pretext•6m ago•0 comments

ShieldFont swaps words in HTML, humans read your words, AI scrapers get a decoy

https://shieldfont.org/
1•ChrisArchitect•7m ago•0 comments

Show HN: Quickertype – quicktype in Rust, same output, up to 89x faster

https://github.com/EloB/quickertype
1•FragaOlle•7m ago•0 comments

Gemini is finally changing their privacy policy

1•EbNar•7m ago•0 comments

The unlikely queen of the bagel business

https://thehustle.co/newsletters/04-11-2026
1•paulpauper•7m ago•0 comments

The Buddhist A.I. Sect That Wants to Take over Silicon Valley

https://www.newyorker.com/magazine/2026/11/23/the-buddhist-ai-sect-that-wants-to-take-over-silico...
3•paulpauper•7m ago•0 comments

Show HN: Confidential Cloud – CPU and GPU Confidential VMs

https://console.confidential.ai/sign-in?returnTo=%2Finstances
1•geomatrick•8m ago•0 comments

Show HN: Peppy, a simplified robotics framework for regular devs

https://docs.peppy.bot/quickstart/
1•Ekami•8m ago•0 comments

Show HN: Puzzle for AI shopping agents inside our skincare store

https://keralabotanics.com/
1•rauschermate•9m ago•0 comments

Charles Schwab website goes down

https://www.businessinsider.com/charles-scwab-website-outage-reports-2026-11
2•tedd4u•10m ago•0 comments

Jensen Huang Breaks Silence, Signals Willingness to Return as Nvidia CEO

https://www.wsj.com/tech/jensen-huang-statement-nvidia-ceo-9dedca36
2•spenvo•11m ago•0 comments

RL Misalignment Under Accurate Rewards and Explicit Requirements

https://www.lesswrong.com/posts/yNYco6JrA3Q9rLMJS/rl-misalignment-under-accurate-rewards-and-expl...
1•faxipay349•11m ago•0 comments