Why open source local AI benchmarking on Apple Silicon matters - and why your benchmark submission is more valuable than you think.

The narrative around AI has been almost entirely cloud-centric. You send a prompt to a data center, tokens come back, and you try not to think about the latency, cost, or privacy implications. For a long time, that was the only game in town.

Apple Silicon - from M1 through the M4 Pro/Max shipping today, with M5 on the horizon - has quietly become one of the most capable local AI compute platforms on the planet. The unified memory architecture means an M4 Max with 128GB can run models that would require a dedicated GPU workstation elsewhere. At laptop wattages. Offline. Without sending a single token to a third party.

This shift is legitimately great for all parties (except cloud ones that want your money), but it comes with an unsolved problem: we don't have great, community-driven data on how these machines actually perform in the wild.

That's why I built Anubis OSS.

The Fragmented Local LLM Ecosystem

If you've run local models on macOS, you've felt this friction. Chat wrappers like Ollama and LM Studio are great for conversation but not built for systematic testing. Hardware monitors like asitop show GPU utilization but have no concept of what model is loaded or what the prompt context is. Eval frameworks like promptfoo require terminal fluency that puts them out of reach for many practitioners.

None of these tools correlate hardware behavior with inference performance. You can watch your GPU spike during generation, but you can't easily answer: Is Gemma 3 12B Q4_K_M more watt-efficient than Mistral Small 3.1 on an M3 Pro? How does TTFT scale with context length on 32GB vs. 64GB?

Anubis answers those questions. It's a native SwiftUI app - no Electron, no Python runtime, no external dependencies - that runs benchmark sessions against any OpenAI-compatible backend (Ollama, LM Studio, mlx-lm, and more) while simultaneously pulling real hardware telemetry via IOReport: GPU/CPU utilization, power draw in watts, ANE activity, memory including Metal allocations, and thermal state.

Why the Open Dataset Is the Real Story

The leaderboard submissions aren't a scoreboard - they're the start of a real-world, community-sourced performance dataset across diverse Apple Silicon configs, model families, quantizations, and backends.

This data is hard to get any other way. Formal chipmaker benchmarks are synthetic. Reviewer benchmarks cover a handful of models. Nobody has the hardware budget to run a full cross-product matrix. But collectively, the community does.

For backend developers, the dataset surfaces which chip/memory configurations are underperforming their theoretical bandwidth, where TTFT degrades under long contexts, and what the real-world power envelope looks like under sustained load. For quantization authors, it shows efficiency curves across real hardware, ANE utilization patterns, and whether a quantization actually reduces memory pressure or just parameter count.

Running a benchmark takes about two minutes. Submitting takes one click.

Your hardware is probably underrepresented. The matrix of chip × memory × backend × thermal environment is enormous — every submission fills a cell nobody else may have covered.

The dataset is open. This isn't data disappearing into a corporate analytics pipeline. It's a community resource for anyone building tools, writing research, or optimizing for the platform.

Anubis OSS is working toward 75 GitHub stars to qualify for Homebrew Cask distribution, which would make installation dramatically easier. A star is a genuinely meaningful contribution.

Download from the latest GitHub release — notarized macOS app, no build required Run a benchmark against any model in your preferred backend Submit results to the community leaderboard Star the repo at github.com/uncSoft/anubis-oss

Max Schwarzer is leaving OpenAI for Anthropic

Tabular Foundation Models Still Need One Thing: Multi-Table Aggregation

AI perspective from a former Block DevRel

Regulator contacts Meta over workers watching intimate AI glasses videos

Show HN: PostgreSQL for AI – A book on pgvector, RAG, and in-database ML

TurboCast – Turn YouTube videos and articles into AI podcasts

More kids, teens injured in e-bike wrecks, study finds

Coasty.ai Is Generally Available

Hello Developer: March 2026

Iran War, Taiwanese Chips, and a Blueprint for Species Survival

Maybe There's a Pattern Here?

What Python's asyncio primitives get wrong about shared state

Manipulating AI memory for profit: The rise of AI Recommendation Poisoning

JWST reveals extreme temperature shifts in Jupiter's auroral footprints

An Evolved Antenna for Deployment on NASA's Space Technology 5 Mission [pdf]

Show HN: Extracted tech from 5.6M sites (plus versions) and made some dashboards

AI Coding Startup Cursor Hits $2B Annual Sales Rate

Hybrid Images: Principles and Implementation

Show HN: A Combinator – A parody YC for AI agents ("Make something agents want")

Show HN: Open dataset of real-world LLM performance on Apple Silicon

Building My Own Canva over the Weekend

"conservation pool" water management could provide a way forward for CO River

AI Thoughts

Once Upon a Boot - visual tracing of boot process

Z80 Sans – a disassembler in a font (2024)

Show HN: MarkNote – Local-First Wysiwyg Markdown Editor (Tauri/Rust)

Ytm-player: full-featured YouTube Music player for the terminal

I left work early to get a hug from my wife

Jensen Huang says Nvidia is pulling back from OpenAI and Anthropic

Google Research Language Explorer

Show HN: Raw Binary Program Analysis Tool