Apple adds matmul acceleration to A19 Pro GPU

15•aurareturn•5mo ago

It's equivalent to Tensor cores from Nvidia. This means prompt processing will get a significant boost on Apple devices, making local LLMs far more viable on future Macs (as soon as M5).

Comments

Nokinside•5mo ago

The first SoC including Neural Engine was the A11 Bionic, used in iPhone 8, 8 Plus and iPhone X, introduced in 2017. Since then, every Apple A-series SoC has included a Neural Engine.

runjake•5mo ago

The only official mention I've seen so far was a brief mention in a slide during the presentation.

Useful /r/LocalLlama discussion: https://www.reddit.com/r/LocalLLaMA/comments/1ncprrq/apple_a...

ActorNightly•5mo ago

>, making local LLMs far more viable on future Macs

If you want to run on a Mac Pro or iMac, this will be fine, but at the price points, youd be silly to spend money on either when you can do a dual nvidias for the same ram, and that will be dedicated ram.

For portable Apple devices, the max memory you can get currently is 24GB IIRC and thats probably not going to change any time soon. The only decent model that can run locally is Gemma 27B QAT which will eat up 17gb at the minimum, and that model really struggles with some stuff that you can do for free on ChatGPT or Gemini

So yeah, speed is not gonna matter when results are shit.

Someone•4mo ago

> For portable Apple devices, the max memory you can get currently is 24GB IIRC

It isn’t cheap, but you can buy a 16 inch MacBook Pro with 128GB unified memory today.

cibernox•4mo ago

24gb is the minimum ram in the Pro lineup, the max is 128gb and can run 200B models.

The mac ultra has up 512gb and while expensive has more than twice the memory than any GPU alternative on a similar price point.

What was dragging it behind was the lack of Matmul acceleration, which seems that will change soon. Likely nvidia cards will still be faster and have better support, but paying a very big premium for it (ironic that apple is the cheaper option for once)

andyjohnson0•5mo ago

Please don't Ask as a blogging platform. Just submit a link to an informative article, or comment on an existing article.

https://news.ycombinator.com/newsfaq.html

TOSTracker – The AI Training Asymmetry

The Devil Inside GitHub

Show HN: Distill – Migrate LLM agents from expensive to cheap models

Show HN: Sigma Runtime – Maintaining 100% Fact Integrity over 120 LLM Cycles

Make a local open-source AI chatbot with access to Fedora documentation

Introduce the Vouch/Denouncement Contribution Model by Mitchellh

Software Factories and the Agentic Moment

The Neuroscience Behind Nutrition for Developers and Founders

Bang bang he murdered math {the musical } (2024)

A Night Without the Nerds – Claude Opus 4.6, Field-Tested

Could ionospheric disturbances influence earthquakes?

SpaceX's next astronaut launch for NASA is officially on for Feb. 11 as FAA clea

Show HN: One-click AI employee with its own cloud desktop

Show HN: Poddley – Search podcasts by who's speaking

Same Surface, Different Weight

The Rise of Spec Driven Development

The first good Raspberry Pi Laptop

Seas to Rise Around the World – But Not in Greenland

Will Future Generations Think We're Gross?

State Department will delete Xitter posts from before Trump returned to office

Show HN: Verifiable server roundtrip demo for a decision interruption system

Impl Rust – Avro IDL Tool in Rust via Antlr

Stories from 25 Years of Software Development

minikeyvalue

Neomacs: GPU-accelerated Emacs with inline video, WebKit, and terminal via wgpu

Show HN: Moli P2P – An ephemeral, serverless image gallery (Rust and WebRTC)

How I grow my X presence?

What's the cost of the most expensive Super Bowl ad slot?

What if you just did a startup instead?

Hacking up your own shell completion (2020)