I switched to Colima and pointed docker context use colima. Then added Portainer for UI (docker run -d -p 9000:9000 ...).
With the same compose setup, Colima usage dropped to ~0.2%
Portainer provides containers/images/console UI
All while keeping the Docker CLI workflow
Full write-up (with screenshots) here: https://muthuishere.medium.com/docker-desktop-colima-portainer-my-mac-finally-breathes-b70d82d6cf0f