Being able to record and manage takes directly in Blender would be an awesome feature and first thing that pops into mind :)

I would love to play around with this, but I don't own an iphone :/ using a webcam + local model for detection as input would be awesome

Yessssssss!!!!! However, we need something that doesn’t rely on iPhone. We need webcam. You can use your iPhone as a webcam. You can also use more powerful video devices as a webcam. I would love a DIY mudface map that’s a b/w displacement map so you can capture the wrinkles of the face and map that with blender trackers. Seriously though, this is a huge leap towards that future.

There's a nice Blender extension called "FaceIt" that I used a few years ago for rigging and producing ARKit-compatible facial animations and characters. It worked quite well (for what it was designed), and I recommend it!

https://superhivemarket.com/products/faceit

>Faceit is a Blender Add-on that assists you in creating complex facial expressions for arbitrary 3D characters.

>An intuitive, semi-automatic and non-destructive workflow guides you through the creation of facial shape keys that are perfectly adapted to your 3D model's topology and morphology, whether it’s a photorealistic human model or a cartoonish character. You maintain full artistic control while saving a ton of time and energy.

https://faceit-doc.readthedocs.io/en/latest/FAQ/

This is a great explanation of how FaceIt works, facial animation, shape keys, face rigs, ARKit, etc:

This addon automates Facial Animation (FACEIT Tut 1)

https://www.youtube.com/watch?v=KQ32KRYq6RA&list=PLdcL5aF8Zc...

Would it be possible to integrate directly into Blender instead of an add on? If so I think Blender is GPLv2 and this is GPLv3. If merg8ng is something you see in the future you may want to change that.