The neural model under the hood is a CLAP-style model (CLIP for audio): basically during the training you force the model make inner text and audio representations to match. And then you can use such model as 0-shot audio classifier. For POC I used LAION’s clap-htsat model but it turned out to be quite "heavy" so I’ve trained a smaller one by myself.
As for text encoder I used model from the EfficientAT family; for audio I used jhu-clsp/ettin-encoder-32m. The training was performed on my mighty 2x1080 Ti home server. The trained weights can be found on my HF: https://huggingface.co/mirth/earshot-clap-mn10-ettin32
I would appreciate it if you could test this application (especially if you could report if an app missed the sound).
A couple of disclaimers:
1. The sound never leaves you device.
2. It can miss the sound so it’s not a replacement for your monitoring system (but an extra pair of ears).
3. I’m aware that similar functionality is presented in the Apple HomePod but there is a minor difference: to add very custom sound for detection on HomePod you need first to record it (to train inner HomePod model I presume). In my case you can just describe the sound in the text format.
Happy to answer any questions!