Author here, welcoming feedback and ideas for how you'd approach challenges such as these. There was a long road of operational tasks to get to a "quick" iteration loop in my debug UI, that I'll be ready to share that story once my new ranker is outperforming my old one. As of right now, the entire data + ranking operation runs on an few-years-old gaming PC with 24GB of ram and 12 cores.

The NMF model was fairly simple to get off the ground, thanks to the maturity of the deployment ecosystem surrounding it. Getting the Set Transformer to beat it in my internal benchmarks has been a journey, hopefully I'll share later this year!