I’ve been using wandb quite a bit and looked into neptune ai and some open source alternatives, but I’ve always felt that collaboration and version control (e.g. associating code snapshots with training runs etc) is clunky. I was also thinking it’d be nice to have some kind of monitoring on my longer runs to alert me on certain criteria, or even be able to stop or restart a run with hyperparam modifications remotely (take potentially agentic actions), like from my phone.

I was curious what all of your experiences have been with these (and similar) AI developer platforms / observability layers and what you’ve found lacking or gripes you have with the existing solutions (if anything). I've found the research process extremely painful and was wondering if this was just me.