Believe it or not, it's still super common for research / data-analysis teams to handoff work to Engineering because they aren't able to easily scale analyses on their own.
Engineering managers we spoke with frequently attempt to onboard these teams to tools like Apache Airflow, Prefect, Dask, or Ray, but these tools come with hidden tradeoffs and complexity that researchers often struggle with on their own. This is can be very painful for engineering managers who wind up acting like highly paid support/education staff, instead of focusing their real work. It's also very painful for research teams who are often blocked waiting on engineering/devops to help them.
This is why we created Burla, it's designed to be simple enough for even complete beginners to scale python across thousands of VM's.
It's open-source, works with GPU's, custom containers, and up to 10,000 CPU's in a single cluster.
We just launched our managed service and are giving away free compute to anyone who we think might a good fit, if you want to give it a try please let us know, and we'll send you a personalized instance today!