So it's decorators for scheduling (@scheduler.every(5).minutes, @scheduler.daily.at("09:00")), state saves to JSON so jobs survive restarts, and there's an optional FastAPI dashboard if you want to see what's running.
No Redis, no message broker, runs in-process with your app. Trade-off is it's single process only — if you need distributed workers, stick with Celery.
I can think of two major ways to operationalize a Python script that needs to run continuously. One is with containerization, which usually means Kubernetes, which already has a perfectly fine resource definition for cronjobs. The other approach is to run the script in a bare metal or VM, which would mean defining a service to ensure that the process can be managed appropriately, restarted if it dies, and the like. In other words, defining a service is about just as much effort as defining a cronjob, and there's no escape from some amount of "ops work" that isn't encapsulated in a Python script.
So why not just use the tried-and-true prior art than worry about building and supporting your own secret third thing that others would need to learn, support, maintain, and keep in mind when debugging a problem?
- Form submission in front-end admin panel for users, for "new scheduled job"
- Form allows defining a job name, job type (while I let users define jobs, I limit it to a subset of python functions that I trust but are still general enough), job parameters (just a json blob for kwargs for the python func), frequency, and timeout.
- For the whitelisting of functions it's easiest to have a directory in your src which has the different python functions that are usable, and when you do a "get" for the form you populate the dropdown with the scripts from that directory / some metadata.
- The backend (fastAPI) saves the details in a DB and creates a CRON file for the job, to add to the crontab. I basically have a template bash statement with the timeout built in and logging hooked up to it. And when python functions are called I have a helper which grabs the JSON kwargs from the DB or a filecache (if remote DB and I don't want to hit it every minute or something) to avoid cmd line injection.
- Also have an "edit scheduled job" which opens the same form view but with items pre-populated, and the submit going to a patch endpoint instead of a post.
It's stupid simple but lets users define a range of jobs. Things like having daily backups, where to send the backups, pulling things from / pushing things to an API frequently, etc.
fucalost•3w ago
michielme•3w ago