I’ve been quietly working on this in my free time since around March and while there is still a lot to do, I’m feeling really happy with where things are at and how well it all works. Zizq is a single binary, zero-dependency job queue that sits in any stack and can enqueue and process jobs across programming language boundaries. It’s designed to be really easy to get up and running with, and it’s designed to make it easy to manage jobs that have already been enqueued.

I’ve been having a lot of fun thinking hard, designing and implementing this, and it solves some real problems I’ve come across building software at non-trivial scale. My favourite feature to this point has probably been batched jobs, where successive enqueues of jobs of the same type are folded/merged into one another to be processed as a larger batch when eventually picked up by a worker.

As of now, Zizq can already:

* Do all the basic FIFO / backoff + retry / concurrent worker stuff you just expect * Efficiently schedule jobs to run in the future * Set up custom retention and backoff policies * Enqueue to and process from any number of queues * Idempotently set up recurring (a.k.a cron) schedules for jobs * Ensure enqueue-time uniqueness of jobs * Use jq-expression filters to identify jobs by their payload contents * Update existing jobs on the queue using the same filters as above * Fold successive enqueues of the same type of job into one another to act as one larger job * Provides control and visibility through a Terminal UI (`zizq top`) * Use the Ruby, Node or Rust official clients on top of its HTTP API (more to follow…)

In terms of implementation, it’s built on a LSM-tree database (fjall) and some constrained use of lock-free data structures (skip lists/skip sets) which it turns out is really fast and durable/recoverable across crashes and power loss.

I’ve still got heaps I want to get done/will get done — currently working on per-job concurrency control/rate limiting — but would love any feedback on these early beginnings. It’s pretty much download + read the docs for whatever client you use (or the underlying HTTP API if you want to go down that path).