I wrote this up after having written a tutorial on doing this in GenStage around a decade ago, and thought it was interesting to have the two of them side by side to consider. I linked the original in the document linked to here.
Overall, I'm wildly impressed at how this Elixir code held up, and it was a joy to revisit this.
dangerousnight•2h ago
As someone who has been learning Elixir on and off for over a year, this looks really exciting on first skim through. Excited to give this a deep read this weekend!
rhgraysonii•2h ago
If you have any feedback or anything is unclear feel free to open an issue. I am thinking I am going to take this and expand the concepts to start as a beginners primer going through the primary concepts in brief akin to Elixir School, and then expand it into building this and a web service that is using it and offering some real time features.
sbrother•2h ago
FYI if anyone is looking for a production ready job runner in Elixir, I strongly recommend Oban. They have a paid pro version but the open source version is still absolutely fantastic.
victorbjorklund•2h ago
Just to add to the parent the Open Source version and the Pro Version just differs in that the Pro Version has a few plugins that the normal version has (for example workflows where you can have multiple workers working together) but it is not something you need for most use-cases.
pikdum•43m ago
It's more than just a few - even more basic things like rate limiting or concurrency controls are gated behind Pro. It works extremely well, but I've been reluctant to use it in open source projects because there's quite a bit in there I'd need to rebuild.
rhgraysonii•2h ago
Oban is great! This is definitely just met to be a learning exercise. But it could go quite a ways on its own in prod.
cpursley•1h ago
I’d love to see one for Elixir built around the Postgres pgmq and pg_crom extensions similar to what pgflow is doing: https://www.pgflow.dev/
rhgraysonii•1h ago
That would be super cool. Pretty much everything Supabase is up to is awesome.
cpursley•1h ago
Yeah, their realtime thing is powered by Elixir (listens to Postgres WAL).
jamal-kumar•1h ago
The absolute facility with which you can do distributed applications in Elixir is the big thing that sold me in the first place. Nice to see this tutorial!
hmmokidk•1h ago
Incredibly well written. You clearly put a lot of thought and effort into this. Thank you.
Also FOR UPDATE SKIP LOCKED is interesting.
rhgraysonii•1h ago
You also could use an advisory exact lock, which would serialize the order of input and force everyone to wait in line, but the advantage here is that we are allowed shared concurrent reads while maintaining our lock. Thank you for the feedback. It was fun to revisit after 10 years. The initial piece came after a Columbus, OH Ruby meetup that Jose attended. It was quite fun to see him put it together as a whole and I ran with things a bit further than his first ideas with this.
huqedato•43m ago
Years ago I built something similar, a Pub/Sub notification system: https://github.com/huqedato/qnotix. It is still running (with small modifications), in production, at my ex-customer.
Such a pity the industry (customers) reacts with skepticism every time I propose solutions based in Elixir/Erlang. I always hear: "Elixir, what? We want Java/.Net/Python/php"
jbverschoor•27m ago
It means they don't trust your judgement, aka you're just there to execute their plans. You're the floor guy, not the architect, owner, or developer of a house.
huqedato•20m ago
Not really. I am usually the one they pay to solve their problems. It's not a matter of trusting my judgement, rather they are trapped in an institutional/corporate mindset (old patterns are most suitable, 'best practice', 'reliable, proven tech' and such)
rhgraysonii•5h ago
Overall, I'm wildly impressed at how this Elixir code held up, and it was a joy to revisit this.