Vercel recently released a "resumable-stream" package (https://github.com/vercel/resumable-stream) to solve this problem using Redis as a backing store.
I felt that we could simplify the approach using s2.dev (I'm an engineer on the team), and created a fork which ended up borrowing the API but otherwise completely rewritten: https://github.com/s2-streamstore/resumable-stream
- S2 streams are cheap, so instead of optimizing to flush only when there is another reader, the stream is always created and written to.
- Unlike the typically ephemeral nature of Redis, tokens are always durable, and there are no memory constraints around number of streams or how much data is on them.
- Relies on S2’s built-in concurrency control. A unique fencing token ensures a single writer, and a sequence number is optimistically supplied to match the one expected to be assigned to the next record, to deduplicate in case of retries.
It integrates nicely with Chat SDK, and you can try this out here: https://ai-chatbot-s2.vercel.app by reloading the page during a larger response!
There are some interesting future possibilities, like building a multiplayer chat experience around shared streams.