It was so delightful to build thanks to jazz.tools.

Vercel's Fluid compute makes it run efficient and scale endlessly

How does it work? Each message is created locally and added to the chat. At the same time I "wake up" a Vercel Function which consumes the chat, runs AI and writes directly into Jazz

The client just renders the state from Jazz. No real streaming required.

Source: https://github.com/tobiaslins/jazz-ai-chat

Whats Fluid Compute? When you run Serverless Functions, usually one invocations serves one request. With Fluid, multiple requests can be served by one function invocation, which is super useful when using AI (low CPU usage, only waiting for upstreams)

Whats Jazz? It's a new way to build apps. It's persistence (like a graph database) with permissions and sync built in.

You can access/write Jazz even from backend workers like here from Vercel Functions.