Using stream delegation you can have AI agents producing tokens in parallel with sequential consumption without blocking output. This lets you break down monolithic prompts into recursive prompts and helps with agent meta-prompting.
Comments
uxabhishek•5mo ago
The "relay pattern" and use of yield* for delegating between async iterables is elegant.
How does the overhead of managing the 50 agents affect overall resource utilization? Also, how does the system handle errors in one of the child streams? Does it cascade and halt the entire process, or is there a mechanism for retrying or gracefully degrading?
Looking forward to seeing how this develops!
etler•5mo ago
Very good questions! I still need to flesh out the patterns. While streams of streams are common in functional programming environments, I simply haven't seen them in class based streaming patterns anywhere so they're things to figure out.
Of course, more streams means more resource utilization, there's not getting around that, but that's the cost of parallelism. The use of `yield*` should keep the overhead to a bare minimum. Since the streams are being left alone and aren't consumed until needed, that should preserve some of the back-pressure behavior, although I do need to look into that more deeply.
How the system handles errors probably doesn't have a single solution that works for all frameworks, so I think it should be up to the specific requirements of each use case, but there's also definitely more work to do to explore the options and the patterns.
These are all things I definitely want to hear ideas for as well!
The next thing I'm exploring is applying these patterns to web rendering which will be a real stress test for how they can be used.
uxabhishek•5mo ago
How does the overhead of managing the 50 agents affect overall resource utilization? Also, how does the system handle errors in one of the child streams? Does it cascade and halt the entire process, or is there a mechanism for retrying or gracefully degrading?
Looking forward to seeing how this develops!
etler•5mo ago
Of course, more streams means more resource utilization, there's not getting around that, but that's the cost of parallelism. The use of `yield*` should keep the overhead to a bare minimum. Since the streams are being left alone and aren't consumed until needed, that should preserve some of the back-pressure behavior, although I do need to look into that more deeply.
How the system handles errors probably doesn't have a single solution that works for all frameworks, so I think it should be up to the specific requirements of each use case, but there's also definitely more work to do to explore the options and the patterns.
These are all things I definitely want to hear ideas for as well!
The next thing I'm exploring is applying these patterns to web rendering which will be a real stress test for how they can be used.