I'm excited to share Toller, a lightweight Python library I've been working on to make asynchronous calls more robust and easier to manage. I built this after a painful incident with one of my R2R (https://github.com/SciPhi-AI/R2R) clients where Azure OpenAI went down unexpectedly. While we were technically propagating errors correctly, we lacked clean, accessible error patterns that would allow the client to implement proper mitigation strategies. They were fully dependent on our infrastructure to handle the outage, with no way to gracefully degrade or implement custom fallbacks.
This highlighted a critical gap in the Python async ecosystem: while libraries exist for basic async operations, there's nothing that combines resilience patterns in a comprehensive, accessible way. More and more, applications today depend on many external async calls:
- How do you consistently retry transient errors (like 503s from Azure's API) with proper backoff and jitter? - How do you stop hammering a service that's clearly down (circuit breaking) while automatically testing recovery? - How do you manage API rate limits across concurrent tasks without complex manual logic? - And importantly, how do you provide standardized, actionable error information to both your server and any clients depending on your API?
Toller aims to solve this with a simple `@toller.task` decorator that wraps your `async` functions and adds: Rate Limiting:Async-safe token bucket-based limiter with adaptive throttling that responds to service feedback Retries: Customizable strategies (max attempts, delay, exponential backoff, jitter) on specific exceptions Circuit Breaker: Standard CLOSED/OPEN/HALF_OPEN states with configurable thresholds and recovery testing Transparent Error Handling: Rich context in exceptions that makes it clear what happened and what to do about it
Unlike alternatives like Tenacity (focused on retries but lacking circuit breaking) or raw asyncio primitives (which require significant boilerplate), Toller provides an integrated solution specifically designed for modern async workflows, particularly those involving LLMs and other AI services where reliability is critical.
A clear exception hierarchy (`TransientError`, `FatalError`, `MaxRetriesExceeded`, `OpenCircuitError`) aims to make error handling predictable. The goal is to take the "unruly" nature of distributed async calls and "lure" them into well-managed flows (hence the name, inspired by the Nova Scotia Duck Tolling Retriever!). It's async-native, lightweight, and I've tried to make it as straightforward as possible.
I'd love to get your feedback, suggestions, or hear about any use cases you might have for it! Thanks!