Resilience¶
Real dependencies fail — a database hiccups, an API returns 503, the event broker blinks. The Dory Processor SDK gives your processor a few building blocks to ride out those failures instead of crashing: retries that absorb transient faults, circuit breakers that stop hammering a dependency that's genuinely down, and output buffering so events aren't lost during a broker blip.
Retry with backoff¶
Transient failures — a dropped connection, a momentary 429 — usually clear up on their own. Wrap the call in retry_with_backoff and the SDK re-attempts it with an exponentially growing delay between tries. It works on both async and sync functions:
from dory.resilience import retry_with_backoff
@retry_with_backoff(max_attempts=3, retryable_exceptions=(ConnectionError,))
async def call_api():
return await client.get("/data")
| Option | Default | What it does |
|---|---|---|
max_attempts |
3 |
Total tries, including the first. |
initial_delay |
1.0 |
Base delay (seconds) before the first retry. |
max_delay |
30.0 |
Cap on any single delay. |
multiplier |
2.0 |
How fast the delay grows each attempt. |
jitter |
True |
Randomizes delays so many processors don't retry in lockstep. |
retryable_exceptions |
(Exception,) |
Which exceptions trigger a retry. |
Set retryable_exceptions to the faults that are actually worth retrying (e.g. connection errors), so genuine bugs fail fast instead of retrying pointlessly.
How backoff behaves¶
flowchart TD
A[Call dependency] --> B{Succeeded?}
B -- yes --> C[Return result]
B -- no --> D{Retryable<br/>& attempts left?}
D -- no --> E[Raise the error]
D -- yes --> F[Wait: delay grows<br/>each attempt, with jitter]
F --> A
Each wait is roughly initial_delay × multiplier^attempt, capped at max_delay, with a little random jitter added so a fleet of processors doesn't all wake up at the same instant (the "thundering herd").
Circuit breakers¶
Retrying helps with brief faults. But if a dependency is down, retrying just piles on load and slows everyone down. A circuit breaker watches for repeated failures and, once they cross a threshold, "opens" — failing fast for a while instead of calling the broken dependency at all. After a cooldown it cautiously tests whether the dependency has recovered.
Wrap a call with call():
from dory.resilience import CircuitBreaker, CircuitOpenError
breaker = CircuitBreaker("payments", failure_threshold=3, timeout_seconds=30.0)
try:
result = await breaker.call(charge_card, amount)
except CircuitOpenError:
result = None # breaker is open — fail fast, skip the call
| Option | Default | What it does |
|---|---|---|
failure_threshold |
5 |
Consecutive failures before the breaker opens. |
success_threshold |
2 |
Successful trial calls needed to close it again. |
timeout_seconds |
60.0 |
How long to stay open before testing recovery. |
Breaker states¶
stateDiagram-v2
[*] --> Closed
Closed --> Open: failures reach threshold
Open --> HalfOpen: cooldown elapses
HalfOpen --> Closed: trial calls succeed
HalfOpen --> Open: a trial call fails
- Closed — normal; calls pass straight through.
- Open — failing fast;
call()raisesCircuitOpenErrorwithout touching the dependency. - Half-open — letting a few trial calls through to see if the dependency is back. Success closes the breaker; a failure re-opens it.
Output buffering¶
When your processor publishes events and the broker is briefly unreachable, the SDK's output path doesn't drop those events on the floor. It buffers them in memory and flushes them once the connection comes back — and it uses its own circuit breaker so a struggling broker doesn't stall your processing loop. This happens on the platform's publish path without extra code from you; just publish your results as usual.
Buffering is a cushion, not a queue
In-memory buffering covers short outages. It is bounded and lives in the processor's memory, so don't rely on it for long broker downtime or for durability across a restart — that's what your event broker and durable stores are for.
Putting it together¶
These primitives compose. A defensive call to a flaky dependency typically retries transient errors and lets a breaker shed load when the dependency is truly down:
@retry_with_backoff(max_attempts=3, retryable_exceptions=(ConnectionError,))
async def query():
return await breaker.call(db.fetch, sql)
Related¶
- State Management — preserving work across moves and restarts.
- Observability — seeing failures and recoveries in metrics and traces.
- Configuration — settings that tune these behaviors.