Skip to content

Resilience

Real dependencies fail — a database hiccups, an API returns 503, the event broker blinks. The Dory Processor SDK gives your processor a few building blocks to ride out those failures instead of crashing: retries that absorb transient faults, circuit breakers that stop hammering a dependency that's genuinely down, and output buffering so events aren't lost during a broker blip.

Retry with backoff

Transient failures — a dropped connection, a momentary 429 — usually clear up on their own. Wrap the call in retry_with_backoff and the SDK re-attempts it with an exponentially growing delay between tries. It works on both async and sync functions:

from dory.resilience import retry_with_backoff

@retry_with_backoff(max_attempts=3, retryable_exceptions=(ConnectionError,))
async def call_api():
    return await client.get("/data")
Option Default What it does
max_attempts 3 Total tries, including the first.
initial_delay 1.0 Base delay (seconds) before the first retry.
max_delay 30.0 Cap on any single delay.
multiplier 2.0 How fast the delay grows each attempt.
jitter True Randomizes delays so many processors don't retry in lockstep.
retryable_exceptions (Exception,) Which exceptions trigger a retry.

Set retryable_exceptions to the faults that are actually worth retrying (e.g. connection errors), so genuine bugs fail fast instead of retrying pointlessly.

How backoff behaves

flowchart TD
    A[Call dependency] --> B{Succeeded?}
    B -- yes --> C[Return result]
    B -- no --> D{Retryable<br/>& attempts left?}
    D -- no --> E[Raise the error]
    D -- yes --> F[Wait: delay grows<br/>each attempt, with jitter]
    F --> A

Each wait is roughly initial_delay × multiplier^attempt, capped at max_delay, with a little random jitter added so a fleet of processors doesn't all wake up at the same instant (the "thundering herd").

Circuit breakers

Retrying helps with brief faults. But if a dependency is down, retrying just piles on load and slows everyone down. A circuit breaker watches for repeated failures and, once they cross a threshold, "opens" — failing fast for a while instead of calling the broken dependency at all. After a cooldown it cautiously tests whether the dependency has recovered.

Wrap a call with call():

from dory.resilience import CircuitBreaker, CircuitOpenError

breaker = CircuitBreaker("payments", failure_threshold=3, timeout_seconds=30.0)

try:
    result = await breaker.call(charge_card, amount)
except CircuitOpenError:
    result = None  # breaker is open — fail fast, skip the call
Option Default What it does
failure_threshold 5 Consecutive failures before the breaker opens.
success_threshold 2 Successful trial calls needed to close it again.
timeout_seconds 60.0 How long to stay open before testing recovery.

Breaker states

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failures reach threshold
    Open --> HalfOpen: cooldown elapses
    HalfOpen --> Closed: trial calls succeed
    HalfOpen --> Open: a trial call fails
  • Closed — normal; calls pass straight through.
  • Open — failing fast; call() raises CircuitOpenError without touching the dependency.
  • Half-open — letting a few trial calls through to see if the dependency is back. Success closes the breaker; a failure re-opens it.

Output buffering

When your processor publishes events and the broker is briefly unreachable, the SDK's output path doesn't drop those events on the floor. It buffers them in memory and flushes them once the connection comes back — and it uses its own circuit breaker so a struggling broker doesn't stall your processing loop. This happens on the platform's publish path without extra code from you; just publish your results as usual.

Buffering is a cushion, not a queue

In-memory buffering covers short outages. It is bounded and lives in the processor's memory, so don't rely on it for long broker downtime or for durability across a restart — that's what your event broker and durable stores are for.

Putting it together

These primitives compose. A defensive call to a flaky dependency typically retries transient errors and lets a breaker shed load when the dependency is truly down:

@retry_with_backoff(max_attempts=3, retryable_exceptions=(ConnectionError,))
async def query():
    return await breaker.call(db.fetch, sql)