Skip to content

Edge Failover & Failback

The Dory platform is self-healing. It continuously watches the health of every processor and every machine hosting one, and when something goes wrong it recovers automatically — bringing the affected processor back somewhere healthy with no operator involvement. When the original location recovers, the platform can move the processor back home. There is no pager, no manual restart, no rebuild step: recovery is part of how the platform runs.

This matters most for edge-hosted processors — processors running on machines out at the edge (on-site hardware, remote locations) rather than in the cloud. Edge machines are more likely to lose power, lose network, or simply go offline. When that happens, the platform relocates the affected processors to managed cloud capacity so the work keeps going, and returns them to the edge once it's back.

See Architecture for how the platform's control loop is structured, and State Migration for the planned, zero-downtime moves used when a healthy processor needs to relocate.

How the platform knows something is wrong

Recovery starts with detection, and the platform uses two independent signals so a failure can't slip through unnoticed:

  • Machine health. The platform watches whether each edge machine is reporting itself as healthy and reachable. A machine that drops out, stops responding, or reports itself unhealthy is a candidate for recovery.
  • Processor heartbeats. Each processor also checks in on a regular heartbeat. If a processor stops checking in — even while its machine still looks healthy from the outside — the platform treats it as unhealthy. This catches the subtle case where the hardware is fine but the processor itself has silently stopped.

Combining both signals means the platform notices both blunt failures (the whole machine vanishes) and quiet ones (the machine is up but the processor isn't really working). Alongside this, the platform actively polls processors' health so its picture is always current rather than waiting to be told.

Patience before action

The platform deliberately waits out brief blips before acting. A machine has to be unhealthy for a sustained grace period — not just a momentary flicker — before it's considered truly failed, and a heartbeat has to be missing for a meaningful stretch (several consecutive missed check-ins) before it counts as stale. This prevents the platform from overreacting to a one-second network hiccup and needlessly shuffling processors around.

flowchart LR
    A[Edge machine<br/>health signal] --> C{Healthy?}
    B[Processor<br/>heartbeat] --> C
    C -- Yes --> D[Leave in place]
    C -- "No, but only briefly" --> E[Wait out the blip]
    E --> C
    C -- "No, sustained" --> F[Mark failed → recover]

Automatic failover: recovering elsewhere

Once a processor's machine is confirmed failed (or the processor has gone silent), the platform fails it over: it brings the same processor up on healthy managed cloud capacity, so the work resumes without anyone stepping in.

Crucially, the recovered processor comes back with its state. Each processor's accumulated state is persisted in durable, platform-managed storage, so when the replacement starts in the cloud it restores from that store and continues where the original left off — rather than starting from scratch. This is what separates failover from simply "restarting the app": the processor's progress survives the failure of its machine.

The platform is careful not to double-recover. Before creating a replacement, it checks whether one already exists and skips any processor that's already on its way back. So even if a failure is detected across several cycles, each processor is brought back exactly once — never duplicated.

The replacement also remembers where it came from. It's tagged with its original edge home so that, when that edge machine recovers, the platform knows this processor belongs back there.

Automatic failback: returning home

Edge processors usually run at the edge for a reason — proximity to sensors, local hardware, lower latency. So once a failed-over processor's original edge machine comes back online and is confirmed healthy again, the platform fails it back: it moves the processor from the temporary cloud location back to its original edge home, again carrying its state along, and gracefully retires the cloud stand-in.

Failback is just as automatic as failover. No one has to remember that a processor was relocated or manually move it back — the platform tracks which processors are currently displaced and returns each one as soon as its home is ready.

The full self-healing cycle

Put together, failover and failback form a closed loop that keeps edge processors available through machine failures and brings them home afterward — all without manual intervention:

stateDiagram-v2
    [*] --> Healthy
    Healthy: Healthy — running at the edge
    FailedOver: Failed over — running in cloud, state restored

    Healthy --> Detecting: Heartbeat lost / machine unhealthy
    Detecting --> Healthy: Recovers within grace period
    Detecting --> FailedOver: Sustained failure confirmed
    FailedOver --> Healthy: Edge machine recovers → fail back home
  1. Healthy — the processor runs at its edge home and checks in normally.
  2. Detecting — a missed heartbeat or an unhealthy machine raises a flag, but the platform waits out brief blips.
  3. Failed over — a sustained failure triggers automatic recovery onto managed cloud capacity, with the processor's state restored so it picks up where it left off.
  4. Back to healthy — when the original edge machine recovers, the platform automatically fails the processor back home.

Throughout this cycle, the operator's involvement is zero. The platform detects the problem, recovers the processor, and restores normal operation on its own — which is exactly the point: edge failures are routine, and routine failures should heal themselves.

The platform surfaces this activity through its standard metrics — edge machine health, how many processors are currently failed over, and counts of failover and failback events — so you can see recovery happening at a glance. See Metrics for the full list.