Skip to content

State Migration & Node Drain

When the platform needs to move a running processor to a different machine, it does so without dropping a beat. Your processor keeps serving the whole time, and the state it has built up — counters, in-flight progress, accumulated context — comes along with it. This page explains that guarantee and the situations that trigger a move.

Why a processor moves

A processor doesn't stay on one machine forever. The platform may decide to relocate it for several ordinary, healthy reasons:

  • Cost-driven consolidation — when the cluster is running more machines than it needs, the platform packs processors onto fewer machines so the empty ones can be released.
  • Scale-down or maintenance — when a machine is being retired, drained for maintenance, or reclaimed (for example, a spot instance being taken back), the processors on it must move elsewhere first.
  • A machine going away — capacity changes that require the workload to land somewhere new.

In every one of these cases, moving the processor is routine — and the platform treats "no interruption, no lost state" as a hard requirement, not a best effort.

The zero-downtime guarantee

The core promise is simple: the original keeps running until the replacement is proven ready. The platform never tears down the old copy and hopes the new one comes up. Instead it overlaps them.

The move follows a strict create-before-delete order:

  1. Start a replacement on the new machine while the original keeps serving traffic.
  2. Transfer the processor's state from the original into the replacement, so the replacement picks up exactly where the original left off.
  3. Wait until the replacement is fully ready — not just started, but passing its readiness checks and able to actually do work.
  4. Only then remove the original.

Because steps 1–3 happen while the original is still live, there is a window where both copies exist at once. That overlap is precisely what makes the move seamless: at no point is there zero healthy copies of the processor. Service continues throughout, and no state is lost.

sequenceDiagram
    participant Orch as Orchestrator
    participant Old as Old Pod (Node A)
    participant New as New Pod (Node B)

    Note over Old: Serving traffic
    Orch->>New: Start replacement (Old still serving)
    Orch->>Old: Capture current state
    Old-->>New: Transfer state
    New->>New: Restore transferred state
    Note over Old,New: Both running — overlap = zero-downtime window
    New-->>Orch: Report Ready (readiness checks pass)
    Orch->>Orch: Switch over to replacement
    Orch->>Old: Stop original
    Note over New: Serving traffic, state intact

What "ready" really means

The platform deliberately waits for the replacement to be ready to serve, not merely started. A container that has launched but not yet passed its readiness checks cannot reliably handle work. If the platform removed the original the moment the replacement started, a replacement that then failed to come up healthy would leave the processor with no working copy. By holding the original open until the replacement passes its readiness checks, the platform guarantees a genuine handover.

If the replacement doesn't come up

If the replacement fails to become ready in time, the move is safely abandoned: the platform cleans up the failed replacement and leaves the original running, untouched. A failed migration attempt has zero impact on the live processor — the worst case is that the move simply doesn't happen this time, and the original keeps serving as before.

State comes with the processor

A move is only useful if the processor doesn't forget what it was doing. As part of the overlap window, the platform captures the processor's accumulated state from the original and restores it into the replacement before completing the switch. The replacement resumes from that state rather than starting cold.

State transfer is treated as best-effort and non-blocking: if the state can't be captured or restored for some reason, the move still completes and the replacement starts fresh rather than the processor being left stranded with no running copy. The platform always favors keeping the processor available. For a deep dive on how processors capture and restore their own state, see the SDK's state guide.

Node drain: moving everything off a machine

When a whole machine is being taken out of service, every managed processor on it has to move. The platform handles this proactively: rather than waiting for the machine to be forcibly emptied (which would interrupt the processors), it gets ahead of the drain.

For each processor on the departing machine, the platform:

  1. Captures the processor's current state before anything is torn down.
  2. Starts a replacement on a healthy machine and restores the captured state into it.
  3. Waits for the replacement to be ready.
  4. Lets the original be removed as the machine drains.

This is the same create-before-delete guarantee applied across every processor on the node, so draining a machine never costs a processor its availability or its state.

When there's nowhere to put it yet

If no existing machine has room for the replacement, the platform doesn't give up or cram it onto an overloaded machine. Instead it requests new capacity and waits — with a longer, more patient timeout — for a fresh machine to come online before placing the replacement there. This avoids the trap of forcing a processor onto a machine that can't actually hold it (which would just make it fail and bounce around). Capacity is added on demand and released again automatically once it's no longer needed.

sequenceDiagram
    participant Orch as Orchestrator
    participant Old as Processor (departing machine)
    participant New as Replacement (healthy machine)

    Note over Old: Machine marked for drain
    Orch->>Old: Capture state before drain proceeds
    Orch->>New: Start replacement on a node with room
    Note over New: If no room exists, request new capacity and wait
    Old-->>New: Restore captured state
    New-->>Orch: Report Ready
    Note over Old: Original removed as machine drains
    Note over New: Resumes with state intact

Consolidation: doing more with fewer machines

Consolidation is the cost-saving side of migration. When the cluster is stable — nothing being added or removed, nothing mid-startup — the platform looks for opportunities to pack processors onto fewer machines so emptied machines can be released and stop costing money.

Each processor that needs to move during consolidation moves under the exact same zero-downtime guarantee described above: a replacement is started and made ready before the original is removed. Several such moves can run at once, but the platform deliberately paces itself:

  • It only consolidates when the system is settled, never in the middle of other changes.
  • It honors a cooldown between consolidation rounds so it isn't churning constantly.
  • If a consolidation round runs into trouble, it backs off for a longer cooldown before trying again, rather than retrying the same doomed move every cycle.

The result is a cluster that quietly right-sizes itself — releasing idle capacity to save cost — without ever interrupting the processors it's rearranging. See Architecture for where consolidation sits in the platform's overall control loop.