Skip to content

Pod Lifecycle

A processor is one running container for one sensor and one template. You never start one yourself: the platform requests a processor when a subscription needs that sensor × template pair and no live processor already serves it, and processors are shared between every subscription that needs them (see Subscriptions & Events). This page follows one processor instance from the moment it is requested until it is stopped, and explains what the platform guarantees along the way.

The two guarantees worth holding onto:

  • An instance is not counted as ready until it has finished starting. Readiness means startup is complete, including restoring any state saved by an earlier run. The platform waits for it before it finishes moving a processor, and prefers a ready copy over one that is still starting.
  • An instance is asked to wind down before it is stopped. It is told first, stops reporting ready and saves its state, and only then is the process stopped.

The lifecycle at a glance

stateDiagram-v2
    [*] --> Scheduling: processor requested for a subscription
    Scheduling --> Starting: placed on compute
    Starting --> Running: process up
    Running --> Ready: startup and state restore finished
    Ready --> Draining: no longer needed / being moved
    Running --> Draining: no longer needed / being moved
    Draining --> Terminated: state saved, stopped
    Terminated --> [*]

    Scheduling: Scheduling — may provision new compute
    Starting: Starting — image download + startup
    Running: Running — process is up
    Ready: Ready — finished starting
    Draining: Draining — stop reporting ready, save state

When an instance starts and stops

The orchestrator polls the platform's list of wanted processors at a regular interval and reconciles what is running against it on every poll.

  • Start. When a subscription is provisioned and a sensor × template pair has no live processor, the platform records a new one. The orchestrator starts it on its next poll. How often it polls is deployment configuration, so no start time is promised.
  • Stop because nothing uses it. When the last subscription using a processor is released, the processor is removed from the list and the orchestrator stops its instance on its next poll.
  • Stop because its template is no longer live. Only templates with an approved, active version run. If a template no longer has one, for example because its live version was revoked, the orchestrator stops that template's running instances on its next poll and marks them terminated. The hourly subscription re-check then updates each affected subscription's coverage.

Scheduling

When the orchestrator starts an instance, it first finds somewhere to run it. If existing compute has room, the instance is placed there. If nothing fits, fresh compute is provisioned automatically and the instance waits for it to come online.

Why a first start can take longer

Starting a processor on fresh compute may mean provisioning a new machine and downloading the processor's image before anything can start. Compute that is already running, and already holds the image, skips both steps.

Starting

Once placed, the instance's container starts up. For a processor built on the Dory SDK, this is where it does its own initialization: running its startup code, connecting its event publisher and, if it has state saved by an earlier run (for example before a restart or a move), restoring that saved state.

During this stage the instance is up but not yet ready. The SDK reports ready only after startup and state restore have finished.

Ready

The platform repeatedly asks each instance a simple question: have you finished starting? The answer is what the platform relies on whenever it has to choose between copies of a processor:

  • When a processor is moved, the platform removes the original only after the replacement reports ready.
  • If the platform ever finds more than one copy of the same processor, it keeps the ready one and removes the rest.

An instance being ready is not a subscription being ready

A subscription's ready control message means its delivery queue is bound to the processors serving it. It does not wait for a newly started instance: events from that processor begin once its instance is up and publishing, which can be after ready. Events from a processor that was already running can arrive before ready.

Running and staying healthy

While an instance runs, the platform keeps checking that it is still alive and responsive, and records the result. If an instance stops responding, its container is restarted.

If an instance fails outright, the orchestrator removes it and starts a fresh one on the next poll, as long as the processor is still wanted. None of this needs anything from you. And if a processor ends up failed or terminated anyway, the hourly subscription re-check moves every subscription it served onto a live processor.

Draining: graceful shutdown

An instance is taken down when it is no longer needed, as described above, or when the platform is moving it (for example, off a machine that is being retired, or onto fewer machines to use capacity efficiently). Whatever the reason, shutdown follows the same order:

  1. The instance is told it is stopping before it is actually stopped.
  2. It stops reporting ready and saves its state.
  3. Only after it has had that chance to wind down is the process stopped.

The platform allows a grace window for this wind-down to complete. When an instance is being moved rather than simply removed, its replacement is brought up and confirmed ready before the platform removes the old one. That state-preserving move is covered in State Migration & Node Drain.

Graceful shutdown depends on the SDK

Readiness reporting and graceful, state-saving shutdown are provided by the Dory Processor SDK. Processors built on the SDK get them automatically. See the Processor SDK docs for how a processor takes part in its own lifecycle.

What this means for you

  • Processors follow subscriptions. An instance starts when a subscription needs its sensor × template pair, and stops when no subscription uses it or its template has no live version.
  • First starts on fresh compute are slower. New compute and an image download can come first; no start time is promised.
  • Readiness makes moves safe. The platform removes a moved processor's original only after its replacement has finished starting.
  • Failures self-heal. An unresponsive instance is restarted, a failed one is replaced, and the hourly re-check moves subscriptions off a processor that is gone.
  • Shutdowns are clean. An instance is always told first and given the chance to save its state before it stops.