Skip to content

System Architecture

This page shows how the whole Dory platform fits together end to end — from the sensors that observe the world, through the processors that turn raw data into events, across the event stream, to the subscriptions that deliver matching events to your application. Around all of it sits a management & identity plane where each organization configures what runs and who can see it.

Two flows run through the platform:

  • Data flow — sensor data becomes events, events flow down the stream, and matching events arrive at your application.
  • Control flow — the Portal and API record what your organization wants (sensors, lenses, access, subscription requests); a worker turns each recorded subscription into running processors and a bound delivery queue and re-checks it every hour; the Orchestrator runs whatever processors are recorded and recovers them.

The planes

Plane What lives here Who works with it
Producer side Sensors, processors, the Orchestrator that runs and recovers them. Processor developers, operators.
Event stream The exchange processors publish to, and one delivery queue per subscription. Mostly invisible — the platform manages it.
Consumer side Subscriptions and the applications that receive their events. App developers / consumers.
Management & identity The Web Portal and REST API, scoped per organization; the provisioning worker and its hourly re-check; and the identity provider that authenticates everyone. Organization admins and every authenticated caller — the worker is not something you call.

Whole-platform architecture

flowchart LR
    subgraph PROD["Producer side"]
        direction LR
        S["Sensors<br/>own, or published by<br/>another organization"] --> P["Processors<br/>one per sensor × template<br/>(Processor SDK)"]
        REC[("Desired processors<br/>(recorded)")] -. "polled" .-> ORCH["Orchestrator"]
        ORCH -. "starts, recovers,<br/>moves, stops" .-> P
    end

    subgraph STREAM["Event stream"]
        direction LR
        EX(["Exchange"]) -->|"one binding per<br/>processor × event type"| Q[("Delivery queue<br/>one per subscription")]
    end

    P -->|"publish events"| EX

    subgraph CONS["Consumer side"]
        APP["Your application<br/>(Application SDK)"]
    end

    Q -->|"events + control messages<br/>(ready · updated · error)"| APP

    subgraph MGMT["Management &amp; identity plane (per organization)"]
        direction LR
        PORTAL["Web Portal"]
        API["REST API"]
        WRK["Provisioning worker<br/>+ hourly re-check"]
        IDP["OAuth 2.0 / OIDC<br/>identity provider"]
    end

    APP -->|"subscribe · status · unsubscribe<br/>(HTTP, app credential)"| API
    API -->|"creates the delivery queue<br/>before answering 202"| Q
    API -->|"hands off the<br/>recorded request"| WRK
    WRK -->|"matches sensors,<br/>records processors"| REC
    WRK -->|"binds per processor,<br/>drops control messages in"| Q
    PORTAL -. "register sensors, define lenses,<br/>issue credentials, list &amp; delete subscriptions" .-> API
    IDP -. "authenticate users &amp; apps" .-> PORTAL
    IDP -. "tokens" .-> APP

Read the figure by who owns what. The REST API owns the recorded request and the delivery queue: it validates a subscribe call, records the request, creates the queue, hands the request to the worker and answers before any processing starts. The worker owns processors and bindings: it decides which sensors and templates serve the request, records the processors that should exist, binds the queue to them, and drops control messages into that same queue. The Orchestrator takes its instructions only from the record of processors — nothing in the management plane calls it. And the delivery queue flows one way, toward your application: it carries events and control messages, while every request your application makes travels over HTTP to the API.

The rest of this page walks the planes in turn.

Producer side — from sensors to events

This is where raw observations become events.

  1. Sensors are registered, located devices (e.g. cameras). Each belongs to an organization and has a type, a position on the map, and an area of view — the area it watches, which is what subscriptions are matched against. A sensor is private to its organization unless it is published.
  2. Processors — programs built with the Processor SDK — interpret sensor data and publish events: typed, located observations such as person, vehicle, or accident. One processor runs per sensor × template, whichever and however many subscriptions need that pair.
  3. The Orchestrator runs each processor and keeps it running. It places processors on compute, scales capacity up and down with demand, moves processors between machines without dropping their work, and recovers them — state intact — when a machine fails. It polls the platform's record of desired processors; how often is a matter of deployment configuration. A recorded processor means a container should be running; a record removed means it should stop. It also stops, within a poll, a processor whose template version has been revoked.

Which processing turns a given sensor's data into events is decided by the lens chosen on a subscription. A lens is a named selection of processor templates — your organization's own, or ones another organization has published. A subscription may use any of its own organization's lenses or any public lens; a public lens can only contain public templates. Two lenses over the same sensors can produce different event streams, and consumers pick a lens by its short alias and never deal with templates directly. Because templates are shared, the platform also avoids redundant work: the same template on the same sensor runs as one processor however many subscriptions — in whichever organizations — need it; each subscription's delivery queue is bound to that processor, and the processor is released when the last of them is deleted.

The reliability promise

The defining property of the producer side is that a processor keeps running without anyone tending it. When a processor must move to another machine, the platform brings up the replacement and transfers the processor's state before stopping the original — so there is no gap and no lost work.

flowchart LR
    OLD["Processor<br/>(current machine)"] -->|1. capture state| MOVE{{Move}}
    MOVE -->|2. start replacement| NEW["Processor<br/>(new machine)<br/>same processor identity"]
    MOVE -->|3. restore state| NEW
    NEW -->|"4. keep publishing —<br/>subscriptions stay bound"| ES(["Event stream"])
    OLD -.->|5. stop only after<br/>replacement is ready| X["·"]

The replacement keeps the processor's identity, so the bindings that deliver its events to subscriptions are untouched by the move. The same promise covers machine failures (the processor comes back automatically) and edge deployments (a processor running on edge hardware can fail over to the cloud and back). The Orchestrator section describes how this works in depth, including state migration and edge failover.

The event stream

Processors publish their results to a shared event stream — a topic exchange. Each event is a JSON object — conceptually a versioned envelope wrapping a payload — published under a key that names the processor, the event type, and the location the template stamped on the event.

Each subscription gets its own durable delivery queue, created before the subscribe call returns. The queue keeps messages for up to an hour and holds at most 10 000 at a time. The worker binds it to each processor serving the subscription, one binding per requested event type — so the queue receives only the requested types, and only from those processors. Events are not filtered by the location they carry: the subscription's regions chose the sensors, and every event their processors publish for the requested types is delivered.

flowchart LR
    PA["Processor A<br/>sensor 1 × template X"] -->|"person · vehicle"| EX(["Exchange"])
    PB["Processor B<br/>sensor 2 × template X"] -->|"person · vehicle"| EX
    EX -->|"A · person<br/>A · vehicle"| Q1[("Queue — subscription 1<br/>person + vehicle in a region<br/>reaching sensor 1 only")]
    EX -->|"A · person<br/>B · person"| Q2[("Queue — subscription 2<br/>person, everywhere")]
    W["Worker"] -. "control messages<br/>straight into the queue" .-> Q1
    W -. "control messages" .-> Q2

A binding names a processor and an event type, never a place. The same processor feeds every queue bound to it, and the platform's own control messages — ready, updated and error — are dropped straight into the same queue. The Application SDK handles those itself and never passes one to your event handler.

You don't manage the stream. You declare what you want (event types), where (regions), and how (a lens) when you create a subscription, and the platform wires up delivery. Delivery is at-least-once, so an application should be ready to see the occasional duplicate — see Subscriptions & Events.

Consumer side — from subscriptions to applications

A subscription is a standing, server-side request defined by three things:

  • What — the event types you want (each event type is a tag used as a filter).
  • Where — one or more regions: geohash cells of one to nine characters (shorter is wider), or * on its own for everywhere you can see. A region selects every sensor whose area of view reaches into it.
  • How — the lens that decides how raw sensor data becomes the events you receive: a lens of your own organization, or a public one.

A subscription is created by your application, with an application credential, through the Application SDK or the REST API, and it belongs to the credential that created it.

How a subscription is provisioned

The API answers before the slow work is done; the worker and the Orchestrator finish it.

sequenceDiagram
    autonumber
    participant App as Your application (SDK)
    participant API as REST API
    participant Q as Delivery queue
    participant W as Worker
    participant R as Platform records
    participant O as Orchestrator
    participant P as Processor

    App->>API: subscribe (event types, regions, lens)
    API->>R: record the request as pending
    API->>Q: create the queue
    API->>W: hand off
    API-->>App: 202 pending — subscription id + queue name
    App->>Q: start consuming
    W->>R: match sensors by area of view and the lens's templates
    W->>R: record one processor per sensor × template (reuse a live one)
    W->>Q: bind the queue to each processor, per event type
    W->>Q: control message ready (coverage report)
    W->>R: mark the subscription active
    O->>R: poll
    O->>P: start the container
    P->>Q: events, via the exchange
    Q-->>App: events — possibly before ready
    Note over App,Q: The SDK handles control messages itself — your handler only ever sees events.
  1. The API validates the request — event types, lens, region bounds — records it, creates the delivery queue, hands it off to the worker, and answers accepted (HTTP 202) with the subscription id, the queue name and status pending. A refusal here creates nothing — except in the rare case where the delivery queue could be neither created nor cleaned up, which leaves a subscription in error (see Statuses).
  2. Your SDK starts consuming the queue at once.
  3. The worker matches sensors by area of view — your organization's own and any published ones, while active and not decommissioned — and the lens's templates that have an approved, active version, choosing per sensor the smallest set of templates that covers every requested event type.
  4. It records one processor per sensor × template, reusing the one already recorded for that pair unless it has failed or been terminated.
  5. It binds the queue to each processor, per requested event type, drops a ready control message carrying the coverage report into the queue, and marks the subscription active.
  6. The Orchestrator picks up newly recorded processors within a poll and starts them. Events may reach your queue before ready does.

The coverage report says, for each requested event type, whether it is covered and if not why, plus four counts: sensors matched, sensors served, processors started, processors reused. It never names a sensor or a processor. The same facts can be pulled from the API at any time by reading the subscription.

Nothing matched is not an error

A subscription with no matching sensor, or whose lens produces none of the requested event types, still becomes active — with a coverage report that marks every event type uncovered and says why. The hourly re-check fills it in when the world changes.

If the hand-off to the worker is lost, the subscribe call still answers 202; a later hourly run finds the subscription still pending and provisions it.

Kept current every hour

Once an hour the same matching runs again for every active subscription, and only the difference is applied: bindings added or dropped, processors started or released. That catches:

  • a sensor installed, published, unpublished, moved or decommissioned;
  • a lens made private or deleted — the subscription stays active with an empty coverage report that says the lens is not accessible; if the lens becomes accessible again, the next re-check restores everything;
  • a template attached to or detached from the lens, or a template version revoked (revocation also makes the Orchestrator stop the container within a poll);
  • a processor that failed or was terminated — the subscription is re-attached on a live one;
  • a broker restored from backup — the delivery queue is re-created and its bindings re-bound from the platform's record;
  • a lost hand-off — a subscription still pending, or one whose deletion never finished, is re-driven.

An updated control message is sent only when the coverage actually changes.

Statuses

stateDiagram-v2
    [*] --> pending : subscribe accepted (202)
    [*] --> error : created before the current delivery system
    pending --> active : worker finished — ready message with the coverage report
    pending --> error : set-up failed — an error code says why
    active --> active : hourly re-check — updated message when coverage changes
    pending --> deleting : unsubscribe (202)
    active --> deleting : unsubscribe (202)
    error --> deleting : unsubscribe (202)
    deleting --> [*] : queue removed at once, processors released by the worker
  • pending — accepted; the worker has not finished yet.
  • active — provisioned. Coverage may be empty; that is not an error.
  • deleting — deletion accepted; the delivery queue is normally already gone.
  • error — the subscription could not be set up, and it carries a code saying why: the message broker refused a request (BROKER_REFUSED); set-up kept failing on temporary problems until five runs were used up (PROVISIONING_FAILED); the delivery queue could be neither created nor cleaned up (QUEUE_ORPHANED); or the subscription was created before the current delivery system (LEGACY_RESUBSCRIBE). An active subscription never turns into error. You are shown the code and a plain sentence, never the underlying diagnostic, and the first two are also announced by an error control message in the queue. Whatever the code, the remedy is the same: delete it and subscribe again. When a subscription is in error has the full table.

Deleting

Deleting a subscription is accepted with HTTP 202. The delivery queue is removed inside the request, so delivery stops at once and the SDK's consumer is cancelled; if the broker cannot be reached at that moment, the worker removes the queue as its first step. The worker releases the subscription's processors shortly after; a processor no other subscription uses is stopped by the Orchestrator within a poll.

Your application receives events through the Application SDK, which handles authentication, the connection, control messages and event delivery so you can focus on what to do with each event. The REST API is how you create and manage subscriptions programmatically — the control surface the SDK and Portal use — not an event-delivery channel. Managing subscriptions covers the SDK side.

Management & identity plane

Everything above is configured through the management plane, and everything is scoped to an organization — the tenant and isolation boundary that owns members, application credentials, sensors, tags, lenses, and subscriptions.

Surface What you do with it
Web Portal Manage members and access, register sensors, define lenses, issue application credentials, compose a subscription request and get the SDK snippet that creates it, and see or delete each credential's subscriptions. See Portal overview.
REST API The same capabilities, programmatically — the interface behind the Portal and the SDKs — including creating a subscription (accepted asynchronously), reading its status and coverage, and listing and deleting subscriptions. See API overview.
Provisioning worker & re-check Not a surface you call. It turns a recorded subscription into processors and bindings, tells you through the delivery queue, and re-checks every hour.
Identity provider An OAuth 2.0 / OIDC identity provider authenticates both people (Portal users) and applications (machine-to-machine credentials), and issues the tokens every API call carries. See Identity & Access.

Everything is private by default. An organization can publish a sensor, a template, or a lens; other organizations' subscriptions may then draw on it, but a published sensor's connection details and metadata are never exposed, a subscription is told how many sensors it reached and never which, and subscriptions themselves are never visible outside the organization that owns them.

How the planes connect

Connection Direction What moves
Sensors → Processors data Raw observations to interpret.
Processors → Exchange data Published events.
Exchange → Delivery queue → Application data Events of the requested types from the processors serving the subscription, plus control messages (ready · updated · error).
Application → REST API control Subscribe, read status, unsubscribe — over HTTP, with an application credential.
Portal → REST API control Sensors, lenses, credentials; listing and deleting subscriptions.
REST API → Worker control The hand-off of a recorded request, sent just before the 202 is answered.
Hourly schedule → Worker control The re-check of every active subscription, and the re-drive of anything left unfinished.
Worker → record of desired processors → Orchestrator control The worker records which processors should exist (one per sensor × template); the Orchestrator polls that record and starts or stops containers to match.
Worker → Delivery queue control Bindings, and the control messages.
Orchestrator → Processors control Start, placement, moves, recovery, stop.
Identity provider → Portal & Applications control Authentication and access tokens.

Where to go next