Skip to content

Observability

Every Dory processor emits the three standard observability signals — logs, metrics, and traces — with no setup. The platform collects them centrally, so you can see what a processor is doing in production without wiring up any infrastructure yourself.

The three signals

Signal What you get Your part
Logs Structured JSON logs to stdout, collected by the platform. Nothing. Use the SDK logger for your own messages.
Metrics Standard dory_* metrics (startup, state, restarts, health) exposed for the platform to scrape. Nothing. Add a custom metric in one line if you want.
Traces End-to-end distributed traces across the whole event pipeline. Mostly automatic — add a span around custom hot-loop work.

Logs

The SDK writes structured JSON logs to stdout; the platform collects them. Use the SDK logger for your own messages so they share the same structured format:

from dory.logging import get_logger

log = get_logger(__name__)
log.info("processed batch", extra={"count": 100})

Metrics

Every processor automatically produces a standard set of dory_* metrics and exposes them for the platform to scrape — no setup required. They cover the processor's lifecycle, state migration, and health:

Metric Tells you
dory_startup_duration_seconds How long the processor took to start.
dory_shutdown_duration_seconds How long it took to shut down gracefully.
dory_restart_count How many times the processor has restarted.
dory_state_save_duration_seconds Time to capture state (e.g. before a migration).
dory_state_load_duration_seconds Time to restore state after a move.
dory_state_size_bytes Size of the most recent saved state.
dory_health_check_failures_total Count of failed health checks.
dory_golden_image_resets_total Times the processor reset to a clean baseline.
dory_processor_info Identity labels (processor id, version, pod).

Custom metrics

Record your own application metrics alongside the standard ones — they appear on the same endpoint:

from dory.metrics import MetricsCollector

metrics = MetricsCollector()
metrics.set_gauge("queue_depth", 42, labels={"queue": "ingest"})
metrics.inc_counter("messages_processed_total", labels={"topic": "events"})

Use set_gauge for values that move up and down (queue depth, temperature) and inc_counter for things that only accumulate (messages processed, errors). See the API Reference for full signatures.

Traces

Tracing shows the path of work through your processor — and across processors — as a single connected trace. On the platform it is on automatically: every processor exports its traces to the platform's tracing backend, where you can view them in the dashboard. When you run a processor locally without that backend, traces simply print to the console.

What's automatic

  • Request handlers are traced for you. Any handler method on your processor is automatically wrapped in a span — request/response processors get tracing for free.
  • Traces follow your events. When a processor publishes an event, the SDK attaches the current trace context to the message. The downstream consumer continues the same trace, so you get one unbroken, end-to-end view across every processor an event touches.

What you add

If your processor does its work in a custom loop (for example, a camera frame loop) rather than in request handlers, wrap each unit of work in a span so it shows up in the trace:

with self.span("inference", {"model": "yolov8n"}):
    detections = await self.run_inference(frame)

self.span(...) is always safe to call — it does nothing when tracing isn't active, so you never need to guard it. (This is exactly how the bundled camera-detector example is instrumented.)

Trace flow

flowchart LR
    SRC[Camera / data source] --> PROC

    subgraph PROC[Processor]
        WORK["Processing work<br/>• request handlers — traced automatically<br/>• custom loops — add a span"]
    end

    PROC -- "events carry trace context" --> BROKER[Event broker]
    BROKER -- "same trace continues" --> CONSUMER[Downstream processor / consumer]

    PROC -. "traces" .-> COLLECTOR[Telemetry collector]
    CONSUMER -. "traces" .-> COLLECTOR
    COLLECTOR --> BACKEND[Tracing backend]
    BACKEND --> UI[Dashboard]

Solid arrows are the event flow: each event carries its trace context to the next processor, so a trace started upstream continues unbroken downstream. Dotted arrows are the traces themselves, which every processor exports to the platform's collector and tracing backend for viewing in the dashboard.

Health & readiness

The SDK also serves the liveness and readiness signals the platform uses to health-check your processor and decide when to route work to it. You don't manage this: the SDK marks the processor ready automatically once startup — and any state restore — is complete, so the platform never sends it work before it can handle it.

See API Reference for the methods you override and call.