Observability¶
Every Dory processor emits the three standard observability signals — logs, metrics, and traces — with no setup. The platform collects them centrally, so you can see what a processor is doing in production without wiring up any infrastructure yourself.
The three signals¶
| Signal | What you get | Your part |
|---|---|---|
| Logs | Structured JSON logs to stdout, collected by the platform. | Nothing. Use the SDK logger for your own messages. |
| Metrics | Standard dory_* metrics (startup, state, restarts, health) exposed for the platform to scrape. |
Nothing. Add a custom metric in one line if you want. |
| Traces | End-to-end distributed traces across the whole event pipeline. | Mostly automatic — add a span around custom hot-loop work. |
Logs¶
The SDK writes structured JSON logs to stdout; the platform collects them. Use the SDK logger for your own messages so they share the same structured format:
from dory.logging import get_logger
log = get_logger(__name__)
log.info("processed batch", extra={"count": 100})
Metrics¶
Every processor automatically produces a standard set of dory_* metrics and exposes them for the platform to scrape — no setup required. They cover the processor's lifecycle, state migration, and health:
| Metric | Tells you |
|---|---|
dory_startup_duration_seconds |
How long the processor took to start. |
dory_shutdown_duration_seconds |
How long it took to shut down gracefully. |
dory_restart_count |
How many times the processor has restarted. |
dory_state_save_duration_seconds |
Time to capture state (e.g. before a migration). |
dory_state_load_duration_seconds |
Time to restore state after a move. |
dory_state_size_bytes |
Size of the most recent saved state. |
dory_health_check_failures_total |
Count of failed health checks. |
dory_golden_image_resets_total |
Times the processor reset to a clean baseline. |
dory_processor_info |
Identity labels (processor id, version, pod). |
Custom metrics¶
Record your own application metrics alongside the standard ones — they appear on the same endpoint:
from dory.metrics import MetricsCollector
metrics = MetricsCollector()
metrics.set_gauge("queue_depth", 42, labels={"queue": "ingest"})
metrics.inc_counter("messages_processed_total", labels={"topic": "events"})
Use set_gauge for values that move up and down (queue depth, temperature) and inc_counter for things that only accumulate (messages processed, errors). See the API Reference for full signatures.
Traces¶
Tracing shows the path of work through your processor — and across processors — as a single connected trace. On the platform it is on automatically: every processor exports its traces to the platform's tracing backend, where you can view them in the dashboard. When you run a processor locally without that backend, traces simply print to the console.
What's automatic¶
- Request handlers are traced for you. Any handler method on your processor is automatically wrapped in a span — request/response processors get tracing for free.
- Traces follow your events. When a processor publishes an event, the SDK attaches the current trace context to the message. The downstream consumer continues the same trace, so you get one unbroken, end-to-end view across every processor an event touches.
What you add¶
If your processor does its work in a custom loop (for example, a camera frame loop) rather than in request handlers, wrap each unit of work in a span so it shows up in the trace:
self.span(...) is always safe to call — it does nothing when tracing isn't active, so you never need to guard it. (This is exactly how the bundled camera-detector example is instrumented.)
Trace flow¶
flowchart LR
SRC[Camera / data source] --> PROC
subgraph PROC[Processor]
WORK["Processing work<br/>• request handlers — traced automatically<br/>• custom loops — add a span"]
end
PROC -- "events carry trace context" --> BROKER[Event broker]
BROKER -- "same trace continues" --> CONSUMER[Downstream processor / consumer]
PROC -. "traces" .-> COLLECTOR[Telemetry collector]
CONSUMER -. "traces" .-> COLLECTOR
COLLECTOR --> BACKEND[Tracing backend]
BACKEND --> UI[Dashboard]
Solid arrows are the event flow: each event carries its trace context to the next processor, so a trace started upstream continues unbroken downstream. Dotted arrows are the traces themselves, which every processor exports to the platform's collector and tracing backend for viewing in the dashboard.
Health & readiness¶
The SDK also serves the liveness and readiness signals the platform uses to health-check your processor and decide when to route work to it. You don't manage this: the SDK marks the processor ready automatically once startup — and any state restore — is complete, so the platform never sends it work before it can handle it.
See API Reference for the methods you override and call.