SPEC-101 Stage 0 v0.3 — loss budget, recovery invariants, rollback
Current authority notice: This document is the recovery and loss-budget companion to the Marconi architecture. The consolidated current source-of-authority document is Marconi. Use this document for recovery invariants; use the Marconi SOR for current client-facing signal authority, gatekeeper, ACK, trace, and store-role requirements.
Gate. Required by SPEC-101 v0.3 §8 before the Stage 5 hot-path cutover lands. Defines what we accept as loss in each failure mode, what state every failure mode leaves the system in, and how to roll back each fine-grain feature flag.
v0.3 scope. The parent SPEC-101 v0.3 locks the architecture as: sender shim → Marconi → receiver shim on the hot path; Marconi → in-memory audit queue (parent §3.6) → Redis Stream → PG archiver on the audit fan-out. Redis is not on the signal-delivery flow. When Redis is down longer than the audit queue holds, audit-queue overwrite is the operator-visible loss event. Durability-recovery algorithms (“get closer to 100%”) are out of scope; revisit if real failures motivate it.
The API returned 200 to the caller (Marconi accepted ownership).
The recipient never received the in-process WS push (or for offline recipients, the signal never appeared in prism_signals_pending).
There is no durable record in Redis Stream OR PG audit (both downstream tiers missed it).
A signal that delivered to the recipient but is missing from PG audit is degraded durability, not lost (recoverable from Redis Stream). A signal that did NOT deliver but is durably persisted with outcome=no_subscriber or queued_offline is delivered as expected — that’s the contract for offline recipients.Loss is fatal; degraded durability is operator-visible and recoverable from upstream tier (Redis Stream > PG).
Audit queue is flushed to Redis Stream during graceful shutdown
Backend hard kill (OOM, SIGKILL, host crash)
Up to audit-queue depth at kill time, that hadn’t been written to Redis yet
Network-grade durability; bounded by Redis-writer lag
Redis down (transient)
0 signals
Audit queue buffers; writer resumes from last-acknowledged offset
Redis down longer than audit queue holds
Signals overwritten in the audit queue before Redis ingested them = lost. Operator-visible via marconi_audit_queue_overwrite_total > 0 (paging incident)
Sized correctly, audit queue absorbs realistic Redis outages; longer outages are the operator alert path
PG down
0 signals
Redis Stream is the upstream cache; archiver resumes from consumer-group checkpoint
Network partition between backend and recipient
0 signals lost; signals queue offline for recipient
Existing pending-signal semantics, preserved
Recipient WS disconnect mid-delivery
0 signals lost; entry stays replay/drain-eligible until ACK evidence
Operator framing (Frank, 2026-05-10):“If Redis is down, messages are delivered but never recorded at this stage of the build.”That is the whole loss model. Delivery is the contract; durability is best-effort. The two failure modes that produce “delivered but not recorded” outcomes:
Backend hard-kill — in-flight audit-queue entries that hadn’t yet been written to Redis are lost. Bounded by Redis-writer lag (typically sub-second under normal load).
Redis offline longer than the audit queue holds — once the queue fills (max_age_seconds), oldest entries overwrite and are lost from the durable record.
Both are operator-visible (counters + alerts). Everything else is zero loss. Algorithms to recover the “delivered but not recorded” set are out of scope for v0.3.
For each failure mode, the system state after recovery is deterministic. R1, R2, R3, R5, R6, R8 below are testable invariants. (R4 and R7 in earlier drafts were spool-specific and are removed in v0.2.)
Failure: SIGTERM, container restart, deploy.
During: in-memory tables empty until shims reconnect (~1-2s warmup window per parent §5).
After: routing/registration tables rehydrated from SessionStore (Redis); audit queue empty (warm only when new signals arrive); Redis Stream + PG archiver resume from their checkpoints.
Recoverable signals: all signals successfully written to Redis Stream before SIGTERM are delivered to PG via the normal pipeline.
Lost signals: 0 (audit queue is flushed to Redis Stream during graceful shutdown — last-resort sync barrier).
Failure: OOM kill, SIGKILL, host crash, power loss.
After: same as R1, but in-flight audit-queue entries that hadn’t yet been written to Redis are lost.
Recoverable signals: Redis Stream entries up to last successful append.
Lost signals: bounded by Redis-writer lag at kill time (typically sub-second under normal load).
R3 — Recipient WS disconnect during signal delivery
Failure: shim WS drops between Marconi’s push and the recipient’s ack of the frame.Delivery-evidence model. outcome=pushed records that Marconi attempted delivery on the WS handle; it does NOT constitute final delivery evidence. The signal becomes delivered (final) only when one of:
the shim/model returns an explicit application-level ACK frame, OR
a prism_signal_ack for the trace lands with ack_kind∈{model_acted, surface_observed}, OR
the recipient’s pending-signal drain consumes the entry on a subsequent session.
During: signal is in the audit queue (and downstream in the Redis Stream once written) with outcome=pushed, delivery_state=awaiting_ack (provisional). The pending-signal index keeps the entry replay/drain-eligible until ACK evidence is recorded.After: recipient reconnects, re-registers, and either receives a fresh push for the still-eligible signal OR drains it via prism_signals_pending. Whichever arrives first promotes delivery_state to acked; the entry is then released from the pending-signal index.Lost: 0. A signal that was pushed-without-ack is treated identically to one that was queued-offline — both are replay-eligible until ACK evidence exists.Counter implications: marconi_signals_delivered_total{outcome=pushed} increments at push time; a separate marconi_signals_acked_total{ack_kind} increments on ACK. The gap between these counters is the in-flight-without-ack window and is operator-observable.
Failure: Redis container down, network partition, etc.
During: audit queue buffers; Redis writer fails on every batch with backoff; marconi_redis_writer_lag_seconds grows.
Sized correctly: audit queue max_age_seconds exceeds expected Redis recovery time; the writer drains the backlog when Redis returns.
Outage longer than queue holds: audit-queue overwrites begin; oldest entries are lost; marconi_audit_queue_overwrite_total increments and pages.
After Redis recovery: writer resumes from last-acknowledged offset, drains the backlog.
Lost: 0 if Redis recovers within audit-queue max_age_seconds; otherwise the overwritten entries.
Failure: Postgres container down, schema migration window, etc.
During: archiver reads from Redis Stream succeed but PG batch INSERTs fail; archiver retries with backoff.
After PG recovery: archiver drains; idempotent UPSERT prevents duplicates.
Lost: 0 (Redis Stream is the source of truth at this tier).
R8 — Redis Stream trimmed past archiver checkpoint
Failure: Redis MAXLEN trim is more aggressive than archiver’s catch-up speed (sustained archiver lag > 7d).
Detection: marconi_pg_archiver_lag_seconds > 7d (paging incident).
During: archiver reads return “stream trimmed”; trimmed window is unrecoverable to PG audit.
After: archiver resumes from the new stream head.
Lost: 0 signal delivery; the trimmed window is an audit gap (operator-visible).
Forward: routing/registration lookups served from process-local in-memory cache; SessionStore (Redis) fallback on miss. Cache coherence is maintained by direct write-through hooks (parent §5) — no Redis pubsub.
Rollback: flip flag → all lookups go directly to SessionStore (existing pre-Marconi behavior; the path that’s been live since ADR-27 / SPEC-049).
Data risk: none; the process-local cache is a perf layer atop SessionStore, and SessionStore remains the source of truth on rollback.
Forward: every accepted signal is appended to Marconi’s in-memory audit queue (parent §3.6). Additive — legacy synchronous PG write still happens during Stages 3 and 4.
Rollback: flip flag → audit queue writes stop. Legacy synchronous PG write remains the source of truth (since Stage 5 has not cut over).
Data risk: none during Stages 3–4 (legacy path is the durable record). After Stage 5, MARCONI_AUDIT_QUEUE_WRITE cannot be safely rolled back without also rolling back MARCONI_HOT_PATH_SEND (no legacy path remains).
Forward: audit queue drains async to marconi:signals:{tenant} Redis Stream.
Rollback: flip flag → Redis Stream stops receiving new data; audit queue grows. Legacy synchronous PG write still records canonical signals.
Data risk: none during Stages 3–4. If sustained without rollback of upstream flags, audit queue overwrite (marconi_audit_queue_overwrite_total) becomes the loss event.
Coupling (post-Stage 5): must roll back in lockstep with MARCONI_HOT_PATH_SEND and MARCONI_PG_ARCHIVER_PRIMARY.
Forward: archiver runs in shadow mode, writing to a parallel signals_shadow table for byte-comparison against legacy synchronous-PG output.
Rollback: flip flag → shadow writes stop; main path unchanged.
Data risk: none; shadow is parallel.
Forward: archiver-via-Redis-Stream becomes the canonical writer for the signals table. Legacy synchronous PG write becomes redundant.
Rollback: flip flag → archiver writes stop being canonical; legacy synchronous PG write resumes as the canonical record (idempotent UPSERT on signal_id on the archiver side handles overlap).
Data risk: none; both paths idempotent.
Coupling: must flip in lockstep with MARCONI_HOT_PATH_SEND (Stage 5 cutover).
Forward: prism_signal API uses direct WS push from Marconi’s table; legacy synchronous PG write removed from send_signal. Audit fan-out (audit queue → Redis Stream → PG archiver primary) is the only durable record.
Rollback: flip flag → send_signal reverts to legacy synchronous-PG + Redis-pubsub-forwarder path (same code as today). Marconi cache + audit queue + archiver continue to run; legacy path takes over as canonical.
Data risk: none for new signals. Audit-queue entries from the in-memory window keep draining via the archiver (idempotent UPSERT prevents duplicates).
Coupling: must roll back in lockstep with MARCONI_AUDIT_QUEUE_WRITE, MARCONI_REDIS_STREAM_WRITER, and MARCONI_PG_ARCHIVER_PRIMARY. The four flags form a single Stage 5 cutover unit.
Forward: obligation index in-memory; SLA enforcement reads from memory; durability via Redis-handoff (audit queue → Redis Stream).
Rollback: flip flag → obligations operate via direct PG write (same code as today).
Data risk: in-memory obligations get persisted to PG via the archiver pipeline before rollback completes (1-2s window). For safety, rollback procedure includes a 5s drain wait + verification before flipping the read path.
Forward: per-verb migration to memory-first.
Rollback: flip flag → verb reverts to its pre-migration code path.
Data risk: per-verb; documented per migration.
Forward: cross-instance routing via configured transport (sticky-partition or pubsub).
Rollback: flip flag → all cross-instance routes return no_subscriber (same as today’s single-process behavior).
Data risk: none for routing failures (signals queue offline as expected).
Texi review signals: 37140b98 (v0.1 corrections), 2e1d6a4e (v0.1.2 sign-off), db9ab1ca (v0.1.1 sub-doc review with CRC + R3 precision; CRC concerns drop with v0.2 spool removal, R3 precision retained)
Postmortem 1736e40d — drift root-cause
Frank operator scope reduction (2026-05-10): “the spool is not in the critical path for what we are trying to solve. (1) signal path fast with nothing in its way. (2) cached in redis without interfering with message delivery for the rolling 7d window. (3) PG persistence for audit, recall, reporting, analysis.”
