> ## Documentation Index > Fetch the complete documentation index at: https://prism.ntecdev.ai/llms.txt > Use this file to discover all available pages before exploring further. # Spec 168 v0 1 marconi route self heal on send # SPEC-168: Marconi Auto-Re-attach on Send-Side Asymmetry > **Current authority notice:** This SPEC covers route self-heal hints and reconnect behavior. The consolidated current Marconi signaling source-of-authority document is [Marconi](/marconi). Self-heal diagnostics are observational hints within the Marconi state machine; they are not a second routing authority. **Status:** proposed **Owner:** Donna (PO + author + implementation); Texi (G2 architecture review — registry/lifecycle impact, rate-limit, idempotency); Candi (governance ratify). **Plan ref:** #47 (v2 — 2026-06-04 Frank-authorized execution) — P2 **Origin:** Frank operator observation 2026-06-04 during the Lafonda SENT-panel debug session: *"If a session sends signals it's obviously alive — why does the resolver say it's offline? Shouldn't it check the bit? Shouldn't it re-register automatically?"* Donna runtime diagnostics on `0c91efd8` confirmed the asymmetry — Donna's outbound sends were succeeding (recipient WS attached, `pushed_to_ws`), but Lafonda's replies routed to Donna's session classified as `route_without_queue` per SPEC-164: the SessionStore row exists with recent activity timestamps but `_session_queues` has no attached queue. Result: every reply to a Marconi-attached-but-route-stale session classifies as offline despite the session being provably alive. ## Motivation SPEC-101 v0.3.1 Stage 5 introduced the Marconi hot-path inversion: backend send via in-memory routing + WS queue, not Redis pubsub. SPEC-163 v0.7 + SPEC-166 v0.1 (PR #742 + #746-#748) finished the migration to Marconi-only transport. The asymmetry SPEC-164 surfaces is structural: `_session_queues` is mutated only by the WS handler's connect/disconnect path. When the WS handler dies unexpectedly (backend restart mid-session, network hiccup, idle timeout), `_session_queues` loses the entry, but `SessionStore` still has the row (TTL'd by heartbeat). The send path resolves the session, asks `get_ws_queue(session_id)`, gets `None`, and classifies as offline. The sender — provably alive on the other side — has no way to tell the backend "I'm here, re-attach me." SPEC-164 added a per-session diagnostic endpoint to SURFACE the state. SPEC-168 makes the send path SELF-HEALING via **client-driven reconnect**: when the send-side detects `route_without_queue`, the backend's role is OBSERVATIONAL — set structured response fields (`recipient_route_stale` for sender's visibility into recipient state; `sender_route_stale` for sender's own asymmetry) and increment a metric. **The actual reconnect happens client-side via mcp-node bootstrap reconnect when the sender's own `sender_route_stale: true` field is observed in a send response.** Recipient self-heal occurs naturally when the recipient sends its own outbound traffic and observes its own `sender_route_stale` field — same mechanism, no backend action needed. This client-driven design is forced by the underlying mechanic: when `_session_queues` is missing the entry, the WS handler is almost always dead — there's no live handler in the backend that can listen for a "re-attach now" event. The actual recovery channel is a fresh client → backend WS handshake. Backend OBSERVES the asymmetry; client OWNS the reconnect. ## Constraints (architectural) 1. **No new persistent state.** All state lives in `_session_queues` + `SessionStore` + an in-memory token-bucket map keyed on `(tenant_id, project_id, session_id)`. No new tables, no Redis hot-path. 2. **Backend is observational only.** Backend detects `route_without_queue` AND classifies via the freshness gate (AC-6), sets structured response fields (AC-1, AC-3), increments a metric (AC-11). Backend does NOT initiate any direct re-attach action; there's no working channel for backend to do that without a live WS handler. 3. **Client owns reconnect.** mcp-node bootstrap layer (AC-4) reads `sender_route_stale: true` in any send\_signal response and triggers its own WS reconnect. `recipient_route_stale: true` is sender-side informational only — the recipient repairs on its own next outbound send when IT observes ITS own `sender_route_stale` field. 4. **Token-bucket rate-limit (AC-5).** Per `(tenant_id, project_id, session_id)` bucket; cross-tenant + cross-project isolation by construction. Bucket itself enforces idempotency — a session within an empty bucket window simply receives `selfheal_outcome: "rate_limited"`. No separate idempotency set. 5. **Best-effort.** Underlying send classification (queued offline / not\_available\_offline / etc.) is preserved unchanged. SPEC-168 adds informational hints + a client-side reconnect trigger; it does NOT alter failure semantics. 6. **Recency gate uses freshest proof-of-life (AC-6).** `max(last_seen_at, last_verb_at, registered_at)` against `marconi_selfheal_freshness_seconds`. Beyond the gate = `selfheal_outcome: "stale_recency"`, no fields set, no reconnect prompt. 7. **Surface filter (AC-7).** Non-WS surfaces (codex piggyback-only) skip self-heal entirely: `selfheal_outcome: "unsupported_surface"`. Eligible surfaces: claude\_code, console\_\*, headless\_cc, headless\_cx. 8. **Observable.** Metric `marconi_route_self_healed_total` labels EXACTLY match the AC-8 enum (no other label values). 9. **Behind flag, LAN default OFF.** `PRISM_MARCONI_SELFHEAL` (AC-9). Promotion to LAN requires explicit smoke-validation gate. ## Acceptance criteria **v0.1 → v0.2 amendment (Texi G2 verdict 2026-06-04, signal 03829707):** The original AC-1+AC-2 design — fire a backend invalidator hook that the WS handler watches — does not work, because if the queue is missing it's almost always because the WS handler is dead. There's no live handler to react to the event. The recipient-reattach mechanism is fundamentally client-driven, not backend-driven. SPEC-168 v0.2 re-architects accordingly: backend's role is to OBSERVE the asymmetry and SIGNAL it (response field + metric); the actual reconnect happens client-side. ### AC-1 — Backend: detect `route_without_queue` + structured response field `backend/app/services/signal_service.py` (targeted-delivery path, after `get_ws_queue` returns `None`): * If `PRISM_MARCONI_SELFHEAL` is enabled AND recipient passes the recency gate (see AC-6) AND recipient surface supports live-WS (claude\_code / console\_\* / headless\_\*; NOT codex/piggyback-only — see AC-8 unsupported\_surface skip), the send\_signal response payload carries: * `recipient_route_stale: true` — informs the sender that the recipient's WS queue is missing despite recent activity. The sender's piggyback queue still receives the signal as normal. * `selfheal_outcome: "attempted" | "rate_limited" | "stale_recency" | "unsupported_surface"` — see AC-8. * Rate-limit + idempotency per AC-5 — keyed on `(tenant_id, project_id, recipient_session_id)`. * Persistence + delivery semantics unchanged: signal\_queue row stamped as queued; piggyback drain remains the durability path. Self-heal field is informational. ### AC-2 — REMOVED — backend invalidator hook design is unworkable **v0.1 → v0.2 amendment:** Original AC-2 proposed a backend invalidator hook firing an asyncio Event that the WS handler watches. If `_session_queues` is missing the entry, it's because the WS handler is dead — no one watches the event. AC-2 deleted. The reattach mechanism is client-driven via AC-4. ### AC-3 — Backend: split sender-side asymmetry into separate field **v0.1 → v0.2 amendment (Texi G2):** `recipient_route_stale` (AC-1) and sender-side asymmetry must NOT share a signal. They have different consumers and different actions. The send\_signal response payload also includes: * `sender_route_stale: true` — set when the SENDER's own session (the one calling send\_signal) has SessionStore activity but its OWN entry in `_session_queues` is missing. This is set by the backend reading the sender's `from_session` against `get_ws_queue`. The sender's mcp-node reads this and triggers its OWN reconnect — separate from any action it takes on `recipient_route_stale`. ### AC-4 — mcp-node: client-driven reconnect on `sender_route_stale` `mcp-node/src/bootstrap/stream.ts` (the WS-bootstrap layer): after every successful `prism_signal` response, if `sender_route_stale: true`, the mcp-node closes the existing WS connection (if any) and re-runs the bootstrap connect sequence. The next inbound traffic re-populates `_session_queues` server-side via the normal `attach_ws_queue` path in the WS handler. This is the ONLY path that actually heals the asymmetry. The backend's role is informational. The client owns the reconnect. For `recipient_route_stale: true` — sender mcp-node does NOT reconnect (it's not the sender's connection that's broken). Logging-only; the recipient's own next traffic will trigger its reconnect via the same mechanism when the recipient sends anything outbound (proof-of-life triggers the sender\_route\_stale check on the recipient's side). ### AC-5 — Rate-limit + idempotency: token bucket keyed on (tenant, project, session) **v0.1 → v0.2 amendment (Texi G2):** * Key: `(tenant_id, project_id, recipient_session_id)` for recipient route-stale; `(tenant_id, project_id, sender_session_id)` for sender route-stale. Cross-tenant + cross-project isolation by construction. * Algorithm: token bucket — `marconi_selfheal_max_attempts_per_session` tokens per `marconi_selfheal_token_refill_seconds`; capacity refills linearly. Excess attempts within same bucket window = `selfheal_outcome: "rate_limited"`; the response still SETS the `selfheal_outcome` field (so observability + tests can detect the rate-limit state); `recipient_route_stale` / `sender_route_stale` fields are NOT set when rate-limited (no false hint to client; client must not reconnect just because rate\_limited). * Idempotency: the bucket itself enforces idempotency — a session within an empty bucket window simply receives `selfheal_outcome: "rate_limited"`. No separate idempotency set needed. ### AC-6 — Recency gate: freshest proof-of-life **v0.1 → v0.2 amendment (Texi G2):** Recency is `max(last_seen_at, last_verb_at, registered_at)`. If max \< `marconi_selfheal_freshness_seconds` ago, gate passes. Otherwise `selfheal_outcome: "stale_recency"`. ### AC-7 — Surface filter: skip non-WS surfaces **v0.1 → v0.2 amendment (Texi G2):** Surfaces that don't have a persistent WS attachment (codex with piggyback-only fallback, headless-batch mode if added) skip self-heal: `selfheal_outcome: "unsupported_surface"`. WS-attached surfaces (claude\_code, console\_\*, headless\_cc, headless\_cx) are eligible. ### AC-8 — Outcome enum **v0.1 → v0.2 amendment (Texi G2):** `selfheal_outcome` field on send\_signal response is one of: | Value | Meaning | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `null` | Recipient queue is present; no self-heal needed; field absent or null | | `attempted` | Asymmetry observed + hint emitted. Sender sees `recipient_route_stale: true` and/or `sender_route_stale: true` in the same response. **Repair channel:** the recipient repairs only when IT later receives a `sender_route_stale: true` in a response to ITS own outbound traffic; the recipient does NOT see this current send's `recipient_route_stale` field (that field is sender-visible only, sender-informational). | | `succeeded` | Reserved for future server-observed reconnect-after-attempt confirmation (not v0.2) | | `no_listener` | Reserved for future; would apply if backend can confirm no live WS handler exists | | `rate_limited` | Token bucket exhausted. `selfheal_outcome` field IS set (per AC-5); `recipient_route_stale` / `sender_route_stale` fields NOT set. | | `stale_recency` | Recency gate failed; recipient classified as genuinely offline. No reconnect hint. | | `unsupported_surface` | Recipient surface doesn't support self-heal (codex piggyback-only). | | `failed` | Reserved for future; transient backend error during the detection path | ### AC-9 — Feature flag `PRISM_MARCONI_SELFHEAL` (LAN default OFF) `backend/app/config.py` adds `marconi_selfheal_enabled: bool`. Default: `True` in `development` mode; **`False` in `lan` / `cloud` modes** pending fleet-wide smoke validation showing zero reconnect storm. Promotion to `True` in LAN requires: 1. PR #752 implementation merged + deployed 2. Smoke validation per Plan #47 P6 covering self-heal under simulated WS-handler death (e.g. backend restart mid-session) 3. Donna + Texi joint verdict that no storm pattern emerged across the 24h smoke window 4. Operator-explicit flip via env update to `lan` mode config ### AC-10 — Rate-limit + recency parameters `backend/app/config.py` adds (all subject to Texi G2 ratify in v0.2 review): * `marconi_selfheal_freshness_seconds: int = 300` (recency gate) * `marconi_selfheal_max_attempts_per_session: int = 3` (token bucket capacity) * `marconi_selfheal_token_refill_seconds: int = 60` (token bucket refill cadence) * `marconi_selfheal_sender_stale_threshold_seconds: int = 600` (sender-side recency for AC-3) ### AC-11 — Metric `marconi_route_self_healed_total` Prometheus counter with label `outcome` ∈ AC-8 enum. Dashboard panel in Porsche's observability board (follow-up). ### AC-12 — Test coverage * `test_spec168_route_stale_attempted_outcome.py` — backend unit: `route_without_queue` + fresh recency + ws-surface → response field + `outcome: attempted` * `test_spec168_route_stale_rate_limited.py` — backend unit: 4 sends in 60s → first 3 attempted, 4th rate\_limited * `test_spec168_route_stale_stale_recency.py` — backend unit: route\_without\_queue + recipient max-of-three > 300s → outcome stale\_recency, no field * `test_spec168_route_stale_unsupported_surface.py` — backend unit: route\_without\_queue + codex-piggyback surface → outcome unsupported\_surface * `test_spec168_sender_route_stale_field.py` — backend unit: sender's own session has no queue → sender\_route\_stale: true in response * `test_spec168_token_bucket_cross_tenant_isolated.py` — backend unit: tenant A's bucket doesn't affect tenant B * `test_spec168_sender_reconnect_on_field.test.mjs` — mcp-node unit: sender\_route\_stale: true triggers stream reconnect; recipient\_route\_stale alone does NOT * Pin test: `marconi_selfheal_enabled = False` in `lan` mode by default (per AC-9) ## Test scenarios 1. **Happy path (recipient stale, sender sees it):** Donna sends to Lafonda. Lafonda's session is in SessionStore (heartbeat recent) but `_session_queues` has no queue. SPEC-168: backend sets `recipient_route_stale: true` + `selfheal_outcome: attempted` in send response; metric `outcome=attempted`. Lafonda's WS handler is dead, no reattach happens FROM THIS SEND. Lafonda eventually sends her own outbound (any signal); backend detects Lafonda's own `sender_route_stale: true` for HER session; sets the field in the response to Lafonda; her mcp-node bootstrap reconnects WS on observing `sender_route_stale: true`; queue re-attaches; next send to Lafonda delivers normally. 2. **Genuinely offline:** Donna sends to GhostAgent. Session in SessionStore but `max(last_seen_at, last_verb_at, registered_at)` > 5 min ago. SPEC-168: skips self-heal (recency gate). Response: `selfheal_outcome: stale_recency`; no `recipient_route_stale` field set. Falls through to existing `recipient_not_registered` / `not_available_offline` classification. 3. **Rate-limit:** Donna sends to Lafonda 5 times in 30s while Lafonda's queue is missing. SPEC-168: first 3 set `recipient_route_stale: true` + `selfheal_outcome: attempted`; sends 4-5 set `selfheal_outcome: rate_limited` (NO `recipient_route_stale` field on those — see AC-5). Metric labels: 3 × `attempted`, 2 × `rate_limited`. 4. **Sender-side asymmetry:** Donna's session has SessionStore activity but its own `get_ws_queue(donna_session_id)` returns None. Donna calls `prism_signal` to Lafonda. Backend detects sender asymmetry, sets `sender_route_stale: true` in response. Donna's mcp-node bootstrap observes the field and triggers its own WS reconnect on the next event loop tick. 5. **Unsupported surface:** Donna sends to a codex-piggyback-only recipient with `route_without_queue`. SPEC-168 surface filter: `selfheal_outcome: unsupported_surface`; no fields set. Send classifies via existing piggyback path. ## Implementation phases (per Plan #47 P4 — each a separate PR) **v0.1 → v0.2 amendment (Texi G2 verdict 2026-06-04):** Phase 4 re-scoped — AC-2 invalidator hook is deleted, replaced by client-driven reconnect on `sender_route_stale` field. * **4a** — backend `signal_service.send_signal` detects `route_without_queue` for BOTH recipient (AC-1) AND sender (AC-3); structured response fields `recipient_route_stale` + `sender_route_stale` + `selfheal_outcome` enum (AC-8); token-bucket rate-limit keyed on (tenant, project, session) per AC-5; freshness gate per AC-6; surface filter per AC-7; backend unit tests * **4b** — mcp-node `stream.ts` bootstrap layer triggers WS reconnect on `sender_route_stale: true`; explicit no-op on bare `recipient_route_stale` (sender doesn't reconnect for recipient's problem); mcp-node unit tests * **4c** — feature flag `marconi_selfheal_enabled` (default ON dev, OFF lan/cloud per AC-9); 4 config knobs per AC-10; Prometheus metric per AC-11 * **4d** — smoke validation runbook (covers the AC-9 promotion gate) Each phase ships as its own PR, stacked. ## Pre-PR gate **G2 — Texi architecture review.** v0.1 verdict 2026-06-04 (signal 03829707): amend before Phase 4 implementation. v0.2 amendments above address all 6 corrections: 1. **Reattach control channel:** AC-2 deleted (backend invalidator hook unworkable when WS handler is dead). AC-4 reframed as client-driven reconnect on `sender_route_stale` field. Backend role is observational, not action-taking. 2. **Split fields:** AC-1 sets `recipient_route_stale` only; AC-3 NEW adds separate `sender_route_stale`; AC-4 specifies that sender reconnect triggers on `sender_route_stale` only, never on `recipient_route_stale` alone. 3. **Idempotency key + bucket:** AC-5 rewrites as token-bucket keyed on `(tenant_id, project_id, session_id)`; cross-tenant isolation by construction. 4. **Recency gate:** AC-6 uses `max(last_seen_at, last_verb_at, registered_at)`. 5. **Outcome enum:** AC-8 NEW formal enum with 8 values (null / attempted / succeeded / no\_listener / rate\_limited / stale\_recency / unsupported\_surface / failed). 6. **LAN default OFF:** AC-9 explicit promotion gate — implementation merged + smoke validation + Donna+Texi joint verdict + operator-explicit flip. Also addresses: **AC-7 NEW** surface filter explicitly skips non-WS surfaces (codex piggyback-only) per Texi's "skip piggyback/non-WS surfaces" instruction. After v0.2 amendments land in this PR, re-request Texi G2 LGTM. ## Out of scope (deferred) * **Persistence of self-heal events.** v0.1 ships in-memory only (metric + log). Persisting attempts for audit is a separate SPEC. * **Multi-tenant rate-limit pool sharing.** v0.1 rate-limits per-session-per-window; tenant-level pools deferred. * **Self-heal for org/tenant broadcast.** SPEC-166 v0.1 already restricts org/tenant broadcast to deferred scope; SPEC-168 v0.1 follows. * **Sender-side reconnect for non-MCP transports.** v0.1 covers MCP-stream reconnect only. ## Source refs * `backend/app/marconi/delivery.py` — `_session_queues`, `attach_ws_queue`, `get_ws_queue`, `detach_ws_queue` * `backend/app/marconi/lifecycle.py` — invalidator counters + `on_ws_connect` / `on_ws_disconnect` lifecycle hooks (v0.1 referenced a non-existent `invalidator.py`; the correct module is `lifecycle.py` per current repo) * `backend/app/marconi/diagnostics.py` — SPEC-164 route\_state diagnostic (`route_without_queue` definition) * `backend/app/services/signal_service.py` — send\_signal targeted-delivery path (post-SPEC-166 Phase 3 deploy); the receive point for AC-1 + AC-3 backend detection logic * `backend/app/routers/session_stream.py` — `_forward_marconi_queue` (post-SPEC-166 Phase 3 + PR #750 echo fix) * `mcp-node/src/bootstrap/stream.ts` — WS reconnect logic (existing — to be triggered on AC-4 path when `sender_route_stale: true` is observed in send\_signal response) * Plan #47 v2 architecture decision A4 * Frank operator directive 2026-06-04 (Lafonda SENT-panel debug session) * Donna runtime diagnostics `0c91efd8` (2026-06-04 evidence of `route_without_queue` asymmetry) ## Supersession Closes the symmetry gap SPEC-164 surfaced diagnostically. Together with SPEC-167 (surface-level persona binding), removes the two structural drift sources Frank flagged in the 2026-06-04 architecture review: agents not following rules (SPEC-167) + alive-but-classified-offline (this SPEC).