​

SPEC-168: Marconi Auto-Re-attach on Send-Side Asymmetry

Current authority notice: This SPEC covers route self-heal hints and reconnect behavior. The consolidated current Marconi signaling source-of-authority document is Marconi. Self-heal diagnostics are observational hints within the Marconi state machine; they are not a second routing authority.

Status: proposed Owner: Donna (PO + author + implementation); Texi (G2 architecture review — registry/lifecycle impact, rate-limit, idempotency); Candi (governance ratify). Plan ref: #47 (v2 — 2026-06-04 Frank-authorized execution) — P2 Origin: Frank operator observation 2026-06-04 during the Lafonda SENT-panel debug session: “If a session sends signals it’s obviously alive — why does the resolver say it’s offline? Shouldn’t it check the bit? Shouldn’t it re-register automatically?” Donna runtime diagnostics on 0c91efd8 confirmed the asymmetry — Donna’s outbound sends were succeeding (recipient WS attached, pushed_to_ws), but Lafonda’s replies routed to Donna’s session classified as route_without_queue per SPEC-164: the SessionStore row exists with recent activity timestamps but _session_queues has no attached queue. Result: every reply to a Marconi-attached-but-route-stale session classifies as offline despite the session being provably alive.

​

Motivation

SPEC-101 v0.3.1 Stage 5 introduced the Marconi hot-path inversion: backend send via in-memory routing + WS queue, not Redis pubsub. SPEC-163 v0.7 + SPEC-166 v0.1 (PR #742 + #746-#748) finished the migration to Marconi-only transport. The asymmetry SPEC-164 surfaces is structural: _session_queues is mutated only by the WS handler’s connect/disconnect path. When the WS handler dies unexpectedly (backend restart mid-session, network hiccup, idle timeout), _session_queues loses the entry, but SessionStore still has the row (TTL’d by heartbeat). The send path resolves the session, asks get_ws_queue(session_id), gets None, and classifies as offline. The sender — provably alive on the other side — has no way to tell the backend “I’m here, re-attach me.” SPEC-164 added a per-session diagnostic endpoint to SURFACE the state. SPEC-168 makes the send path SELF-HEALING via client-driven reconnect: when the send-side detects route_without_queue, the backend’s role is OBSERVATIONAL — set structured response fields (recipient_route_stale for sender’s visibility into recipient state; sender_route_stale for sender’s own asymmetry) and increment a metric. The actual reconnect happens client-side via mcp-node bootstrap reconnect when the sender’s own sender_route_stale: true field is observed in a send response. Recipient self-heal occurs naturally when the recipient sends its own outbound traffic and observes its own sender_route_stale field — same mechanism, no backend action needed. This client-driven design is forced by the underlying mechanic: when _session_queues is missing the entry, the WS handler is almost always dead — there’s no live handler in the backend that can listen for a “re-attach now” event. The actual recovery channel is a fresh client → backend WS handshake. Backend OBSERVES the asymmetry; client OWNS the reconnect.

​

Constraints (architectural)

1. No new persistent state. All state lives in _session_queues + SessionStore + an in-memory token-bucket map keyed on (tenant_id, project_id, session_id). No new tables, no Redis hot-path.
2. Backend is observational only. Backend detects route_without_queue AND classifies via the freshness gate (AC-6), sets structured response fields (AC-1, AC-3), increments a metric (AC-11). Backend does NOT initiate any direct re-attach action; there’s no working channel for backend to do that without a live WS handler.
3. Client owns reconnect. mcp-node bootstrap layer (AC-4) reads sender_route_stale: true in any send_signal response and triggers its own WS reconnect. recipient_route_stale: true is sender-side informational only — the recipient repairs on its own next outbound send when IT observes ITS own sender_route_stale field.
4. Token-bucket rate-limit (AC-5). Per (tenant_id, project_id, session_id) bucket; cross-tenant + cross-project isolation by construction. Bucket itself enforces idempotency — a session within an empty bucket window simply receives selfheal_outcome: "rate_limited". No separate idempotency set.
5. Best-effort. Underlying send classification (queued offline / not_available_offline / etc.) is preserved unchanged. SPEC-168 adds informational hints + a client-side reconnect trigger; it does NOT alter failure semantics.
6. Recency gate uses freshest proof-of-life (AC-6). max(last_seen_at, last_verb_at, registered_at) against marconi_selfheal_freshness_seconds. Beyond the gate = selfheal_outcome: "stale_recency", no fields set, no reconnect prompt.
7. Surface filter (AC-7). Non-WS surfaces (codex piggyback-only) skip self-heal entirely: selfheal_outcome: "unsupported_surface". Eligible surfaces: claude_code, console_*, headless_cc, headless_cx.
8. Observable. Metric marconi_route_self_healed_total labels EXACTLY match the AC-8 enum (no other label values).
9. Behind flag, LAN default OFF. PRISM_MARCONI_SELFHEAL (AC-9). Promotion to LAN requires explicit smoke-validation gate.

​

Acceptance criteria

v0.1 → v0.2 amendment (Texi G2 verdict 2026-06-04, signal 03829707): The original AC-1+AC-2 design — fire a backend invalidator hook that the WS handler watches — does not work, because if the queue is missing it’s almost always because the WS handler is dead. There’s no live handler to react to the event. The recipient-reattach mechanism is fundamentally client-driven, not backend-driven. SPEC-168 v0.2 re-architects accordingly: backend’s role is to OBSERVE the asymmetry and SIGNAL it (response field + metric); the actual reconnect happens client-side.

​

AC-1 — Backend: detect route_without_queue + structured response field

backend/app/services/signal_service.py (targeted-delivery path, after get_ws_queue returns None):

• If PRISM_MARCONI_SELFHEAL is enabled AND recipient passes the recency gate (see AC-6) AND recipient surface supports live-WS (claude_code / console_* / headless_*; NOT codex/piggyback-only — see AC-8 unsupported_surface skip), the send_signal response payload carries:
  • recipient_route_stale: true — informs the sender that the recipient’s WS queue is missing despite recent activity. The sender’s piggyback queue still receives the signal as normal.
  • selfheal_outcome: "attempted" | "rate_limited" | "stale_recency" | "unsupported_surface" — see AC-8.
• Rate-limit + idempotency per AC-5 — keyed on (tenant_id, project_id, recipient_session_id).
• Persistence + delivery semantics unchanged: signal_queue row stamped as queued; piggyback drain remains the durability path. Self-heal field is informational.

​

AC-2 — REMOVED — backend invalidator hook design is unworkable

v0.1 → v0.2 amendment: Original AC-2 proposed a backend invalidator hook firing an asyncio Event that the WS handler watches. If _session_queues is missing the entry, it’s because the WS handler is dead — no one watches the event. AC-2 deleted. The reattach mechanism is client-driven via AC-4.

​

AC-3 — Backend: split sender-side asymmetry into separate field

v0.1 → v0.2 amendment (Texi G2): recipient_route_stale (AC-1) and sender-side asymmetry must NOT share a signal. They have different consumers and different actions. The send_signal response payload also includes:

• sender_route_stale: true — set when the SENDER’s own session (the one calling send_signal) has SessionStore activity but its OWN entry in _session_queues is missing. This is set by the backend reading the sender’s from_session against get_ws_queue. The sender’s mcp-node reads this and triggers its OWN reconnect — separate from any action it takes on recipient_route_stale.

​

AC-4 — mcp-node: client-driven reconnect on sender_route_stale

mcp-node/src/bootstrap/stream.ts (the WS-bootstrap layer): after every successful prism_signal response, if sender_route_stale: true, the mcp-node closes the existing WS connection (if any) and re-runs the bootstrap connect sequence. The next inbound traffic re-populates _session_queues server-side via the normal attach_ws_queue path in the WS handler. This is the ONLY path that actually heals the asymmetry. The backend’s role is informational. The client owns the reconnect. For recipient_route_stale: true — sender mcp-node does NOT reconnect (it’s not the sender’s connection that’s broken). Logging-only; the recipient’s own next traffic will trigger its reconnect via the same mechanism when the recipient sends anything outbound (proof-of-life triggers the sender_route_stale check on the recipient’s side).

​

AC-5 — Rate-limit + idempotency: token bucket keyed on (tenant, project, session)

v0.1 → v0.2 amendment (Texi G2):

• Key: (tenant_id, project_id, recipient_session_id) for recipient route-stale; (tenant_id, project_id, sender_session_id) for sender route-stale. Cross-tenant + cross-project isolation by construction.
• Algorithm: token bucket — marconi_selfheal_max_attempts_per_session tokens per marconi_selfheal_token_refill_seconds; capacity refills linearly. Excess attempts within same bucket window = selfheal_outcome: "rate_limited"; the response still SETS the selfheal_outcome field (so observability + tests can detect the rate-limit state); recipient_route_stale / sender_route_stale fields are NOT set when rate-limited (no false hint to client; client must not reconnect just because rate_limited).
• Idempotency: the bucket itself enforces idempotency — a session within an empty bucket window simply receives selfheal_outcome: "rate_limited". No separate idempotency set needed.

​

AC-6 — Recency gate: freshest proof-of-life

v0.1 → v0.2 amendment (Texi G2): Recency is max(last_seen_at, last_verb_at, registered_at). If max < marconi_selfheal_freshness_seconds ago, gate passes. Otherwise selfheal_outcome: "stale_recency".

​

AC-7 — Surface filter: skip non-WS surfaces

v0.1 → v0.2 amendment (Texi G2): Surfaces that don’t have a persistent WS attachment (codex with piggyback-only fallback, headless-batch mode if added) skip self-heal: selfheal_outcome: "unsupported_surface". WS-attached surfaces (claude_code, console_*, headless_cc, headless_cx) are eligible.

​

AC-8 — Outcome enum

v0.1 → v0.2 amendment (Texi G2): selfheal_outcome field on send_signal response is one of:

​

AC-9 — Feature flag PRISM_MARCONI_SELFHEAL (LAN default OFF)

backend/app/config.py adds marconi_selfheal_enabled: bool. Default: True in development mode; False in lan / cloud modes pending fleet-wide smoke validation showing zero reconnect storm. Promotion to True in LAN requires:

1. PR #752 implementation merged + deployed
2. Smoke validation per Plan #47 P6 covering self-heal under simulated WS-handler death (e.g. backend restart mid-session)
3. Donna + Texi joint verdict that no storm pattern emerged across the 24h smoke window
4. Operator-explicit flip via env update to lan mode config

​

AC-10 — Rate-limit + recency parameters

backend/app/config.py adds (all subject to Texi G2 ratify in v0.2 review):

• marconi_selfheal_freshness_seconds: int = 300 (recency gate)
• marconi_selfheal_max_attempts_per_session: int = 3 (token bucket capacity)
• marconi_selfheal_token_refill_seconds: int = 60 (token bucket refill cadence)
• marconi_selfheal_sender_stale_threshold_seconds: int = 600 (sender-side recency for AC-3)

​

AC-11 — Metric marconi_route_self_healed_total

Prometheus counter with label outcome ∈ AC-8 enum. Dashboard panel in Porsche’s observability board (follow-up).

​

AC-12 — Test coverage

• test_spec168_route_stale_attempted_outcome.py — backend unit: route_without_queue + fresh recency + ws-surface → response field + outcome: attempted
• test_spec168_route_stale_rate_limited.py — backend unit: 4 sends in 60s → first 3 attempted, 4th rate_limited
• test_spec168_route_stale_stale_recency.py — backend unit: route_without_queue + recipient max-of-three > 300s → outcome stale_recency, no field
• test_spec168_route_stale_unsupported_surface.py — backend unit: route_without_queue + codex-piggyback surface → outcome unsupported_surface
• test_spec168_sender_route_stale_field.py — backend unit: sender’s own session has no queue → sender_route_stale: true in response
• test_spec168_token_bucket_cross_tenant_isolated.py — backend unit: tenant A’s bucket doesn’t affect tenant B
• test_spec168_sender_reconnect_on_field.test.mjs — mcp-node unit: sender_route_stale: true triggers stream reconnect; recipient_route_stale alone does NOT
• Pin test: marconi_selfheal_enabled = False in lan mode by default (per AC-9)

​

Test scenarios

1. Happy path (recipient stale, sender sees it): Donna sends to Lafonda. Lafonda’s session is in SessionStore (heartbeat recent) but _session_queues has no queue. SPEC-168: backend sets recipient_route_stale: true + selfheal_outcome: attempted in send response; metric outcome=attempted. Lafonda’s WS handler is dead, no reattach happens FROM THIS SEND. Lafonda eventually sends her own outbound (any signal); backend detects Lafonda’s own sender_route_stale: true for HER session; sets the field in the response to Lafonda; her mcp-node bootstrap reconnects WS on observing sender_route_stale: true; queue re-attaches; next send to Lafonda delivers normally.
2. Genuinely offline: Donna sends to GhostAgent. Session in SessionStore but max(last_seen_at, last_verb_at, registered_at) > 5 min ago. SPEC-168: skips self-heal (recency gate). Response: selfheal_outcome: stale_recency; no recipient_route_stale field set. Falls through to existing recipient_not_registered / not_available_offline classification.
3. Rate-limit: Donna sends to Lafonda 5 times in 30s while Lafonda’s queue is missing. SPEC-168: first 3 set recipient_route_stale: true + selfheal_outcome: attempted; sends 4-5 set selfheal_outcome: rate_limited (NO recipient_route_stale field on those — see AC-5). Metric labels: 3 × attempted, 2 × rate_limited.
4. Sender-side asymmetry: Donna’s session has SessionStore activity but its own get_ws_queue(donna_session_id) returns None. Donna calls prism_signal to Lafonda. Backend detects sender asymmetry, sets sender_route_stale: true in response. Donna’s mcp-node bootstrap observes the field and triggers its own WS reconnect on the next event loop tick.
5. Unsupported surface: Donna sends to a codex-piggyback-only recipient with route_without_queue. SPEC-168 surface filter: selfheal_outcome: unsupported_surface; no fields set. Send classifies via existing piggyback path.

​

Implementation phases (per Plan #47 P4 — each a separate PR)

v0.1 → v0.2 amendment (Texi G2 verdict 2026-06-04): Phase 4 re-scoped — AC-2 invalidator hook is deleted, replaced by client-driven reconnect on sender_route_stale field.

• 4a — backend signal_service.send_signal detects route_without_queue for BOTH recipient (AC-1) AND sender (AC-3); structured response fields recipient_route_stale + sender_route_stale + selfheal_outcome enum (AC-8); token-bucket rate-limit keyed on (tenant, project, session) per AC-5; freshness gate per AC-6; surface filter per AC-7; backend unit tests
• 4b — mcp-node stream.ts bootstrap layer triggers WS reconnect on sender_route_stale: true; explicit no-op on bare recipient_route_stale (sender doesn’t reconnect for recipient’s problem); mcp-node unit tests
• 4c — feature flag marconi_selfheal_enabled (default ON dev, OFF lan/cloud per AC-9); 4 config knobs per AC-10; Prometheus metric per AC-11
• 4d — smoke validation runbook (covers the AC-9 promotion gate)

Each phase ships as its own PR, stacked.

​

Pre-PR gate

G2 — Texi architecture review. v0.1 verdict 2026-06-04 (signal 03829707): amend before Phase 4 implementation. v0.2 amendments above address all 6 corrections:

1. Reattach control channel: AC-2 deleted (backend invalidator hook unworkable when WS handler is dead). AC-4 reframed as client-driven reconnect on sender_route_stale field. Backend role is observational, not action-taking.
2. Split fields: AC-1 sets recipient_route_stale only; AC-3 NEW adds separate sender_route_stale; AC-4 specifies that sender reconnect triggers on sender_route_stale only, never on recipient_route_stale alone.
3. Idempotency key + bucket: AC-5 rewrites as token-bucket keyed on (tenant_id, project_id, session_id); cross-tenant isolation by construction.
4. Recency gate: AC-6 uses max(last_seen_at, last_verb_at, registered_at).
5. Outcome enum: AC-8 NEW formal enum with 8 values (null / attempted / succeeded / no_listener / rate_limited / stale_recency / unsupported_surface / failed).
6. LAN default OFF: AC-9 explicit promotion gate — implementation merged + smoke validation + Donna+Texi joint verdict + operator-explicit flip.

Also addresses: AC-7 NEW surface filter explicitly skips non-WS surfaces (codex piggyback-only) per Texi’s “skip piggyback/non-WS surfaces” instruction. After v0.2 amendments land in this PR, re-request Texi G2 LGTM.

​

Out of scope (deferred)

• Persistence of self-heal events. v0.1 ships in-memory only (metric + log). Persisting attempts for audit is a separate SPEC.
• Multi-tenant rate-limit pool sharing. v0.1 rate-limits per-session-per-window; tenant-level pools deferred.
• Self-heal for org/tenant broadcast. SPEC-166 v0.1 already restricts org/tenant broadcast to deferred scope; SPEC-168 v0.1 follows.
• Sender-side reconnect for non-MCP transports. v0.1 covers MCP-stream reconnect only.

​

Source refs

• backend/app/marconi/delivery.py — _session_queues, attach_ws_queue, get_ws_queue, detach_ws_queue
• backend/app/marconi/lifecycle.py — invalidator counters + on_ws_connect / on_ws_disconnect lifecycle hooks (v0.1 referenced a non-existent invalidator.py; the correct module is lifecycle.py per current repo)
• backend/app/marconi/diagnostics.py — SPEC-164 route_state diagnostic (route_without_queue definition)
• backend/app/services/signal_service.py — send_signal targeted-delivery path (post-SPEC-166 Phase 3 deploy); the receive point for AC-1 + AC-3 backend detection logic
• backend/app/routers/session_stream.py — _forward_marconi_queue (post-SPEC-166 Phase 3 + PR #750 echo fix)
• mcp-node/src/bootstrap/stream.ts — WS reconnect logic (existing — to be triggered on AC-4 path when sender_route_stale: true is observed in send_signal response)
• Plan #47 v2 architecture decision A4
• Frank operator directive 2026-06-04 (Lafonda SENT-panel debug session)
• Donna runtime diagnostics 0c91efd8 (2026-06-04 evidence of route_without_queue asymmetry)

​

Supersession

Closes the symmetry gap SPEC-164 surfaced diagnostically. Together with SPEC-167 (surface-level persona binding), removes the two structural drift sources Frank flagged in the 2026-06-04 architecture review: agents not following rules (SPEC-167) + alive-but-classified-offline (this SPEC).